Blueprint embeds explainability into AI systems from the ground up

Artificial intelligence systems increasingly make decisions in hospitals, banks and on the road, yet most still struggle to explain what they are doing at the moment they do it. A new open-access paper argues that this gap cannot be closed by bolting explanations onto finished models, and proposes instead an architectural blueprint that builds explainability into the system from the start. The framework, called System of Systems for Machine Learning, or SoS-ML, was introduced by Rittika Shamsuddin, Habib B. Tabrizi and Pavan R. Gottimukkula in the journal AI Perspectives & Advances, published on 17 January 2025.

The authors begin from a distinction that runs through the explainable AI literature. Interpretability refers to models whose internal mechanics a human can follow directly, such as decision trees or linear regression. Explainability, by contrast, refers to post-hoc justifications generated for opaque black-box models such as deep neural networks. Popular tools like LIME, developed by Ribeiro and colleagues in 2016, and SHAP, built on cooperative game theory by Lundberg and Lee in 2017, perturb inputs or assign feature contributions to explain individual predictions. The paper argues that these explanations are primarily aimed at developers, are often too technical for end-users, and do not always faithfully reflect the internal workings of the model, raising reliability concerns in critical applications such as healthcare and legal decision-making.

The debate over whether post-hoc explanations are trustworthy enough for high-stakes decisions is well documented. Some researchers, notably Rudin, argue that inherently interpretable models should be preferred wherever transparency and accountability are paramount, because post-hoc explanations can be incomplete or misleading. Others contend that post-hoc methods are a necessary price for the accuracy of complex models. The authors of the new paper position themselves outside this dichotomy: rather than choosing between interpretable models and explained black boxes, they propose designing systems in which transparency emerges from the architecture itself, a concept they call functional interpretability.

SoS-ML draws its design principles from the natural and social sciences. The authors note that deep learning models are often called black boxes because iterative backpropagation disperses the contribution of each neuron across the network, making individual impacts hard to trace. They compare this to the human brain, where no single neuron is responsible for a given function; instead, neuroscientists study regions of interest, clusters of neurons linked to specific tasks such as Broca’s area for speech and grammar and Wernicke’s area for language comprehension. By analogy, the framework defines functional units whose group-level contributions to decisions are understandable even when their internal computations remain complex.

Linguistic theories supply further specifications. From Chomsky’s Innatist Theory, with its hypothetical Language Acquisition Device, the authors borrow the idea of built-in modularity: components designed from the outset to handle distinct, explainable functions. From Vygotsky’s Social Interactionist Theory they take the Zone of Proximal Development and scaffolding, translating them into structured collaboration among diverse, hierarchical units. Insights from philosophy and cognitive science shape the output format: explanations are treated as contextual social constructs that are often contrastive, biased toward a limited set of causes, and preferred as causal narratives over raw statistical probabilities. The framework therefore structures each AI explanation as an evidence-inference pair embedded in context, rather than a bare probability statement.

The resulting blueprint is a system of systems in the engineering sense: a collection of autonomous components that collaborate while retaining their own functionality. Three kinds of functional units appear in the design. Data Agents are machine learning models that process datasets and act as the system’s regions of interest. Interaction Platforms serve as shared spaces where agents exchange information and generate decision-evidence pairs, functioning as the framework’s analogue of a Language Acquisition Device. System Agents coordinate system-level and contextual information, and supporting devices convert decision-evidence pairs into inference-evidence pairs, add context, and, in later phases, use a fine-tuned large language model to produce narrative explanation logs with sections for end-users, developers and interested parties.

Development is phased. Phase 1 establishes a multi-agent configuration in which agents process subsets of data and interact through weighted majority voting. Phase 2 adds system control and initial context detection, allowing agents to be dynamically selected according to task complexity. Phase 3 introduces full contextual response, scaling up the number and diversity of agents, integrating multiple information pathways, and implementing control mechanisms to identify and report weaknesses and biases within the system. The authors describe hypothetical Phase 3 scenarios, including a salary prediction system in which specialized agents detect and flag gender bias, and a hypertension assistant composed of models trained on lifestyle data, gene expression data and electronic medical records, with a decision center that validates outputs against a corpus of domain knowledge. These scenarios are design illustrations, not tested systems.

The empirical evidence comes from two proof-of-concept experiments. In Phase 1, the team tested whether multi-agent interaction improves evidentiary information on the Pima Indian Diabetes dataset, using a k-nearest neighbor model and a small neural network with a single hidden layer of two neurons, combined through weighted majority voting, with no existing interpretable techniques used in the setup. The system reached an overall accuracy of 80 percent, which the authors describe as competitive with the usual 70 to 85 percent range reported for neural networks on this dataset, and the abstract reports a 5 to 10 percent improvement over traditional models. Feature groupings yielded reliance scores with confidence intervals: the grouping of pedigree, BMI and glucose reached 81 percent reliance with a 95 percent confidence interval of 75 to 87.5 percent and a significant p-value, while the grouping of skin thickness, insulin and age reached 74 percent reliance with a non-significant interval, indicating less certainty about those variables.

The second experiment, a Phase 2 Pie Chart Interpreter, was designed to explain pie chart slice sizes from images. Three deep learning agents processed color, intensity and edges, a Q-learning agent with an epsilon-greedy policy selected among seven possible combinations of these agents, and a fine-tuned large language model translated the numerical results into plain English. The image processing module achieved an L1-norm percentage of 84.35 percent, and the numerical relationship module recorded a mean accuracy of 99.54 percent with an AUROC of 99.95 percent. Selection patterns shifted with chart complexity: the color and intensity combination was chosen 45 percent of the time for three-slice charts with a mean squared error of 0.1000, while the intensity and edge combination was selected 88.46 percent of the time for ten-slice charts with an MSE of 7.99, suggesting that edge information becomes more important as the number of slices grows.

The authors argue that SoS-ML addresses documented failures of opaque AI, citing the COMPAS recidivism algorithm’s racial bias, Google’s 2015 image recognition error, Tesla Autopilot crashes, unsafe recommendations from IBM Watson for Oncology, and the 2019 Apple credit card gender discrepancy, though these are illustrative case studies rather than tests of the framework itself. They also acknowledge limitations: the modular approach carries computational complexity that could limit scalability in resource-constrained environments, the full potential of human-agent interaction remains unexplored, including how biases in human feedback might influence the system, and additional evaluation metrics are needed to assess how well SoS-ML meets its social specifications. As a proof of concept on one tabular dataset and one image interpretation task, the work establishes feasibility rather than demonstrated performance in deployed high-stakes settings, and the authors present it as a blueprint for future responsible AI development rather than a finished solution.

Subject of Research: A modular multi-agent framework for explainable artificial intelligence integrating insights from cognitive science and linguistics

Article Title: Towards responsible AI: an implementable blueprint for integrating explainability and social-cognitive frameworks in AI systems

Article References: Shamsuddin, R., Tabrizi, H. B., & Gottimukkula, P. R. (2025). Towards responsible AI: an implementable blueprint for integrating explainability and social-cognitive frameworks in AI systems. AI Perspectives & Advances, 7(1), Article 1. https://doi.org/10.1186/s42467-024-00016-5

Image Credits: AI Generated

DOI: 10.1186/s42467-024-00016-5

Keywords: explainable AI, responsible AI, machine learning, multi-agent systems, interpretability, XAI, system of systems, cognitive science, Pima Indian Diabetes dataset, pie chart interpretation, post-hoc explanations, transparency