AI Needs to Know When Its Own Knowledge Has Gone Stale, Argues New Paper

Artificial intelligence systems that operate inside complex systems such as financial markets, power grids and public health networks need a capability that goes beyond learning and reasoning: the ability to recognise when their own understanding of the world has become unreliable. That is the central argument of an opinion article published on 23 October 2026 in PLOS Complex Systems by Alexiei Dingli of the Department of AI at the University of Malta. The article, which presents an argument rather than new experimental data, contends that as AI becomes embedded in environments it actively changes, the decisive question is no longer whether a model can learn effectively or reason logically, but whether it can tell when its assumptions have stopped holding.

The author builds the case from the observation that complex systems rarely fail because we know nothing about them. More often, failure arises because knowledge is incomplete, because the assumptions behind a model no longer hold, or because the system changes faster than the model can adjust. Financial markets, power grids, ecosystems, transportation networks and public health systems are all shaped by nonlinear interactions, feedback loops, emergent behaviour and rare events that can alter overall dynamics in unexpected ways. Their behaviour, the article argues, cannot be fully captured through fixed rules or historical data alone.

Complexity science already has a name for part of this problem: early warning signals. These are simple measures, such as a system becoming more erratic or slower to settle after a small disturbance, that tend to appear before a system tips into a different state and old descriptions stop working. The article proposes turning that same monitoring logic around and pointing it at the model rather than at the world. What matters, in this framing, is not whether the system is about to break down, but whether the model’s assumptions are.

The argument starts from neuro-symbolic AI, an approach the article describes as one of the most promising routes to trustworthy artificial intelligence. Neuro-symbolic systems combine the strengths of neural networks and symbolic reasoning. Neural networks are highly effective at finding patterns in noisy, high-dimensional data, while symbolic systems contribute explicit knowledge, logical structure and greater interpretability. The combination offers a balance between adaptability and transparency that purely data-driven approaches often struggle to achieve, which is why it has attracted attention as a foundation for AI that people can audit and trust.

Yet learning and reasoning alone, the author argues, do not guarantee adaptability in genuinely novel situations. Neural components can remain overconfident when faced with data that differ significantly from their training experience, while symbolic knowledge can gradually become outdated as circumstances change. The result is a failure mode that is easy to miss: a hybrid system may generate perfectly explainable decisions that are nonetheless based on assumptions no longer reflecting reality. In other words, a system may be able to explain its reasoning without realising that the reasoning itself has become flawed.

The problem is sharpened, according to the article, by the fact that complex adaptive systems are constantly changing, often in response to the models themselves. Markets react to forecasts. Traffic flows shift when navigation systems redirect drivers. Public health interventions influence behaviour, which then alters patterns of disease transmission. In each case the model is not simply observing the system from the outside; its predictions can affect the behaviour of the very system it is trying to understand. Once a model’s outputs are acted upon, the model helps to create the very patterns it is trying to measure, a phenomenon studied under the headings of performative prediction and strategic responses to deployed models. Two consequences follow, and together they form the core of the argument. First, a model cannot assume that any change it notices came from outside; if it does, it will treat the system’s reaction to its own advice as though it were something new in the world. Second, a layer that spots change and responds to it is itself acting on the system, and so becomes one more loop within it — a practical version of the question of what watches the watcher.

The proposed answer is neuro-symbolic AI with meta-cognitive capabilities. In cognitive science, meta-cognition refers to the capacity to reflect on one’s own thinking, recognise uncertainty and revise beliefs when new evidence challenges existing assumptions — an idea the article notes is not new, having been reviewed two decades ago. What has changed, the author argues, is the setting. Earlier work mostly concerned a system watching its own thinking, noticing when a plan had failed or when it was spending too long on a problem, in a world assumed to sit still outside it. For AI systems working inside complex systems today, meta-cognition would mean monitoring the reliability of predictions, detecting conflicts between learned representations and symbolic knowledge, identifying distributional shifts, and recognising when established rules no longer match the environment. The goal, as the article puts it, is not to create machines that always know the right answer, but to create systems that can recognise when they may no longer have one.

The stakes are highest in safety-critical settings. Rare and high-impact events often occur in parts of the problem space where historical data are limited and confidence estimates are weakest. Neuro-symbolic methods can improve robustness by embedding physical laws and operational constraints into learning algorithms, helping to reduce unsafe behaviour outside permissible operating conditions. Even so, the article argues, such systems need mechanisms for recognising situations that were never anticipated during design or training, and in many real-world applications recognising the limits of a model may be more valuable than achieving a small increase in predictive accuracy.

The article is candid about the costs and difficulties. Continuous monitoring, uncertainty estimation and consistency checking all require additional computational resources, and in real-time environments the resulting latency may itself create operational risks. There is also a deeper theoretical concern: if a meta-cognitive layer monitors the reasoning process, what monitors the meta-cognitive layer? Any attempt at recursive self-monitoring eventually confronts the problem of infinite regress. The author’s proposed escape is to keep the monitor deliberately simple — a handful of fixed measures and a set of agreed responses, rather than a second model trying to learn about the first. Such a monitor has a short and known list of ways it can fail and can be tested in advance against past cases where conditions changed, making its reliability a matter of testing rather than of adding yet another layer above it. The trade-off is stated plainly: a simple monitor will miss problems that a richer one would catch. Rather than pursuing an unrealistic form of perfect self-awareness, the article calls for bounded meta-cognition — practical mechanisms that identify specific classes of failure, estimate uncertainty, detect concept drift, and trigger appropriate responses such as model adaptation, human intervention or safe degradation.

To show what this asks for in practice, the article works through a short-term electricity demand forecast for a local network with many rooftop solar panels, where the forecast is used to switch loads automatically. Three things are watched. The first is whether the model’s errors are as large as it said they would be, measured over a rolling 24-hour period. The second is how often the network breaks one of the rules the model has been given, counted as the hours in which power flows backwards along a line where the rules assume it cannot. The third is how much of the incoming data looks unlike anything seen during training. Each measure has a clear trigger and a clear response: if errors run half as large again as predicted for two dispatch periods in a row, the reserve margin is widened and the system falls back to a cautious schedule; if a rule is broken, automatic switching stops on the line concerned and a human operator is called; if unfamiliar data persist for long enough, the model is retrained and the rules themselves are reviewed. Because switching loads changes demand, a randomly chosen group of lines is left out of the automatic scheme and used for comparison, so that genuine change can be told apart from the network simply reacting to the model. Nothing in the example is exotic, the author notes — the parts already exist, and what is missing is the expectation that they be named, reported and tested together.

The article closes with a challenge to how AI systems are evaluated. Current benchmarks tend to emphasise predictive accuracy, logical consistency and interpretability, and while all three remain important, they do not fully capture the challenges posed by complex adaptive systems. The author frames the request as a matter of what researchers report rather than a call for new machinery: four figures should be given alongside accuracy for any model meant to be used in a changing environment. These are how often the model declines to answer, together with how accurate it is on the cases where it does; how long it takes to notice a change, counted as the number of observations between a shift being introduced and the model flagging it; whether its confidence still means what it claims once conditions have moved, rather than only on data held back from training; and how much of the running cost is spent on watching. Where a model will act on the system it predicts, papers should also say whether any part of the deployment was left out of that action, since without such a comparison there is no way to tell what caused the changes being reported. Trustworthy AI, the article concludes, will depend not just on building more capable systems, but on building systems that recognise the boundaries of their own knowledge and notice when those boundaries have moved because of what the systems themselves have done — an advance that may prove more important than larger models, more data or more sophisticated algorithms alone.

Subject of Research: Meta-cognitive capabilities for AI systems operating inside complex adaptive systems

Article Title: Beyond learning and reasoning: Meta-cognition for AI working inside complex systems

Article References: Dingli, A. (2026). Beyond learning and reasoning: Meta-cognition for AI working inside complex systems. PLOS Complex Systems, 3(10), e0000122. https://doi.org/10.1371/journal.pcsy.0000122

Image Credits: AI Generated

DOI: 10.1371/journal.pcsy.0000122

Keywords: artificial intelligence, neuro-symbolic AI, meta-cognition, complex systems, early warning signals, performative prediction, concept drift, trustworthy AI, uncertainty estimation, safety-critical systems, electricity demand forecasting, model evaluation