When AI Guesses and Computes Alike: New Framework Exposes Hidden Risk in Chatbot Answers

A spreadsheet and a chatbot were handed the same set of financial figures. The spreadsheet returned one answer; the conversational AI system returned another. When the researcher pressed the system to explain the discrepancy, it eventually admitted something unsettling: it had treated the numbers as language rather than as quantities to be computed, and had in effect been guessing. Nothing in its confident, fluent response had signaled that. That single episode, described by Pandi Sudarsono of Indonesia Open University in a new open forum article published in AI & Society on 21 September 2026, became the seed of a conceptual framework that argues the most dangerous feature of modern AI may not be what the systems do, but what users cannot see them doing.

Sudarsono names the condition operational opacity: the absence of any visible distinction, at the point of interaction, between AI outputs produced through probabilistic generation and those produced through reliable computation. When a chatbot calculates a sum, it may be running a genuine computational process, or it may be generating statistically plausible text that resembles a calculation. To the user staring at the interface, the two are indistinguishable. Both arrive in the same clean, authoritative prose. Both carry the same tone of competence. The difference between a verified computation and a sophisticated guess is structurally invisible, and that invisibility, the paper argues, is not a cosmetic flaw but a defining epistemic hazard of the AI era.

The framework is careful to distinguish operational opacity from two concepts already well established in the literature. The first is algorithmic opacity, the familiar problem that the internal workings of machine learning systems are inscrutable even to their designers. Sudarsono relocates the problem: operational opacity lives not inside the architecture but at the moment of interaction itself, on the surface where human meets machine. A perfectly explainable system could still be operationally opaque if its interface fails to signal which mode of processing produced a given answer. The second distinction is from automation bias, the documented human tendency to over-trust automated systems, studied extensively since Parasuraman and Manzey’s influential 2010 synthesis in Human Factors. Automation bias describes a failure of evaluation, a user accepting an output too readily. Operational opacity names a condition that precedes evaluation: the user cannot even begin to judge an output properly because nothing tells them what kind of process generated it.

What fills that vacuum, according to the paper, is a culturally embedded assumption Sudarsono calls the omnicompetence assumption: the belief, absorbed from decades of computing culture, that a machine labeled intelligent is competent across all domains. A calculator computes; a search engine retrieves; but a system that converses fluently about history, law, medicine, and arithmetic seems, to the untrained eye, to master everything at once. When this assumption meets an uninformative interface, the result is modality collapse, the inability to distinguish what kind of process generated a given response. And the downstream consequence of modality collapse is what the paper terms capability illusion: users systematically overestimate what the system is actually doing reliably, because the surface presentation of a guess and the surface presentation of a computation are identical.

The empirical backdrop for these concerns is growing. Survey research published in the Journal of Medical Internet Research in 2023 by Choudhury and Shamszare examined how user trust shapes the adoption and use of ChatGPT, and a 2026 study by Zhang, Tian, and Deng in Interacting with Computers explored the dimensionality and structure of user trust in the same system. Related work by Steyvers and colleagues, posted as a preprint in 2026, probes what the authors call AI-mediated metacognitive decoupling, suggesting that the usual calibration between confidence and competence may break down in distinctive ways when judgment is outsourced to AI. Trust, in other words, is flowing toward systems whose operational modes users cannot inspect, and the trust literature is only beginning to map what that asymmetry does to human reasoning.

Against this backdrop, Sudarsono constructs the Dual-Mode Awareness Framework, or DMAF, built around three analytically independent dimensions. The first is modality perceptibility: can the user perceive, from the interaction alone, whether a response was probabilistically generated or reliably computed? The second is verification effort: how much work does it take to check the output independently? Verifying a chatbot’s summary of a historical event might require minutes of archival digging; verifying its arithmetic requires a calculator and seconds. The third is stakes sensitivity: what happens if the answer is wrong? A hallucinated recipe costs a dinner; a hallucinated drug dosage or a miscomputed financial figure can cost far more. The framework’s central analytical claim is that epistemic risk is determined by the interaction of these three dimensions rather than by any one of them in isolation. A low-stakes exchange with high verification ease may tolerate near-total modality collapse; a high-stakes exchange with difficult verification and imperceptible modality is where the framework locates the greatest danger.

The normative backbone of the framework draws on two philosophical traditions chosen, as the paper explains, because each names a feature of AI-mediated exchange directly rather than by analogy. From Bernard Stiegler’s pharmacological account of technology, developed across works including Technics and Time and elaborated with Antoinette Rouvroy in their 2016 analysis of the digital regime of truth, DMAF takes the view that technology is always both poison and remedy: the same systems that erode epistemic independence could be redesigned to support it. From Jürgen Habermas’s theory of communicative action, first published in English translation in 1984, the framework borrows an ethics of communicative transparency, the idea that participants in genuine communication must be able to understand the conditions under which claims are being made. An interface that hides whether a claim is computed or generated, on this view, violates a basic precondition of rational exchange.

Notably, DMAF does not prescribe interface redesign. Sudarsono is explicit that the framework operates at the level of user awareness, offering a diagnosis rather than an engineering specification. But the diagnosis carries a pointed distributional claim: primary responsibility for operational opacity falls on those who design and deploy AI systems, not on the users who encounter them. The argument is that users cannot be expected to detect a distinction that the interaction surface is structurally engineered, or at least structurally neglected, into invisibility. At the same time, the paper shows why this responsibility distribution does not require the framework to dictate exactly how designers should discharge their obligation, preserving room for plural solutions ranging from labeling conventions to confidence signaling to mode indicators.

The article situates itself within a broader lineage of human-centered computing thought. It reaches back to Card, Moran, and Newell’s foundational 1983 work on the psychology of human-computer interaction, and engages Ben Shneiderman’s 2022 manifesto Human-Centered AI, which insists that automated and human-controlled modes should be designed to complement one another. It also dialogues with Tim Miller’s 2019 survey of explanation in artificial intelligence, the Partnership on AI’s 2021 workstream report on explainable AI in practice, and Langdon Winner’s classic 1980 question of whether artifacts have politics. By connecting these strands, Sudarsono positions operational opacity as a gap that the explainability movement, focused as it is on opening the black box, has largely overlooked: even a fully explained system can still fail to tell you, in the moment, whether today’s answer was computed or merely composed.

The stakes of the argument extend beyond interface design into what the paper’s subtitle calls epistemic sovereignty, the capacity of individuals and societies to govern the conditions under which they know things. If fluency is mistaken for computation at scale, in classrooms, newsrooms, clinics, and finance departments, the collective ability to distinguish verified knowledge from plausible generation erodes. DMAF’s contribution is to give that erosion a name, a structure, and a locus of responsibility. Whether designers respond with visible mode indicators, verification affordances, or entirely new interaction conventions remains an open question, but the framework makes one thing clear: the problem is not that users are careless with AI. It is that the interface, as currently constituted, gives them nothing careful to be careless with.

Subject of Research: A conceptual framework for operational opacity and epistemic risk in human-AI interaction

Article Title: The dual-mode awareness framework (DMAF): toward epistemic sovereignty in AI-mediated knowledge societies

Article References: Sudarsono, P. (2026). The dual-mode awareness framework (DMAF): toward epistemic sovereignty in AI-mediated knowledge societies. AI & SOCIETY. https://doi.org/10.1007/s00146-026-03376-0

Image Credits: AI Generated

DOI: 10.1007/s00146-026-03376-0

Keywords: operational opacity, Dual-Mode Awareness Framework, human-AI interaction, epistemic sovereignty, large language models, modality collapse, capability illusion, automation bias, philosophy of technology, explainable AI, epistemic responsibility, AI trust