When Clinical AI Outpaces Human Oversight

Human oversight has become the default answer to nearly every anxiety surrounding artificial intelligence in medicine. Regulators demand it, ethical frameworks invoke it, and hospital procurement documents routinely list “clinician in the loop” as a condition of safe deployment. Yet a new commentary in the Journal of Medical Systems argues that this widely celebrated safeguard is, in practice, often hollow—clinicians remain legally and morally responsible for AI-assisted decisions long after they have lost any realistic ability to review how those decisions were produced. The paper, authored by Jonas Ver Berne and Reinhilde Jacobs of KU Leuven and Karolinska Institutet, offers a conceptual distinction designed to expose and address this gap: the difference between cognitive and normative oversight.

The authors’ central claim is that oversight has been treated as a single, stable property of clinical AI systems, when in fact it is a bundle of quite different capacities. Cognitive oversight refers to a clinician’s ability to genuinely understand the basis of an AI output—what data the system saw, what reasoning path it followed, what evidence or priors shaped its recommendation, and what uncertainty attaches to the result. Normative oversight, by contrast, refers to the clinician’s ability to take responsibility for the decision: to endorse, modify, or reject the AI’s suggestion in light of the patient’s values, the clinical context, and professional standards. The commentary argues that as AI systems grow more capable, the first capacity quietly erodes while the second remains fully intact, producing a dangerous asymmetry in which accountability persists but comprehension does not.

This asymmetry, the authors suggest, is becoming sharper with each new generation of medical AI. The field has moved decisively beyond narrow, single-task classifiers toward multimodal systems that integrate imaging, text, genomics, and laboratory data into unified assessments. It has moved, too, toward agentic AI—systems that do not merely classify a single image but plan multi-step actions, query other tools, draft clinical documentation, and in some architectures coordinate with other AI agents in a pipeline. Each step along this trajectory increases the technical opacity of the system. A radiologist can at least interrogate the salience maps of a convolutional network flagging a lesion on a chest film; no comparable inspection is available for a large language model synthesizing a treatment plan across a thousand pages of electronic health records, or for an agentic system whose recommendation is the emergent product of a chain of intermediate decisions no single human ever reviewed.

The problem is compounded by what the commentary identifies as a structural feature of modern clinical work: time pressure. Even when explainability tools are provided, clinicians under real-world conditions rarely have the minutes—or hours—required to genuinely audit a complex AI output. Studies of point-of-care decision-making have long shown that clinicians answer most clinical questions through fast, heuristic routes rather than exhaustive evidence review. AI-assisted decisions fit neatly into those existing heuristics. The result is what human-factors researchers have described as rubber-stamping: the human signature is affixed, the human judgment is not genuinely exercised. Responsibility, in other words, is transferred without any corresponding transfer of understanding.

The authors ground their argument in the current regulatory and reporting landscape. The European Union’s Artificial Intelligence Act, Regulation (EU) 2024/1689, explicitly frames human oversight as a fundamental requirement for high-risk AI systems, including many medical applications, and the European Data Protection Supervisor has published analyses of what effective oversight of automated decision-making should entail. In the United States, the Food and Drug Administration’s January 2026 guidance on clinical decision support software grapples with when a clinician’s review of AI output is sufficient to keep a tool outside the device category altogether. Reporting guidelines such as DECIDE-AI for early-stage clinical evaluation of AI-driven decision support, TRIPOD-LLM for studies involving large language models, and the newer CHART statement for chatbot health advice all formalize how such systems should be evaluated and disclosed. Yet, the commentary contends, none of these frameworks adequately distinguishes the kind of oversight being required. A system can nominally satisfy “human in the loop” mandates while the human in that loop possesses no meaningful cognitive access to the machine’s reasoning.

The distinction the authors propose has concrete implications for how AI systems are evaluated before deployment. Current validation practices focus overwhelmingly on output accuracy—does the model achieve strong sensitivity and specificity on a benchmark dataset, does it match expert labels, does it improve outcomes in a trial. Cognitive oversight demands an additional axis of evaluation: can a qualified clinician, under realistic conditions, reconstruct the basis of the output well enough to catch errors the model is prone to? This is not the same as demanding that every clinician understand every weight in a neural network. It demands instead that systems present their reasoning at the right level of abstraction—the relevant findings, the relevant alternatives considered, the relevant uncertainty—so that a domain expert’s judgment can genuinely engage with it rather than merely ratify it.

Normative oversight, meanwhile, suggests a different set of design requirements. If clinicians are to take authentic responsibility for AI-assisted decisions, systems should be built to invite that responsibility: surfacing the values and tradeoffs embedded in a recommendation, making disagreements between AI output and clinical intuition explicit rather than smoothing them over, and leaving genuine room for the clinician to shape the decision rather than merely confirm it. The authors note that research on professional identity and trust in AI-based decision support shows that clinicians’ willingness to engage critically with these tools depends heavily on how the interaction is designed. Systems that present themselves as authoritative oracles invite passivity; systems that present themselves as consultable collaborators invite scrutiny. The difference is not cosmetic—it determines whether the human in the loop is an overseer or an audience.

The commentary also engages with a growing literature on AI-induced deskilling in medicine. A 2025 mixed-methods review compiled evidence that reliance on AI recommendations can erode the very skills clinicians need to evaluate those recommendations, a feedback loop with obvious safety implications. If cognitive oversight requires clinical expertise to be exercised, and routine reliance on AI allows that expertise to atrophy, then the gap between responsibility and comprehension is not static—it widens over time. Early-career clinicians trained in AI-saturated environments may never develop the baseline competence against which AI outputs could be checked. The authors suggest that preserving cognitive oversight is therefore not only an interface-design problem but an educational and professional-culture problem: training programs and institutions must deliberately cultivate the skills and habits that make meaningful review possible.

The timing of this argument is not incidental. Public interest in AI-enabled clinical decision support has surged, and multimodal, generative, and agentic systems are moving from demonstration projects into production clinical environments at remarkable speed. Commentaries in The Lancet and Lancet Digital Health have recently wrestled with the promises and risks of these systems, including the question of who is “really in the loop” in AI-assisted care. What the Journal of Medical Systems commentary adds is a conceptual tool precise enough to be operationalized. By separating cognitive from normative oversight, it gives developers, regulators, evaluators, and clinicians a shared vocabulary for naming what is often lost in AI deployment: not accountability, which regulators know how to demand, but comprehension, which is far harder to mandate and far easier to lose.

The authors do not argue that clinical AI should be rolled back, nor that oversight is an impossible ideal. Their point is diagnostic rather than obstructive: the safeguard everyone relies on has been specified too loosely, and the looseness becomes untenable precisely as the technology becomes more powerful. As systems grow cognitively ambitious—capable of judgments no individual clinician could reproduce from first principles—the honest question is no longer whether a human is in the loop, but what that human can actually see, understand, and answer for. Until evaluation frameworks, regulatory guidance, and system design confront that question directly, medicine may continue to accumulate AI systems that are genuinely useful and genuinely beyond meaningful oversight at the same time—a combination that, the commentary implies, no amount of formal accountability can make safe.