Medical artificial intelligence is often presented as a story about algorithms: larger datasets, faster processors and increasingly sophisticated models that can detect disease, predict clinical deterioration or recommend treatments. But a systematic review of 330 research papers suggests that the success of medical AI may depend on a less celebrated component—the people expected to use it, trust it and live with its consequences. Across the studies examined, patient-related factors were rarely integrated systematically, were often limited to narrow measures of satisfaction or perceived benefit, and were usually assessed only after an AI system had already been designed and built. The findings indicate that technical performance alone cannot guarantee real-world medical benefit. An algorithm may achieve impressive accuracy in a laboratory or validation cohort, yet still fail to improve care if patients do not understand its role, clinicians cannot explain its recommendations, or people fear that automated decisions are unsafe, unfair or impossible to challenge.
The study, published in Nature Health, examined how medical AI research incorporated four broad categories of patient factors: perceived usability and feasibility, trust and safety, acceptance and perception, and adherence. These dimensions describe how patients understand AI-enabled care and how they respond to it in practice. Usability concerns whether a patient can navigate an AI-supported service or make sense of its outputs. Trust involves confidence in the technology, the clinicians using it and the institutions responsible for it. Safety includes not only whether an algorithm makes technically incorrect predictions, but also whether its use creates new risks, such as delayed treatment, inappropriate reassurance, privacy loss or unequal performance across patient groups. Acceptance and adherence capture whether patients are willing to engage with AI-supported care and follow recommendations generated through it. Together, these factors determine whether a technically capable system becomes a useful clinical intervention or remains an impressive but poorly adopted prototype.
The review found a striking imbalance in what researchers chose to measure. Patient satisfaction appeared in 70.6 per cent of the papers, while perceived benefits were reported in 69.4 per cent. These measures can provide valuable information, but they are often broad and subjective. A patient may report that an AI tool is convenient or beneficial without understanding how its prediction was produced, what data were used or what limitations apply. Satisfaction can also be shaped by short-term novelty, reduced waiting times or enthusiasm for new technology rather than by long-term safety and effectiveness. By contrast, foundational questions were much less common. Trust was examined in only 16.7 per cent of the studies, and safety in 10.9 per cent. The pattern suggests that research has concentrated on whether patients like an AI system, while giving less attention to whether they can rely on it, question it and remain protected when it fails.
This distinction is crucial because medical AI does not operate in a vacuum. Most systems transform patient information—such as medical images, laboratory results, electronic health records, physiological signals or responses to questionnaires—into a probability, classification or recommendation. The output may be statistically accurate at the population level but still uncertain for an individual. A diagnostic model, for example, might estimate the probability that a scan contains a tumour, while a risk-prediction system might calculate the likelihood of hospitalisation within a specified period. Such outputs are not self-explanatory facts; they are model-dependent estimates affected by the quality of the input data, the population on which the model was trained and the clinical setting in which it is deployed. If patients are not told how much uncertainty surrounds a recommendation, they may over-trust an automated result or reject it altogether. Both responses can undermine care.
The timing of patient involvement was even more uneven than the selection of factors. The review reported that 89.4 per cent of assessments across the AI lifecycle occurred during the validation phase, when a system is tested to determine whether it works under defined conditions. Only 3.9 per cent involved patient factors during design and development. The AI lifecycle generally includes problem definition and design, data collection and model development, clinical validation, deployment and subsequent monitoring. By the time validation begins, many fundamental choices have already been made: which clinical problem to target, whose data to include, what outcome to predict, how results will be displayed and which users will be responsible for acting on them. Late-stage feedback can identify usability problems, but it may be unable to repair assumptions embedded in the system from the start.
Early participation could expose problems that technical testing alone cannot detect. Patients may question whether a proposed AI application addresses a meaningful clinical need, whether the information it requires is proportionate or whether its benefits justify the risks. They may identify cultural, linguistic, economic or accessibility barriers that researchers have overlooked. A remote monitoring system that appears straightforward to developers may be unusable for people without reliable internet access, limited digital literacy or disabilities that affect interaction with a device. An automated risk score may also rely on variables that patients consider sensitive or inappropriate, particularly when they do not know how those variables influence the prediction. These are not merely communication issues to be fixed with a user manual. They can alter who benefits from the technology, who is excluded and how clinical decisions are made.
Trust, meanwhile, is not equivalent to believing that an algorithm is accurate. It is a relationship involving patients, clinicians, developers and healthcare institutions. Patients may trust an AI recommendation when a doctor explains its rationale, acknowledges its uncertainty and remains accountable for the final decision. They may distrust the same recommendation if it appears to replace professional judgment or if no one can explain how to contest an error. Trust can also be damaged by the wider history of medical data use, including concerns about surveillance, commercial exploitation and breaches of confidentiality. A system trained on large datasets must therefore be evaluated not only for predictive performance but also for governance: who controls the data, who audits the model, who is responsible when it fails and how patients are informed about automated processing.
Safety presents a similarly broad challenge. Conventional validation may compare an algorithm’s predictions with a reference standard, such as a diagnosis confirmed by specialists or a later clinical outcome. Yet safe deployment requires examining the entire chain from data collection to human action. A model can be accurate in isolation but unsafe if clinicians misinterpret its output, if alerts are so frequent that they are ignored, or if the system performs poorly after patient demographics, equipment or clinical practices change. Safety also requires attention to subgroup performance. If an algorithm has been developed predominantly using data from one population, its error rates may differ for patients of other ages, ethnic backgrounds, sexes, socioeconomic circumstances or disease profiles. Patients need meaningful information about these limitations, while institutions need procedures for detecting drift and withdrawing systems that no longer perform as intended.
The authors’ central message is not that medical AI should be abandoned, but that patient factors must move from the margins to the centre of its development. Future studies could combine technical metrics with structured measures of trust, perceived safety, comprehension, autonomy, accessibility and adherence, following patients over time rather than surveying them once after a trial. Researchers could involve patients in defining research questions, selecting outcomes, designing interfaces and interpreting failures. Clinical validation could assess not just whether a model predicts accurately, but whether its use changes decisions, improves outcomes and avoids harmful disparities in ordinary care. Continuous monitoring after deployment would be equally important because real-world populations and clinical workflows evolve. As AI systems become more visible in diagnosis, screening, monitoring and treatment, the most persuasive measure of progress may not be how powerful a model is on a benchmark, but whether patients can understand its role, participate in decisions and remain safe when the technology is wrong. The review makes clear that the future of medical AI will be shaped as much by human confidence and accountability as by computational performance.
