The concern is not hypothetical. AI systems learn from data, and when the data they scrape or are trained on reflect existing human stereotypes and healthcare disparities, the resulting models can form, copy, and amplify harmful patterns. In everyday consumer applications, biased AI is troubling enough, but in clinical settings the stakes rise sharply. An algorithm that performs well on average may quietly underperform for women, older patients, or members of ethnic minorities, and those failures can translate directly into missed diagnoses, delayed treatment, and worse outcomes. Because medical AI increasingly influences decisions about who gets scanned, who gets referred, and who gets treated, researchers argue that careful evaluation of potential demographic bias is not an optional refinement but a core safety requirement.

To understand how well the field is meeting that requirement, the Osaka Metropolitan University team conducted a scoping review, a form of systematic literature mapping designed to capture the breadth of research on a topic. Their goal was twofold: first, to identify studies that validated commercially available radiology AI products, and second, to document trends in how those studies reported results for demographic subgroups defined by sex, age, and ethnicity. Scoping reviews are particularly useful when a research landscape is sprawling and inconsistent, because they reveal not just what has been measured but what has been left out. In this case, what has been left out turned out to be the most important part of the safety picture.

The scale of the search was substantial. The team collected 545 studies covering 252 commercial AI products that included some reported demographic subgroup data. On its face, that number might suggest a healthy level of attention to demographics. But the picture changed dramatically when the researchers applied stricter criteria. Of those 545 studies, only 77, validating just 52 products, were found to include both demographic details and actual subgroup analysis results. In other words, fewer than one in seven of the studies they identified provided the kind of per-subgroup performance evidence needed to detect bias, and the validated products represented only a fraction of the commercial landscape.

The statistical dimension of the problem is where the findings become especially striking. The researchers applied the Wilson Confidence Interval formula, a standard method for calculating proportions and their uncertainty, to studies validating AI for tuberculosis detection. They found that 67 percent of the reported datasets were at risk of being underpowered for sex subgroup analysis. Underpowered means the dataset was too small to detect a meaningful difference in performance between subgroups even if one existed. An underpowered analysis can produce a reassuring result that is essentially meaningless, because the study simply lacked the statistical capacity to reveal a disparity. For clinicians relying on such studies, the absence of evidence of bias can be mistaken for evidence of the absence of bias.

Perhaps the most sobering conclusion is that this situation has not improved despite years of calls for better demographic reporting. The demand for improved reporting on sex, age, and ethnicity in medical AI has been voiced repeatedly by researchers, ethicists, and regulators, precisely because of the dangers that biased algorithms pose to marginalized groups. Yet the regression analysis used by the Osaka team to map trends across the collected studies revealed that performance reporting for demographic subgroups has not become more common over time. The field, in effect, has been accumulating validation studies without accumulating the evidence needed to confirm equitable performance, a pattern the authors describe as a systemic problem rather than a series of isolated oversights.

Dr. Walston characterized the landscape in stark terms. “This scoping review quantifies how fragmented the commercial validation landscape is, showing that reporting for both the demographics and per-subgroup performance is inadequate for estimating subgroup bias,” she stated. “This systemic problem requires effort from all stakeholders, from researchers to regulatory agencies, encouraging thorough reporting and commercial product validation to support physician and patient trust in medical AI products.” The statement frames the issue as one that no single actor can solve: journal reviewers, device manufacturers, hospital procurement teams, and government regulators all play a role in whether subgroup evidence is produced, demanded, and acted upon.

The fragmentation Dr. Walston describes has practical consequences for how medical AI reaches patients. When validation is scattered across many small studies, each examining a different product with different datasets and different reporting conventions, it becomes nearly impossible to compare products or to build a coherent picture of where bias risk concentrates. A hospital choosing between competing algorithms may find that one has published subgroup results and another has not, or that both report averages without revealing how performance varies by sex or ethnicity. Without standardized, transparent subgroup reporting, the market cannot reward fairness, and regulators cannot easily verify it. The result is a validation ecosystem that quantifies overall accuracy while leaving the distribution of that accuracy across patient groups largely unmeasured.

The study, published in European Radiology under the title describing the current state of demographic subgroup reporting for commercially available AI in radiology, arrives at a moment when health systems worldwide are scaling up their reliance on algorithmic tools. Tuberculosis detection, one of the applications examined in the power analysis, is a case in point: screening programs in high-burden countries increasingly depend on AI readers, and any systematic performance gap between demographic groups could quietly shape which communities receive timely diagnoses. The authors note that the findings expose the need for effective, transparent reporting to confirm the safe and unbiased performance of medical AI products across all patient subgroups, a requirement they argue underpins both clinical safety and public confidence.

The path forward, as the researchers frame it, involves collective accountability. Manufacturers would need to release demographic details and per-subgroup performance data for their products; researchers would need to design validation studies with sufficient statistical power to detect subgroup differences; and regulatory agencies would need to make such reporting a condition of approval rather than a voluntary courtesy. The authors declare a relationship with the Medical AI Promotion Institute, Inc., in their conflict of interest statement, underscoring how closely the commercial and academic sides of this field are intertwined. What the review makes clear is that trust in medical AI cannot be assumed from headline accuracy figures alone. Until subgroup evidence becomes routine, the possibility that these tools are quietly copying and amplifying healthcare bias remains an open question, one that patients, physicians, and regulators deserve to have answered with data rather than assurances.

Subject of Research: Demographic subgroup reporting in validation studies of commercially available radiology AI products

Article Title: Check for copycat bias in medical AI

Article References: Check for copycat bias in medical AI. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: medical AI, radiology, algorithmic bias, demographic subgroups, scoping review, Osaka Metropolitan University, European Radiology, tuberculosis detection, validation studies, health equity, regulatory reporting, statistical power