AI Raters Learn to Score Health Images Like Humans, Study Finds

A new study published in Scientific Reports has taken an unusual step in the world of health communication research: instead of asking human experts to judge whether medical images are biologically meaningful, the researchers trained their attention on multimodal artificial intelligence systems and asked whether these machines could serve as standardized, non-human raters. The work, led by a team that includes a high school student, a retired primary school teacher, a dentist, and a medical oncologist, introduces a structured measurement instrument called the Asclepius Biological Interpretability Scale and tests whether AI models can apply it consistently. The results offer a cautiously optimistic but rigorously caveated picture of what machines can and cannot yet do when asked to interpret health-related visual material.

The central problem the researchers set out to solve is one that has long troubled evaluation and assessment research in health communication. When scientists or clinicians create visual materials—illustrations, artworks, or diagrams intended to convey health information—there is often no standardized, construct-based way to judge whether those images actually communicate biological meaning. Judgments tend to be subjective, varying from one reviewer to the next, and the criteria behind them are rarely made explicit. The team grounded their solution in two established theoretical frameworks: the Biopsychosocial Model, which frames health as an interplay of biological, psychological, and social factors, and Visual Literacy Theory, which describes how viewers construct meaning from images. From these foundations, they built a 20-item scale designed to measure how biologically interpretable a visual stimulus is.

Developing the scale was not a solitary act of authorship but a structured consensus exercise. The researchers employed a two-stage modified Delphi process, a method widely used in instrument development to distill expert opinion into stable, agreed-upon items. In the first stage, a joint panel considered both the images and the candidate items together, allowing reviewers to see how proposed criteria behaved when applied to actual stimuli. In the second stage, an independent panel evaluated the items in isolation, without the images, to test whether each item stood on its own as a clear and meaningful criterion. This dual-panel design was intended to strip out items that only made sense in context and to retain those that could function as reliable, reusable measurement tools.

The final instrument produces scores across four dimensions. Three of these are biomedical subscales, capturing different facets of biological interpretability in health-related imagery, while the fourth, Non-Medical Interpretability, assesses meaning that lies outside the biomedical domain. With the scale in hand, the team turned to the question at the heart of the study: could multimodal AI systems—models that process both images and text—apply this framework in a reproducible way, functioning as standardized raters that do not fatigue, disagree idiosyncratically, or drift in their criteria?

To answer it, the researchers assembled a small but carefully controlled stimulus set: six artworks, four of them health-themed and two serving as controls. Three multimodal AI models and four researchers, who served as an internal human reference, evaluated the images using a standardized protocol. The protocol’s design reveals how seriously the team took the methodological pitfalls of AI assessment. Each model was accessed through independent researcher-operated accounts, memory-isolation procedures were used to prevent the systems from carrying information from one evaluation into the next, and assessments were repeated across two separate days. This allowed the team to measure not only whether different accounts produced similar scores, but whether the same system produced stable scores over time.

The statistical approach was deliberately matched to the limitations of the data. Because the image type variable spanned only six stimuli—four health-themed and two controls—the researchers summarized health-versus-control differences at the image level using Cohen’s d, a standardized measure of effect size, and treated human–AI differences descriptively at the rater level rather than claiming statistical significance. Reproducibility was assessed with intraclass correlation coefficients, or ICCs, which quantify how strongly scores agree across raters or occasions, supplemented by Spearman correlations. This restraint matters: with so few images, the study cannot deliver definitive population-level conclusions, and the authors are explicit that their findings are preliminary methodological evidence rather than established generalizations.

Within that constraint, the pattern of results is instructive. The AI-generated scores were descriptively higher for health-themed images than for control images on the three biomedical subscales, with image-level Cohen’s d values ranging from 0.91 to 1.38—effect sizes that, in a larger sample, would typically be considered large. On the Non-Medical Interpretability subscale, the difference was smaller, at d = 0.49. With only four health-themed and two control images, none of these differences reached statistical significance, with p values ranging from 0.19 to 0.60. In other words, the machines appeared to distinguish health-themed from control material in the expected direction, particularly on biomedical criteria, but the sample was too small to confirm that the distinction was real rather than chance.

Reproducibility told a more nuanced story. Across researcher-operated accounts, agreement was excellent for the three biomedical subscales, with ICCs between 0.90 and 0.97—figures that suggest the AI systems, when run under controlled conditions, applied the biomedical criteria with remarkable consistency. Agreement on the Non-Medical Interpretability subscale was moderate, at an ICC of 0.57, indicating that the machines were considerably less consistent in judging meaning outside the biomedical domain. Most strikingly, agreement across assessment days was poor, with ICCs ranging from just 0.04 to 0.46. Whatever produces this day-to-day instability—whether model updates, sampling variability in the models’ generative processes, or subtle differences in prompt handling—it represents a genuine obstacle to using AI raters as stable measurement instruments without further safeguards.

The comparison with human raters added another layer of complexity. The four researchers’ internal reference scores ran higher than the AI-generated scores on the biomedical subscales for health-themed images, with mean differences of 1.41 to 1.96 points, suggesting the humans saw more biological interpretability in the health-themed artworks than the machines did. On Non-Medical Interpretability, the pattern reversed: the human scores were lower than the AI scores by 2.02 to 2.19 points, meaning the machines detected more non-medical meaning than the human panel. This divergence is intriguing, because it hints that AI systems and trained human raters may weight different aspects of an image differently, and that calibration between human and machine judgment is not automatic. It also flags the Non-Medical Interpretability subscale as the weakest component of the instrument; the authors report that it showed substantial internal-consistency problems, meaning its items did not hang together as a coherent measure, and it requires further refinement.

The authors are careful about what their study does and does not establish. The findings, they conclude, provide preliminary methodological evidence that multimodal AI can reproducibly apply a structured interpretability framework to controlled visual stimuli, supporting the feasibility of using AI as standardized non-human raters within controlled methodological settings. But the caveats are substantial: day-to-day stability must be improved, the Non-Medical Interpretability subscale needs rework, the stimulus set was tiny and artificial, and the human reference was an internal panel rather than a diverse group of independent external raters. Broader validation using authentic health communication materials—real patient education graphics, public health campaigns, clinical illustrations—and independent external human raters remains necessary before AI raters could be trusted in applied settings. Even so, the study opens a provocative line of inquiry for a field that increasingly relies on visual media: if the criteria for biological interpretability can be made explicit enough for a machine to apply them consistently, then the machines may eventually become not just tools that make images, but disciplined judges of what those images actually communicate about health.

Subject of Research: Using multimodal artificial intelligence as standardized non-human raters to assess biological interpretability of health-related visual stimuli

Article Title: Operationalizing biological interpretability in health-related visual stimuli using multimodal artificial intelligence as standardized non-human raters

Article References: Operationalizing biological interpretability in health-related visual stimuli using multimodal artificial intelligence as standardized non-human raters. (n.d.). https://doi.org/10.1038/s41598-026-75309-2

Image Credits: AI Generated

DOI: 10.1038/s41598-026-75309-2

Keywords: artificial intelligence, multimodal AI, health communication, visual literacy, biopsychosocial model, psychometrics, Delphi method, intraclass correlation, medical humanities, image evaluation, Scientific Reports, non-human raters