Karyomorphology, the study of the physical form and structure of chromosome sets, is one of the cheapest and most accessible tools in systematic botany. Researchers spread metaphase plates on a microscope slide, photograph them, and measure the length of the long arm (L) and short arm (S) of each chromosome. From these basic measurements they derive a suite of quantitative traits that describe the karyotype: the total haploid chromosome length (THL), which serves as a rough proxy for genome size; the mean centromeric asymmetry (MCA), which captures how unevenly genetic material is distributed between the two arms of a chromosome; the coefficient of variation of chromosome length (CVCL), which expresses differences in size between chromosomes; and the coefficient of variation of centromeric index (CVCI), which reflects how heterogeneous the centromere positions are across the complement. Together with chromosome number, these parameters underpin countless comparative studies of plant evolution and systematics.

But how reliable are these numbers when different people produce them? The question is far from new. As early as 1959, the cytogeneticist Sybenga distinguished two main sources of variation in karyomorphological data: biological variation in actual chromosome length, driven by differences in the degree of chromosome condensation, and artificial variation caused by inaccuracy of measurement. In 1971, Bentzer and colleagues explicitly tested whether the same measurements made by different people produce consistent data, and showed that they did not, especially when researchers relied on camera lucida drawings of metaphase plates. Since the early 2000s, however, a new era of chromosome measurement through image analysis software has transformed the field. Programs such as ImageJ and dedicated tools like MATO have made measurements far more accurate than the printed microphotographs and hand-drawn plates of the twentieth century. What remained unknown was whether, given the very same microphotograph and the very same software, different observers still introduce measurable biases.

The Pisa course, organized by the Group for Floristics, Systematics and Evolution of the Italian Botanical Society, provided the perfect setting to find out. Lorenzo Peruzzi of the University of Pisa and his colleagues distributed a single metaphase plate of the diploid angiosperm Santolina decumbens subsp. diversifolia, a member of the daisy family with 2n = 18 chromosomes, to all participants. The image had been assembled by pasting together several microphotographs taken at different focal planes, so that all eighteen chromosomes could be seen clearly in one picture. Each participant then independently measured the plate using the software MATO, without conferring with anyone else. Fifteen independent sets of measurements were collected in total.

The results were striking. When the team calculated the coefficient of variation of each karyomorphological trait across the fifteen observers, the highest values concerned CVCI, at 17.3 percent, and CVCL, at 13.4 percent. The lowest values were found for MCA, at 4.7 percent, and THL, at 9.4 percent. In other words, the parameters most sensitive to observer error were exactly those describing the variation of centromere position and chromosome length within the karyotype, while the grosser measures of total chromosome length and average asymmetry were comparatively more stable. For a field that routinely compares karyotypes across species and populations to infer evolutionary relationships, a 17 percent disagreement among trained observers measuring the identical image is not negligible.

The statistical analysis uncovered something even more interesting: the observer bias is selective, not random. The only highly significant correlation among the traits was a negative one between THL and MCA, with a Pearson correlation coefficient of -0.72. The team’s interpretation points towards a systematic tendency among observers to overestimate the length of the short arms of chromosomes. Such an overestimation would simultaneously inflate the total haploid chromosome length, because the sum of all chromosome arms grows, and deflate the mean centromeric asymmetry, because the difference between the proportions L/(L+S) and S/(L+S) shrinks as the two arms appear more equal. This mechanism neatly explains why the two parameters move in opposite directions across observers.

Why, then, does CVCI show the greatest variation of all, yet no significant correlation with THL? The answer lies in the mathematics of the centromeric index, which is calculated as S/(L+S). An overestimation of the short arm affects both the numerator and the denominator of this ratio. On one side, this cancels out any correlation between CVCI and THL; on the other, it injects a great deal of additional variation into the parameter, making it the most vulnerable to observer bias in the entire experiment. The authors suggest that these errors may stem from the different decisions observers make when locating the centromere, the constricted region that divides a chromosome into its two arms, a judgment call for which no standardized protocol has ever been formally proposed.

The study’s conclusions carry practical weight for anyone working in comparative cytogenetics. The parameters most sensitive to observer error are CVCI and CVCL, and to a lesser degree MCA and THL. The authors recommend special attention to recognizing and measuring the short arms of chromosomes correctly, since these are the main source of observer bias in cytogenetics. A homogeneous approach among observers, they argue, could help stabilize results, and the development of a standardized measurement protocol, which the field still lacks, would be a valuable step forward. Until such a protocol exists, researchers comparing karyotypes produced by different laboratories should treat small differences in centromere-related statistics with caution, since these may reflect the hands that made the measurements rather than the biology of the organisms.

Perhaps the most enduring lesson, however, is one that was already articulated more than fifty years ago. Bentzer and colleagues famously advised that, in the course of an investigation, all the measurements should be made by the same person. The Pisa experiment demonstrates that this motto fully applies to the era of image analysis software. Digital tools have eliminated the crudest errors of the camera lucida age, but the human eye and the human judgment of where a centromere lies remain sources of systematic disagreement. As karyomorphological data continue to feed into phylogenetic and taxonomic decisions across the plant sciences, the study serves as a reminder that even the most routine measurements deserve scrutiny, and that the observers behind the numbers are themselves a variable that science must account for.

Subject of Research: Observer bias in karyomorphological measurements using image analysis software

Article Title: Different observers introduce not negligible biases in comparative karyomorphological studies

Article References: Peruzzi, L., Franzoni, J., Tiburtini, M., Abidi, E., Alù, E., Barone, G., Bianchi, E., Cataudella, C., Di Iorio, E., Guerrina, M., Mondello, F., Paino, L., Pentassuglia, M., Porrovecchio, M., Rivieccio, G., Siccardi, E., Stinca, A., Tei, A., Volanti, V., & Giacò, A. (2024). Different observers introduce not negligible biases in comparative karyomorphological studies. Comparative Cytogenetics, 18, 175-182. https://doi.org/10.3897/compcytogen.18.135172

Image Credits: AI Generated

DOI: 10.3897/compcytogen.18.135172

Keywords: cytogenetics, karyomorphology, observer bias, chromosome measurement, image analysis, Santolina decumbens, karyotype asymmetry, centromeric index, plant systematics, measurement standardization, Different, observers

The core problem the framework addresses is familiar to anyone who has followed the AI governance debate: deployed language models are opaque, and the most sophisticated tools for interpreting them — mechanistic interpretability, feature attribution methods such as SHAP — are expensive, fragile, and in some cases provably limited. Impossibility theorems for feature attribution mean that no technique can reliably reveal why a model produced a given output in every case. Meanwhile, AI incidents continue to accumulate in databases cataloging real-world failures, and audits of large language models remain labor-intensive, point-in-time exercises rather than continuous monitoring. What is missing, the paper argues, is a measurement layer: a way to turn messy, free-form conversations between professionals and AI systems into structured, comparable data that institutions can actually monitor.

The proposed solution is a measurement standardization framework built around a grammar of eight interaction fields. Three of these are input–output fields — mission, conclusion, and justification — that capture what a professional asked the model, what the model recommended, and why. The paper illustrates how these fields transfer across domains: in a clinical setting, the mission might be recommending a diagnostic workup for a persistent cough in a heavy smoker, with the conclusion being urgent imaging justified by red-flag features for malignancy; in lending, the mission might be assessing a mortgage application with a low credit score, ending in a rejection justified by underwriting thresholds; in law, evaluating a settlement offer against comparable precedent awards. Because the fields carry the same structure regardless of domain, scores produced in one sector can in principle be compared with those in another.

The remaining fields feed a scoring engine. A large language model acting as a judge — the now well-established “LLM-as-a-judge” paradigm — evaluates each standardized interaction along dimensions such as risk level, policy alignment, and evidential alignment. Policy alignment measures whether the AI’s recommendation conforms to applicable guidelines and regulations; evidential alignment measures whether the justification is actually supported by the evidence cited. Crucially, the judge does not need access to the model’s internals. It works entirely from the observable interaction, which means the framework can be applied to any deployed system, including proprietary models whose weights are hidden even from the institutions using them.

Of course, using one black-box model to grade the outputs of another black-box model introduces what the paper candidly names structural circularity. LLM judges are known to suffer from systematic biases: sycophancy, the tendency to agree with whatever a user asserts; self-preference, the tendency to favor outputs resembling their own generations; and verbosity bias, the tendency to reward longer answers regardless of quality. There is also the problem of non-determinism — even supposedly deterministic settings can produce different judgments across runs, and small prompt changes can have butterfly-effect consequences for model performance. The framework confronts these weaknesses head-on rather than pretending they do not exist.

Its answer is a set of bounded conditions designed to reduce measurement inconsistency: explicit scoring rubrics, structured chain-of-thought reasoning before each verdict, reference documents that anchor judgments to authoritative guidelines, and low-temperature generation to suppress randomness. On top of these, the paper specifies a reliability verification procedure to detect and quantify the residual biases. The statistical machinery is equally explicit: paired bootstrap inference for the main comparisons, DeLong’s test for paired areas under the receiver-operating-characteristic curve as a sensitivity check, a pre-specified one-sided non-inferiority margin of 0.05, and Holm–Bonferroni correction to control for multiple testing. Agreement statistics drawn from the reliability literature, such as intraclass correlation coefficients and the Landis–Koch benchmarks for observer agreement, provide the yardsticks for judging whether the automated judge is consistent enough to be trusted.

To demonstrate feasibility, the paper applies the protocol in a minimal form to a published corpus of expert–AI interactions: the Rheum2Guide vignette study, in which ChatGPT’s treatment recommendations for rheumatic patients were compared against specialist decisions across nineteen clinical cases covering conditions from rheumatoid arthritis to giant cell arteritis. The judge scored each interaction for risk, policy alignment, and evidential alignment, using EULAR guideline documents as the reference corpus. The key finding is narrow but important: the judge reproduced its policy and evidential alignment scores across two independent runs under the specified conditions. In other words, the measurement instrument was stable. The author is careful to stress what this does not show — reliability at scale, across domains and models, remains a question for future empirical work, and the population-level claims the framework is designed to support are the subject of a staged research program, not results claimed in this paper.

If that research program succeeds, the payoff comes in three stages. The first is a reliability claim: under bounded conditions, language models can produce dependable, standardized assessments of expert–AI interactions. The second is a governance claim: alignment scores give experts an immediate signal during deployment — a warning light that flashes when a model’s advice drifts from policy or evidence — while giving institutions a basis for monitoring alignment patterns across mission types, models, and domains. The third, and most ambitious, is the outcome validation claim: once measurement is standardized, aggregate alignment scores could be correlated with downstream outcomes in regulated professional settings, exactly as cholesterol measurements were standardized and then linked to heart disease risk in the Framingham study, and as the lipid standardization programs of the CDC and National Heart, Lung and Blood Institute made population-scale cardiovascular epidemiology possible.

This is where the epidemiological analogy does its heaviest lifting. John Snow did not need to identify the cholera vibrio to map deaths around the Broad Street pump; Doll and Hill built the case against tobacco on correlated variables long before the molecular mechanisms of smoking-induced carcinogenesis were worked out. Hill’s famous 1965 criteria for distinguishing association from causation — strength, consistency, specificity, temporality, and the rest — were tools for reasoning rigorously in the absence of mechanistic certainty. The paper proposes importing precisely this style of reasoning into AI safety: instead of waiting for interpretability research to make model internals transparent, collect standardized measurements at the interface, look for statistical associations with harm, and intervene on the patterns that emerge. It is risk detection by surveillance rather than by dissection.

The framework is honest about its own limits. The demonstration involved a single judge, a single domain, and a small corpus; the circularity of black-box judging is mitigated, not eliminated; and unfaithful chain-of-thought reasoning — language models that do not always say what they think — remains a live concern even for structured scoring pipelines. But the conceptual move is significant. AI governance has long oscillated between two poles: demanding impossible transparency from opaque systems, or resigning itself to anecdotal incident reports. A standardized measurement layer offers a third path, one that medicine took a century to build and that could give institutions a shared language for describing, comparing, and ultimately predicting AI risk. Whether AI epidemiology can graduate from concept paper to working surveillance system will depend on the empirical studies the author has now carefully specified — but the blueprint for the experiment is on the table.

Subject of Research: A measurement standardization framework for detecting risks in deployed AI systems using epidemiological methods

Article Title: Toward AI epidemiology: a measurement standardization framework for prospective risk detection

Article References: Tempest-Walters, K. (2026). Toward AI epidemiology: a measurement standardization framework for prospective risk detection. AI & SOCIETY. https://doi.org/10.1007/s00146-026-03276-3

Image Credits: AI Generated

DOI: 10.1007/s00146-026-03276-3

Keywords: AI epidemiology, measurement standardization, LLM-as-a-judge, AI governance, risk detection, alignment, large language models, epidemiological methods, AI auditing, sycophancy, reliability, AI & Society