AI Imaging Tools May Look Accurate on Single Photos Yet Fail at Measuring Change Over Time

Artificial intelligence systems that measure skin lesions, wounds, and tumors from photographs are usually judged by how well they perform on individual images. A new study argues that this yardstick is fundamentally inadequate for the task these tools are often actually built for: tracking how a lesion changes between two clinic visits. The research, published in Frontiers in Imaging, shows through controlled computational experiments that a system can appear highly accurate on single snapshots while producing badly distorted measurements of change, and that the gap between the two types of performance can be large enough to reverse which of two competing settings is considered better.

The mathematical reason is straightforward but easy to overlook. When change is estimated as the difference between a follow-up measurement and a baseline measurement, the error in that difference depends not only on how large the errors are at each visit but also on how the two errors relate to each other. Errors that persist across visits tend to cancel when the measurements are subtracted, while errors that vary between visits can manufacture apparent change out of nothing or mask real change. The study formalizes this with a variance identity: the spread of paired-change error equals the sum of the visit-specific error variances minus twice their covariance. Because single-image evaluation captures the variances but never the covariance, no amount of single-snapshot testing can reveal how a system will behave longitudinally.

To demonstrate that this distinction matters in practice rather than only in theory, the researchers, Hunter Colson of the University of Kentucky College of Medicine and Manuel Valdebran of the Medical University of South Carolina, built two synthetic lesion-area experiments with exactly known geometry. Synthetic targets were chosen deliberately: because the true area and true interval change are known precisely, any error in the measurement can be attributed to the measurement procedure itself rather than to uncertainty in the reference. The experiments do not rank specific commercial models or establish clinical performance, but they isolate the measurement behavior with unusual clarity.

The first experiment, called the repeated-camera experiment, imaged 72 synthetic lesions at two visits, with 36 lesions unchanged and 36 growing by 10 percent in area. The team crossed multiple acquisition and processing choices, including camera distance, image width, three sources of three-dimensional surface geometry, three sources of lesion boundary masks, and six calibration-marker arrangements, yielding 960 configurations per arm that included at least one trained AI model. Geometry came from either the renderer’s known surface or from monocular depth models MoGe-2 or Depth Pro; boundaries came from the known mask or from SAM ViT-B or MedSAM, which were given a box around the already-located lesion. Eight coded markers of known size provided the scale reference.

The results were striking. Because camera pose, distance, and lighting were repeated between visits, errors at the two visits tended to be similar and largely canceled on subtraction. Assuming independent errors, the conventional shortcut, overestimated the paired-change error standard deviation by a median factor of 6.4 across the 960 no-change configurations, with ratios ranging from 1.8 to 143.5. In one worked example using MoGe-2 geometry and SAM boundaries at a 25-centimeter distance, the cross-visit error correlation reached 0.987, and ignoring the covariance would have inflated the estimated change-error spread by a factor of 8.2. No single conversion factor could translate single-image error into change error, since the ratio varied enormously across configurations.

Rankings were also unreliable. Pooled single-image error and paired-change error were correlated, with a Spearman coefficient of 0.724 in the no-change arm, yet the configuration with the lower single-image error had the higher change error in 23.2 percent of comparisons with unequal single-image values. The reversal rate remained substantial even when the single-image advantage was clear: 23.8 percent for a 1.5-to-2-fold advantage and 13.8 percent for a 2-to-3-fold advantage. When single-image errors were nearly tied, rankings were essentially unstable, flipping in 46.3 percent of comparisons and rarely surviving bootstrap resampling of the 36 lesions. Choosing a measurement protocol on single-image performance alone, the study suggests, is a gamble.

The second experiment tested what happens when the camera and lighting are not repeated between visits, a condition far closer to real clinical follow-up. For the same 36 unchanged lesions, the researchers compared a matched follow-up image, captured with identical camera pose, distance, and lighting, against a resampled follow-up in which viewing direction varied up to 8 degrees, distance varied uniformly between 15 and 25 centimeters, and light energy varied by up to about 20 percent in either direction. Varying the acquisition increased paired-change error in all 48 configurations using a trained AI model, by a median factor of 6.2, while single-image error barely moved. The median cross-visit error correlation fell from 0.993 to 0.752, showing that the comfortable error cancellation seen under laboratory conditions evaporates as soon as imaging conditions vary.

The study also elevates a frequently ignored metric: pair availability, the proportion of scheduled cases that yield valid measurements at both visits. In the repeated-camera experiment, both measurements were obtained for 95.3 percent of 8,640 scheduled pairs, but all 423 final failures occurred at the marker-detection stage, and because nine surface-and-boundary combinations reused each detection, a single failed detection silently produced 3,807 missing measurement rows. In the worst configuration, with recreated markers, a 25-centimeter distance, 512-pixel images, and clustered calibration markers, only 61.1 percent of lesions yielded complete pairs. A protocol can therefore look accurate on the cases it completes while quietly failing on the difficult ones, a pattern the authors argue must be reported separately from error among successful measurements.

Building on quantitative imaging biomarker methodology, including the Quantitative Imaging Biomarkers Alliance’s distinction between cross-sectional and longitudinal performance, the authors propose a reporting framework built on three questions: how accurately does the complete procedure measure change, how often does it return both required measurements and where do failures occur, and under which acquisition conditions and software versions were these properties established. The framework specifies that claims should name the model checkpoint, prompts, preprocessing, calibration, postprocessing, and execution environment, and that substantial software updates should trigger repeat testing, since changes to any of these steps can alter the relationship between errors across visits even when single-image performance improves.

The authors are careful about scope. The numerical findings depend on synthetic targets, controlled acquisition ranges, and segmentation models that were handed the lesion location, so they cannot be read as clinical failure rates or as evidence about any particular product. Recovering known changes in synthetic lesions establishes technical performance, not clinical responsiveness or patient benefit, which would require validation with physical targets, then human images, and finally demonstration of clinical utility. Still, the core message is hard to dismiss: for any AI system intended to measure change over time, single-image accuracy is not just insufficient evidence, it may point in the wrong direction. Direct evaluation of paired-change error, pair availability, and the full measurement protocol, the study concludes, is the only sound basis for a longitudinal performance claim.

Subject of Research: A paired-error framework for evaluating longitudinal technical performance of AI-enabled quantitative imaging

Article Title: Single-image performance alone does not establish longitudinal technical performance: a paired-error framework for artificial intelligence-enabled quantitative imaging

Article References: Single-image performance alone does not establish longitudinal technical performance: a paired-error framework for artificial intelligence-enabled quantitative imaging. (n.d.). https://doi.org/10.3389/fimag.2026.1976334

Image Credits: AI Generated

DOI: 10.3389/fimag.2026.1976334

Keywords: artificial intelligence, quantitative imaging, longitudinal measurement, paired-change error, pair availability, lesion measurement, measurement error, QIBA, image segmentation, technical validation, medical imaging, repeatability