The study, led by Guillermo Gutierrez and colleagues, addresses a foundational problem in clinical measurement: the reference standards against which new devices are validated are themselves imperfect. The team developed EBLA—Evidence-Based Label Adjustment—an algorithm that treats disagreement between two measurement methods as evidence to be weighed rather than error to be assigned, using the joint patterns of agreement and disagreement to estimate which method is more likely at fault on a case-by-case basis.

“The fundamental problem is that when we validate a new monitor against a clinical standard, we assume the standard is correct,” the authors explain. “But clinical assessments are themselves measurements with their own error rates. When the two disagree, conventional analysis simply assigns the error to the new device, which may substantially overstate its true inaccuracy.”

The approach exploits a statistical property of paired imperfect measurements: the relationship between how often two methods agree and their individual error rates follows a predictable mathematical structure. By modeling this structure across a patient population, EBLA can estimate the latent error rates of both methods simultaneously, without requiring either to be treated as ground truth. The algorithm then uses these estimates to adjust the labels on individual cases where the methods diverge, producing a corrected reference standard against which the new device can be more fairly evaluated.

To test the approach, the team turned to a genuine clinical problem: assessing respiratory effort in patients on mechanical ventilators who cannot communicate. The clinical standard—the modified Borg scale, in which nurses rate a patient’s breathing distress on a zero-to-ten scale—is subjective and noisy. The new technology is a continuous, noninvasive monitor of inspiratory effort developed by the same group. In a validation dataset of 1,292 paired measurements from 121 ventilated patients, the two methods agreed only about 71 percent of the time, a level of disagreement typical of comparisons between subjective and continuous measures.

Applying EBLA to this dataset produced a striking result: after label adjustment, the estimated error rate of the continuous monitor fell substantially, while the apparent error rate of the clinical standard rose. The area under the receiver operating characteristic curve for the monitor’s ability to detect elevated respiratory effort improved from 0.79 against the raw clinical ratings to 0.92 against the adjusted labels. “The monitor was being penalized for disagreements that were, in many cases, the fault of the reference standard,” the authors report.

The team validated the algorithm’s logic through simulation, constructing synthetic datasets in which the true error rates of both methods were known by design. Across scenarios varying the prevalence of the target condition and the error rates of each method, EBLA recovered the true error rates with high fidelity and consistently outperformed conventional analyses that treated the clinical standard as correct. The simulations also clarified the method’s boundaries: it requires that the two methods’ errors be at least partially independent, and it performs best when agreement between the methods is neither near-perfect nor near-chance.

The implications extend to any clinical field where new monitoring technologies are judged against imperfect standards—sepsis scoring, pain assessment, delirium screening, and weaning readiness among them. In each case, conventional validation may systematically understate the value of continuous measurement simply because the continuous method detects episodes that intermittent clinical assessment misses. A method that can estimate the reliability of both sides of a comparison, rather than assuming one side is correct, offers a way to break that bias.

The authors caution that EBLA is a tool for specific situations, not a universal replacement for validation studies. It requires paired measurements on the same subjects, reasonable sample sizes, and attention to the independence assumptions underlying the model. When a true gold standard exists, it should be used. But in the many clinical domains where no such standard exists, the choice has traditionally been between accepting an imperfect reference or abandoning validation altogether. This work suggests a third option: treat the disagreement itself as data, and let the mathematics sort out who is likely wrong.

The statistical core of the EBLA approach rests on a simple but powerful observation: when two independently fallible instruments agree, the probability that both are simultaneously wrong is much lower than the probability that either is wrong alone. If each method carries an independent error rate, the expected contamination among concordant observations shrinks roughly with the square of that rate. For a per-method error rate of fifteen percent, for example, the expected error rate among agreeing pairs falls to approximately three percent—a fivefold reduction in label noise. This enrichment of correct labels is what allows the concordant subset to serve as a high-confidence training set, even though neither instrument can be treated as ground truth.

Once the concordant observations are identified, EBLA trains a committee of five deliberately diverse machine-learning classifiers: a random forest, a gradient-boosted tree ensemble, logistic regression, a support vector machine with a radial basis kernel, and k-nearest neighbors. The diversity is not incidental. Because the adjudication of disputed cases depends on the classifiers reaching conclusions independently, the authors deliberately excluded architecturally similar algorithms to minimize correlated errors. Each learner embodies a distinct inductive bias—linear separation, kernel-based mapping, bagged trees, boosting, and instance-based reasoning—so that any systematic weakness in one model is unlikely to be shared by the others. Discordant observations are then adjudicated by threshold voting, with majority rule requiring at least three of five votes, while stricter supermajority and unanimity thresholds leave unresolved cases designated as ambiguous and reported separately rather than forced into a decision.

Because the clinical dyspnea dataset lacks any independent ground truth, the team first stress-tested the method on the Wisconsin Breast Cancer Dataset, a benchmark of 569 cases with pathology-confirmed diagnoses. By randomly flipping class labels at controlled error rates—ranging from symmetric ten-percent noise in both sources to asymmetric combinations such as thirty-percent corruption in one label set and twenty in the other—the researchers could measure exactly how well EBLA recovered the true labels. The fifteen-percent symmetric scenario was designated the primary test case because it approximates both the roughly nineteen-percent discordance observed in the ventilator cohort and a plausible error range for behavioral observation scales. Notably, only the labels were perturbed; the underlying feature values remained intact, isolating the effect of label noise from measurement noise in the features themselves.

The validation protocol incorporated several safeguards against optimistic bias. Classifier performance was assessed with stratified five-fold cross-validation, preserving the proportion of positive and negative cases in each fold, and feature standardization was performed separately within each iteration using parameters estimated only from the training folds. This prevented information from the held-out data from leaking into preprocessing, a common source of inflated performance estimates in machine-learning studies of clinical data.

The authors are explicit about the conditions under which the method is appropriate. Errors between the two instruments must be at least partially independent; if both are influenced by a shared confounder—deep sedation, for instance, might simultaneously blunt behavioral signs of distress and alter the physiological signals—the concordant subset loses its purity and the entire adjudication logic weakens. The concordant subset must also be large enough to support reliable classifier training, with the authors suggesting at least eighty concordant observations in their experience, and informative features must be available that are independent of those used to define the original classification thresholds. Finally, the voting threshold should be specified in advance according to the relative clinical consequences of false-positive and false-negative classifications, rather than chosen after seeing the results.

These caveats matter because real-world measurement error is rarely as clean as the simulated random flips used in the benchmark tests. Systematic biases—an observer who consistently under-rates distress, a monitor whose calibration drifts in a particular direction—violate the independence assumption in ways that random corruption does not. The authors acknowledge this limitation directly, positioning EBLA as a tool for assessing the potential impact of reference-standard error on validation results rather than as a replacement for the primary analysis. The framework is best understood as a sensitivity analysis: it quantifies how much of the observed disagreement between a new measure and an established one can plausibly be attributed to imperfection in the reference itself.

The clinical motivation remains pressing. Dyspnea affects roughly half of mechanically ventilated patients and is associated with prolonged ventilation, increased mortality, and post-traumatic stress disorder, yet in patients who cannot communicate it can only be detected through behavioral observation. The Mechanical Ventilation Respiratory Distress Observation Scale, the most extensively validated instrument for this purpose, is applied intermittently and is susceptible to inter-rater variability—precisely the profile of an imperfect reference standard. A continuous estimate of inspiratory effort derived from routine ventilator airway pressure and flow signals offers the possibility of monitoring that does not depend on observer availability or interpretation, and methods like EBLA provide a principled way to evaluate such measures when the alternative comparator is itself noisy.

Subject of Research: Ensemble‐based label adjudication (EBLA): A method for validating continuous physiological measures against observer‐dependent reference standards

Article Title: Ensemble‐based label adjudication (EBLA): A method for validating continuous physiological measures against observer‐dependent reference standards

Article References: Gutierrez, G., & Papa, F. (2026). Ensemble‐based label adjudication ( EBLA ): A method for validating continuous physiological measures against observer‐dependent reference standards. Physiological Reports, 14(19), Article e71127. https://doi.org/10.14814/phy2.71127

Image Credits: AI Generated

DOI: 10.14814/phy2.71127

Keywords: diagnostics, clinical measurement, machine learning, respiratory monitoring, validation methods, medical devices, methodology, critical care, Ensemble, based, label, adjudication