The problem centers on two small public corpora that recur as evaluation data for objective ABR detection: the PhysioNet earndb tone-pip set and the Mendeley click set. Both are freely available, both are small, and crucially, neither distributed expert annotations of response presence as part of the primary release. Every published detector evaluation on these corpora relied on clinician annotation that the authors commissioned or produced themselves. A reader of the public releases before 2025 therefore had no annotation to use. The one label that can be computed from the released files alone is an automatic proxy, and the natural proxy is the clinical two-replication reproducibility criterion: two independent sub-averages are recorded at each level, and a response is deemed present when they correlate above a threshold. The proxy is attractive because it is objective, cheap, and rooted in clinical practice—but its behavior as a research target had never been tested.

That test became possible when Liu and colleagues, reporting a multi-center comparison of nine deep models trained on 128,123 clinically annotated waveforms, published as supplementary material the audiologist annotations they had commissioned for both public corpora: one invited rater for earndb and six raters for the Mendeley set. A team led by Seyed-Ali Sadegh-Zadeh of the University of Staffordshire mapped those annotations back onto the original recordings, validating the mapping in five independent ways—including exhaustive level-offset grids from −40 to +20 dB, none of which matched as many records as the identity alignment, and direct confirmation from the annotating authors that the published level column is in peak-equivalent SPL rather than sensation level. On identical data, the team constructed the reproducibility surrogate and four detectors spanning the capacity range: the instrument statistic wnsfmp, logistic regression, gradient boosting, and a multilayer perceptron.

The first result is stark. The surrogate agreed with expert judgment at Cohen’s κ of just 0.167 on earndb and 0.248 on the Mendeley set—against a median κ of 0.691 among the six Mendeley audiologists themselves. Disagreement was strongly asymmetric: on earndb, the surrogate declared a response present where the expert declared it absent in 92 of 204 conditions, the reverse in only 5. The divergence concentrated almost entirely at 1 kHz, where the surrogate labeled 82 percent of conditions response-present while the expert found responses in only 16 percent. Whatever the two quantities register, they are not registering the same thing.

Why does the proxy fail so badly? The answer is structural rather than incidental. The expected correlation between two replicates is, in algebraic terms, SNR divided by one plus SNR—for whatever component the replicates share, response or not. Because any common component, including stimulus-locked residue or slow shared activity, raises the correlation, the surrogate can declare a response present in the complete absence of a brainstem response. A simulation of 2,000 response-free conditions reproduced the effect: at the clinical threshold, the surrogate called 53 percent of these response-free conditions present. On the recordings themselves, wnsfmp—a single-replicate signal-to-noise statistic read from the record header, involving no learning—predicted the surrogate with AUROC 0.789 restricted to the 250 sub-averages in which the expert saw no response. The proxy, in other words, largely leaks the input: it is a thresholded, noisy realization of the very quantity the instrument statistic estimates.

This degeneracy explains a mechanical rank reversal. Under the surrogate label, wnsfmp ranked first with a pooled AUROC of 0.827, ahead of all learned detectors. Under expert annotation on the same recordings, wnsfmp fell to last place at 0.662, while the multilayer perceptron rose from 0.713 to 0.898 and logistic regression from 0.752 to 0.837. The label-by-detector interaction for wnsfmp versus the perceptron was −0.350, resolved by an exact subject-level permutation test at p = 0.031. A ranking produced under the surrogate is therefore not evidence about detectors at all—it is evidence that the surrogate leaks the input, which the authors regard as the article’s central finding about the proxy.

The team then asked through which channel the label changes a study’s conclusion, using a 2 × 2 factorial over training label and evaluation label. The answer is chiefly through training. Detectors trained on the proxy and scored against expert judgment lost 0.139 to 0.304 pooled AUROC—for the perceptron, the loss was positive in every informative listener, with an exact p of 0.031. Within a listener and a carrier frequency, the two trainings ranked records about equally well; the cost arises in comparisons across listeners and frequencies, which is exactly what a fixed clinical decision threshold depends on. For the highest-capacity detector, the two labels defined nearly orthogonal prediction problems: cross-label AUROC values of 0.594 and 0.574 sit close to chance. Scoring fixed predictions against either label, by contrast, did not reorder the detectors at a resolvable magnitude.

The rater analysis extends the point to studies that do use expert annotation. Rerunning the full evaluation against each of the six Mendeley audiologists in turn moved AUROC by 0.065 to 0.082 within a detector—nearly as much as replacing one detector with another, and below the smallest difference the eight-listener design can resolve, which the team’s design analysis puts at 0.096 to 0.136. Resolving a true difference of 0.05 AUROC between two detectors at 80 percent power would require roughly 30 to 59 listeners; a difference of 0.03, the order of many reported gaps, would need 82 to 164. The authors are careful about what this does and does not establish: they claim neither label is correct. Expert visual judgment is itself a proxy for physiological response presence, with documented variability, and the single earndb annotator’s 1-kHz judgments are conservative relative to physiological expectation. The claim is conditional and, they argue, more useful: the choice of target determines the conclusion, the two available targets disagree profoundly, and a reader given only a ranking cannot tell which target produced it.

Two further results sharpen the practical message. Label-free baselines performed disturbingly well against the expert label: stimulus level alone reached an AUROC of 0.931 within 1 kHz, exceeding every waveform-based detector at that frequency, and carrier frequency alone reached 0.747 pooled. A detector evaluated on pooled data has not demonstrated waveform-derived information until it beats the stimulus parameters, which the operator knows without any analysis. Frequency-stratified analysis also exposed a pooling artifact: wnsfmp scored 0.734 within 1 kHz and 0.768 within 4 kHz yet only 0.662 pooled, below either stratum—an instance of Simpson’s paradox driven by the very different prevalence the expert label assigns to the two frequencies. About 37 percent of the headline interaction dissolves into this pooling structure, a property of label construction itself.

The findings echo a broader pattern in machine learning, where label errors in widely used test sets have been shown to change which model ranks first, and where benchmark ambiguity has been found to make progress partly track annotator conventions. What distinguishes the present case is that the substituted target is not a clinician’s judgment at all but an automatic statistic, that the statistic is a function of the model input, and that a published annotation now exists against which it can be checked. The team released the label mappings and a CPU-reproducible pipeline that regenerates every figure and reported number, and they recommend five inexpensive reporting practices: state label provenance and proxy validity, report the cost of proxy training where it can be measured, report performance against each annotator, report label-free baselines, and stratify by acquisition condition or report a covariate-adjusted AUROC. None is expensive; together, they distinguish a ranking that can be interpreted from one that cannot. On these corpora, the authors conclude, measured detector performance depends on how the label is constructed and who provided it, at magnitudes equal to or larger than the differences between detectors—meaning the small benchmarks are better suited to methodological study than to model comparisons.

Subject of Research: Label construction and rater identity in auditory brainstem response detector benchmarking

Article Title: Label construction can determine measured detector performance on small auditory brainstem response benchmarks

Article References: Sadegh-Zadeh, S.-A., Saadat, S., Barati, M. A., Sharvia, S., & Jarchi, D. (2026). Label construction can determine measured detector performance on small auditory brainstem response benchmarks. Frontiers in Audiology and Otology, 4, Article 1952229. https://doi.org/10.3389/fauot.2026.1952229

Image Credits: AI Generated

DOI: 10.3389/fauot.2026.1952229

Keywords: auditory brainstem response, benchmark validity, label quality, objective detection, inter-rater reliability, evaluation methodology, machine learning, audiology, AUROC, reproducibility, signal-to-noise ratio, Cohen's kappa