Clinical Experience Decides How Reliably Doctors Label Brain Pressure Waveforms for AI

In the intensive care units where patients fight for survival after traumatic brain injury, stroke, or cardiac arrest, a quiet data crisis is unfolding. High-frequency recordings of intracranial pressure (ICP) and arterial blood pressure (ABP) promise to unlock the brain’s hidden dynamics, feeding machine learning models that could one day predict deterioration before it happens. But these signals are riddled with glitches: sensor flushes, patient movements, disconnections, and subtle distortions that can poison any algorithm trained on them. Now, a study published in Neurocritical Care has delivered one of the most rigorous audits yet of the human “ground truth” that artificial intelligence depends on, and the results reveal a striking truth: how reliably doctors label anomalies depends heavily on both the signal itself and the clinical experience of the person staring at the screen.

The research, led by Josef Škola and colleagues from Bulovka University Hospital, Charles University, and collaborating Czech institutions, set out to answer a deceptively simple question. When different clinicians examine the same stretch of physiological waveform, do they agree on what counts as an anomaly? The answer matters enormously because, despite rapid progress in automated signal cleaning, human annotations remain the de facto reference standard against which machine learning detectors are trained and judged. If the humans disagree, the algorithm inherits their confusion, and the ceiling of achievable model performance is set not by computing power but by the consistency of the labels themselves.

To measure that consistency, the team assembled a dataset of extraordinary granularity. From a single-center neurocritical care database of 39 patients with acute brain injury, they selected 100 hours of synchronized ICP and ABP recordings, slicing them into 32,760 ten-second segments. The ten-second window was no arbitrary choice: it matches the standard averaging interval used to compute the pressure reactivity index (PRx), a cornerstone measure of cerebrovascular autoregulation that is notoriously sensitive to waveform contamination. Six annotators spanning the full clinical spectrum, three senior neurocritical care consultants with more than a decade of experience, two junior residents, and one final-year medical student, independently labeled each segment using custom-built software that displayed the target segment within a minute of surrounding context, in fully randomized order.

The scale of the effort is worth pausing on. Each of the 32,760 segments was reviewed by all six annotators for both signals, yielding roughly 393,000 individual judgments. Annotators worked in isolation, unable to communicate about their progress or impressions, with every action logged to preserve procedural integrity. Crucially, the team adopted a deliberately broad definition of “anomaly” rather than the narrower, wildly variable “artifact” definitions that have fragmented the literature. An anomaly was any segment in which the signal departed from the expected physiological-technical pattern in a way that masked or distorted clinically relevant information, a framing borrowed from data science that the authors argue will generalize better across centers and event types.

The headline finding is reassuring at first glance. Anomalies were rare, just 3.99 percent of ICP segments and 5.83 percent of ABP segments, yet overall agreement was superb, with Gwet’s AC2, a statistic robust to rare events, exceeding 0.98 in every configuration. But the devil, as so often, lives in the details. When the researchers separated agreement on anomalies specifically, rather than agreement on the abundant normal segments, cracks appeared. For ICP waveforms, senior clinicians achieved substantially higher anomaly-specific agreement than the mixed-experience team: positive agreement rose by 12.8 percentage points, and Fleiss’ kappa by 13.5. For ABP, by contrast, experience barely mattered at all once the analysis accounted for the fact that segments from the same patient are statistically entangled.

That statistical subtlety deserves attention, because it is where much of the study’s methodological rigor lies. Ten-second segments are not independent observations; each of the 16 contributing patients supplied many consecutive segments, and anomalies cluster within individuals. Treating segments as independent would artificially inflate confidence. The team therefore reported every estimate twice, once at the segment level and once with patient-level cluster-robust confidence intervals derived from bootstrap resampling of whole patients, and anchored all inferential claims to the more conservative patient-level analysis. Under that stricter lens, the experience effect for ICP remained robustly significant, while most nominal differences for ABP dissolved.

Why would expertise matter so much more for intracranial pressure than for arterial blood pressure? The authors point to the nature of the distortions themselves. ABP anomalies tend to be loud and stereotyped: flushing artifacts, damped traces, sudden dropouts that even a medical student can spot instantly. ICP anomalies, by contrast, often hide in the fine structure of the pulsatile waveform, shifts in the relative heights of the P1 and P2 peaks, subtle changes in pulse morphology that blur the line between technical glitch and genuine physiological change in cerebral compliance. Reading those nuances fluently requires years of exposure to both the monitoring technology and the underlying physiology, exposure that routine clinical practice concentrates in specialized neurocritical care units.

The study also delivers a cautionary tale about statistics. Fleiss’ kappa, one of the most widely used reliability measures, suggested only moderate agreement (0.61 to 0.74) even as Gwet’s AC2 reported near-perfect consistency. This is the well-known kappa paradox: when the event of interest is rare, chance agreement inflates and kappa values sink, systematically understating reliability. No single metric tells the whole story, the authors argue, which is why they triangulated across Gwet’s AC2, Fleiss’ kappa, prevalence- and bias-adjusted kappa, positive and negative agreement, and the intraclass correlation coefficient. Only the combination reveals that the high overall agreement coexists with meaningful, modality-specific differences in how anomalies are actually identified.

The implications for the AI pipeline are direct and practical. Because human inconsistency injects noise into training labels, the reliability figures reported here effectively define the performance ceiling any detector trained on such data can reach. For ABP, that ceiling is high and reachable by mixed teams, meaning datasets can be expanded cheaply with junior annotators under light oversight. For ICP, the authors recommend expert consensus or multi-rater agreement to suppress label noise, higher confidence thresholds for ICP-based clinical predictions, and active-learning strategies that route genuinely ambiguous segments to senior review. They also note a subtle trade-off: senior annotators flagged fewer segments overall, suggesting stricter criteria that produce fewer false positives but may discard borderline events a broader pool would capture.

Limitations remain, and the authors are candid about them. The 100 hours of data, though vast in segment count, came from a single center with standardized equipment; the cohort was dominated by physiology within recommended target ranges, leaving extreme states underexplored; and the binary labeling scheme ignores anomaly severity and mechanism. Future work, they suggest, should purposively sample unstable physiology, dissect subtype-specific reliability, and pursue consensus methods such as Delphi studies to sharpen anomaly definitions. But the core message stands as a milestone for the field: reliable human annotation of brain and blood pressure waveforms is achievable at scale in routine neurocritical care, yet it is not a one-size-fits-all enterprise. As hospitals race to build AI on top of their monitoring streams, this study is a reminder that the most important component of any machine learning system may still be the trained human eye, and knowing exactly when that eye needs a decade of experience behind it.

Subject of Research: Interrater reliability of human anomaly annotation in intracranial pressure and arterial blood pressure waveforms for training machine learning models in neurocritical care

Article Title: Data for Machine Learning Models: Clinical Experience Determines Interrater Reliability in Waveform Anomaly Detection

Article References: Škola, J., Posel, Z., Waldauf, P., Falta, P., Soukup, M., Horáková, L., & Rožánek, M. (2026). Data for Machine Learning Models: Clinical Experience Determines Interrater Reliability in Waveform Anomaly Detection. Neurocritical Care. https://doi.org/10.1007/s12028-026-02655-4

Image Credits: AI Generated

DOI: 10.1007/s12028-026-02655-4

Keywords: neurocritical care, intracranial pressure, arterial blood pressure, waveform anomaly detection, interrater reliability, machine learning, artifact detection, clinical expertise, ground truth annotation, pressure reactivity index, acute brain injury, Gwet's AC2