Python Library Audits Raman Spectroscopy Models by Resolving Regional Spectral Disagreements

Biomedical applications of Raman spectroscopy increasingly depend on machine learning models that are trained on reference spectral libraries and then deployed for extended operational periods in diagnostic and analytical settings.

The newly released software package, designated pypasi for pythonic process-aware spectral inference, reinterprets automated classification by evaluating how consistently distinct wavenumber regions support a given taxonomic or diagnostic assignment. Rather than treating an entire Raman spectrum as an undifferentiated vector of light intensities, the library partitions the vibrational spectrum into discrete, chemically interpretable intervals and tracks how local evidence evolves across individual samples and multi-sample batches. While classic multivariate tools can flag overt statistical outliers, they typically struggle to identify localized distortions that leave global summary statistics largely undisturbed. By introducing an explicit negotiation mechanism among spectral sub-bands, the framework determines whether disparate regions of a vibrational fingerprint genuinely concur or whether an ostensibly confident diagnostic prediction rests on sharp, unacknowledged internal contradictions.

At the architectural core of the library are six interconnected programmatic objects designed to integrate directly with the standard scientific Python ecosystem, including NumPy, SciPy, pandas, and scikit-learn. The workflow begins with the BandSet class, which segments the Raman-shift axis into contiguous spectral bands using equal-width bins, peak-informed intervals, or user-defined boundary edges. Classification can then proceed through the BandNegotiationClassifier, an ensemble object that fits a dedicated sub-model to each individual band and allows them to exchange class evidence across a configurable interaction graph. These graphs can take the form of simple nearest-neighbor chains, k-nearest band topologies, or fully connected networks where every band communicates directly with every other interval. As the models exchange evidence through progressive iterative steps toward a shared consensus, anomalous bands that persistently diverge from their neighbors are systematically flagged, quantified, and governed by selectable mitigation regimes.

To prevent the disruption of already certified clinical pipelines, the library provides an ExternalAudit module that wraps an existing, previously trained host model without altering any of its underlying mathematical predictions. In this wrapping workflow, the primary diagnostic classifier continues to output its certified assignments, while independently parameterized band-level estimators monitor the incoming data stream in parallel to generate diagnostic audit descriptors. Complementing this local audit are the NoveltyDetector, which computes calibrated Hotelling’s T-squared and Q-residual statistics on matched data splits, and a ControlProfile module that tracks batch-level stability across time. To translate these multidimensional measurements into actionable operational decisions, a Triage engine integrates the resulting conflict metrics with classical novelty indices. The system categorizes incoming spectra into four unambiguous procedural actions: accept the prediction directly, route the acquisition for human review, remeasure the biological specimen immediately, or reject the measurement outright as an invalid acquisition.

The mathematical dynamics of the regional negotiation process yield specific numerical descriptors that quantify the exact nature of spectral disagreement before an analytical decision is reached. A key diagnostic index is the total distance traveled by the band evidence vectors during consensus negotiation, a metric termed DG, which reflects the overall magnitude of reconciliation demanded by an ambiguous or degraded spectrum. A companion metric, designated eREDG, quantifies the specific fraction of that communicative distance covered during the earliest iterative cycles, effectively distinguishing between harmonious classifications reached swiftly and contentious determinations produced only after prolonged, forced negotiation. Because batch monitoring in operational facilities involves testing numerous frequency intervals and statistical parameters simultaneously, the ControlProfile module decouples routine within-batch acquisition noise from genuine systemic inter-batch variations. The software dynamically widens its statistical control boundaries to control the family-wise false-alarm rate, ensuring that analytical laboratories are not overwhelmed by spurious quality alerts during routine operations.

Empirical validation of the software framework was conducted using a benchmark reference dataset comprising clinical bacterial isolates originally documented by Ho and colleagues. The 2018 baseline cohort consisted of twenty-five unique clinical isolates providing four hundred spectra each, yielding ten thousand individual vibrational profiles, while the subsequent 2019 cohort contributed twenty-five additional isolates yielding twenty-five hundred spectra. Crucially, the 2019 data were acquired following a documented loss of optical efficiency in the instrument setup, which the original investigators had sought to counterbalance by doubling the detector exposure time. To rigorously evaluate the software under controlled physical stressors, the researcher introduced five realistic instrumental degradations into the 2018 validation split, including continuous wavenumber shifts, resolution broadening via Gaussian convolution, throughput attenuation, local contamination, and varying levels of photon shot noise.

Under these challenging synthetic conditions, where planted instrumental defects lowered overall predictive accuracy by only four point four percentage points, the audit metrics demonstrated robust diagnostic discrimination between corrupted and pristine spectral measurements. The regional conflict score achieved an Area Under the Receiver Operating Characteristic Curve of zero point seven seven seven across held-out isolate-disjoint test splits, significantly outperforming conventional uncertainty metrics, which registered an area under the curve of zero point four five four, and standard novelty detection, which reached zero point four four six. When evaluating specific defect families, regional conflict proved exceptionally sensitive to localized distortions, registering an area under the curve of zero point nine two four against band-specific contamination and zero point nine zero eight against smooth throughput dips centered near ten eighty-seven inverse centimeters. Global resolution loss and instrumental axis shifts were similarly detected with high fidelity, achieving area under the curve metrics of zero point eight zero four and zero point seven five seven, respectively.

The empirical findings revealed clear operational boundaries regarding which physical failure modes are captured by regional conflict analysis and which require supplementary analytical tooling. Because independent measurement noise averages out within broad spectral intervals, minor shot-noise fluctuations induced by reduced exposure times yielded an audit detection score of only zero point four eight five, effectively matching random chance. Even when simulated noise was increased eightfold, causing a substantial ten point six percentage point decline in classification accuracy, the conflict score rose only modestly to zero point five six. Consequently, the authors incorporated dedicated noise estimation functions, such as Savitzky-Golay residual screening, demonstrating that acquisition auditing requires pairing inter-band negotiation diagnostics with direct sensor-level signal-to-noise assessments.

The study also illustrated that spectral novelty and regional conflict measure fundamentally distinct behavioral phenomena within chemometric feature space, exhibiting only a moderate rank correlation of Spearman rho zero point three nine eight. Conventional novelty algorithms evaluate whether an overall acquisition falls within the geometric envelope of the training distribution, whereas conflict assesses whether disparate wavenumber bands support a unified biological identity. In held-out evaluations, conflict successfully detected incorrect model predictions with an area under the curve of zero point six four eight across the complete cohort and maintained an area under the curve of zero point five nine four among spectra formally classified as normal by the novelty detector. When combined with baseline model posterior probabilities, adding conflict metrics enhanced degraded spectrum detection by twenty-two point four area under the curve percentage points, while integrating both conflict and novelty improved identification by twenty-eight point six percentage points.

Beyond issuing scalar diagnostic warnings, the band-resolved framework successfully localized the specific physical regions responsible for spectral anomalies, providing clear guidance for instrument recalibration. When subjected to forty-two controlled perturbation trials across seven equal-width spectral intervals spanning four hundred to seventeen hundred ninety-two inverse centimeters, the system correctly identified the perturbed interval in sixty-two percent of instances, compared with a theoretical random baseline of fourteen percent. Furthermore, an evaluation of five alternative base estimators established that partial least squares discriminant analysis produced the most informative regional conflict scores, registering an area under the curve of zero point seven seven seven against degradation and improving error detection by nine point seven percentage points, even though linear support vector machines achieved marginally higher raw classification accuracy. Benchmarking on consumer-grade workstation hardware confirmed that the implementation remains computationally lightweight, requiring zero point three two seconds to audit one thousand spectra while consuming under sixty-five megabytes of memory.

Subject of Research: Band-resolved classifier auditing and quality control for biomedical Raman spectroscopy

Article Title: pypasi: A python library for band-resolved classifier auditing and acquisition quality control in biomedical Raman spectroscopy

Article References: Kourkoumelis, N. (2026). pypasi: A python library for band-resolved classifier auditing and acquisition quality control in biomedical Raman spectroscopy. SoftwareX, 36, Article 103230. https://doi.org/10.1016/j.softx.2026.103230

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103230

Keywords: Raman spectroscopy, pypasi, spectral inference, chemometrics, machine learning, scikit-learn, quality control, classifier auditing, biomedical optics, instrument drift, spectral degradation, partial least squares, bacterial identification, Hotelling T-squared, novelty detection, batch monitoring, process analytical technology, diagnostic algorithms