Big Data Cannot Save Bad Sampling: Why Process Sensors Need Sampling Theory First

Modern industry has fallen in love with data. Pharmaceutical plants, breweries, mines, and even museums now bristle with spectroscopic sensors that pour out thousands of measurements every second, all in the name of Process Analytical Technology, or PAT, the framework that links real-time measurements to the qualities that actually matter in a product. The promise is seductive: with enough data, any process can be understood, controlled and optimized. But a new review published in Frontiers in Analytical Science argues that this promise rests on a dangerous illusion. No matter how sophisticated the sensors or how powerful the algorithms, if the sampling behind the data is flawed, the entire monitoring system is built on sand.

The study, led by Giulia Gorla of the University of the Basque Country together with Jon Ander Iturrioz, Kim Esbensen and Jose Manuel Amigo, takes aim at what the authors call the persistent misconception that data quantity alone ensures representativity. Their central message is blunt: advanced sensor technology and powerful data modelling cannot compensate for systematic measurement bias introduced at the moment of sampling. Data quality, they insist, originates upstream, before analysis, and the mathematical machinery of chemometrics can only extract information that is physically present in the measurements.

To make this case, the authors turn to the Theory of Sampling, a rigorous framework developed originally for mining and bulk materials by Pierre Gy and later expanded by Esbensen and colleagues. The theory classifies materials by their dimensionality, from zero-dimensional discrete objects like tablets to three-dimensional volumes such as reactors and silos, and identifies the specific errors that arise in each case. Constitutional heterogeneity reflects differences in composition between particles, distribution heterogeneity arises from segregation and clustering, and temporal heterogeneity captures fluctuations over time. In dynamic systems, flow-induced heterogeneity adds further complications through velocity gradients and varying residence times in vessels. Unless a sampling strategy explicitly counteracts these heterogeneities, the collected data will reflect the sampling geometry rather than the true process.

The authors demonstrate this with three striking datasets. The first involved a natural sodalite rock from Brazil examined with a portable Raman spectrometer. Fifteen spectra collected from different points on the rock’s two faces revealed dramatic differences: some spots showed pure sodalite signatures, while others carried clear evidence of calcite, a common inclusion. The lesson is that a single surface measurement can completely miss the three-dimensional character of a heterogeneous material, a situation directly analogous to industrial probes that interrogate only a thin layer of a powder bed or slurry.

The second dataset used short-wave infrared hyperspectral imaging of ground coffee deliberately adulterated with chicory or malt. When the components were left unmixed, principal component analysis readily detected both the adulterant and an added salt marker. Once the samples were homogenized, discrimination became far harder, partly because salt has no distinct near-infrared signal of its own and mainly registers through its water of hydration. Even more revealing, the calculated homogeneity curves depended strongly on which region of the image was analysed: the same mixed sample appeared markedly more inhomogeneous when a smaller region of interest was chosen. Field of view, the authors emphasize, is not a technical afterthought but a decisive factor in whether a measurement claims to represent the bulk.

The third dataset came from beer. Drawing on 377 near-infrared spectra from seven batches of Belgian Pale Ale spanning mashing, boiling, fermentation and carbonation, the team built multivariate statistical process control models. Even a sampling frequency of once per day was enough to track the general arc of fermentation, with the first principal component capturing time evolution. But two abnormal batches drifted differently, and detecting such anomalies early would demand higher temporal resolution. The work also exposed an uncomfortable truth: in the early brewing stages the variability was so large that no clear trends could be distinguished among batches, and the researchers could not tell whether this stemmed from sampling, composition, or genuine chemical information. Poorly planned sampling in timing, location or sensor choice distorts model performance and can fabricate misleading cause-and-effect relationships when transport lags delay the recorded signal.

Perhaps the most provocative part of the paper is its critique of fashionable chemometric remedies. The authors tested SMOTE, a widely used synthetic data augmentation technique, on the beer monitoring models. Synthetic spectra generated by interpolation between existing observations did inflate the training set, yet the geometric structure of the monitoring space remained essentially unchanged. The out-of-control samples still fell outside the same limits. The explanation is geometric: projection-based models are built from the covariance structure of real data, and interpolating between points adds no new directions of variance. Regions of the process that were never measured remain unobserved, and heterogeneity that never reached the sensor cannot be reconstructed algorithmically. Worse, artificial densification can give a false sense of coverage and inflate validation statistics without ever testing the model’s ability to extrapolate to genuinely new conditions.

The review is equally skeptical of Interactive Optimization Technologies, promoted recently as a lean alternative to traditional partial least squares calibration. By using spectra of pure raw materials to track blend homogeneity in pharmaceutical manufacturing, these methods slash the calibration burden, a real advantage when active ingredients are scarce or expensive. But the authors note that this is essentially a return to multiple linear regression on pure component spectra, an approach abandoned decades ago because real industrial and natural samples are never pure components. The verdict: useful in the pharmacopeia-driven world of pharmaceutical manufacturing, but no universal panacea.

On the quantitative side, the authors integrate sampling into the framework of measurement uncertainty, distinguishing the total sampling error, the total analytical error, and a newly proposed total data modelling error that captures preprocessing artefacts, overfitting and model misspecification. The hierarchy is stark: sampling error often dominates the analytical error by factors of five to ten, and in highly heterogeneous materials by up to twenty-five. The greatest gains in reliability therefore come from better sampling, not better algorithms. Practical tools exist to quantify this: replication experiments that duplicate the entire lot-to-aliquot pathway, heterogeneity tests, variograms and homogeneity curves, all codified in standards such as the Danish DS 3077.

The authors close with concrete recommendations. Sensor type, position, field of view and penetration depth should be characterized against the physics of the process, not chosen by convenience. Sampling frequency should match process dynamics, since oversampling merely adds redundancy while undersampling misses critical events. Sampling design should be embedded in Quality by Design workflows and evaluated during risk analysis, as regulators are increasingly expected to demand. Emerging technologies, from adaptive sampling that lets exploratory models decide where to measure next, to digital twins that simulate processes virtually, can help, but only when built on informative primary sensors that comply with the Theory of Sampling. In regulated sectors like pharmaceuticals, where multivariate models define the design space under ICH guidelines, sampling representativity is not best practice but a prerequisite. The paper’s conclusion is a warning worth repeating across every industry drowning in sensor data: analysis can refine what was measured, but it can never recover information that was never observed.

Subject of Research: The role of the Theory of Sampling in ensuring representative data acquisition for reliable Process Analytical Technology and chemometric process monitoring

Article Title: Getting “sampling” right for reliable process monitoring: main challenges and solutions

Article References: Gorla, G., Iturrioz, J. A., Esbensen, K. H., & Amigo, J. M. (2026). Getting “sampling” right for reliable process monitoring: main challenges and solutions. Frontiers in Analytical Science, 6, Article 1839552. https://doi.org/10.3389/frans.2026.1839552

Image Credits: AI Generated

DOI: 10.3389/frans.2026.1839552

Keywords: Process Analytical Technology, Theory of Sampling, chemometrics, representative sampling, near-infrared spectroscopy, Raman spectroscopy, hyperspectral imaging, data augmentation, measurement uncertainty, heterogeneity, process monitoring, Quality by Design