Same-Condition Audit Reveals CPU-Only Machine-Learning Efficiency Numbers Are Far Less Stable Than They Look

…

…

The distinction between computational determinism and temporal stability deserves closer examination, because it explains why the audit’s findings should not surprise careful readers yet remain widely ignored in practice. When a numerical routine produces identical outputs across repeated invocations, that consistency reflects the arithmetic of the underlying operations, not the behavior of the hardware and software layers that schedule, cache, and interrupt those operations. A matrix multiplication executed twice on the same CPU may return bit-identical results while taking measurably different amounts of wall-clock time, because the second execution encounters a different state of the instruction cache, branch predictors, memory prefetchers, and operating-system scheduler queues. The audit’s central insight is that this temporal variability is not noise to be eliminated but a property of the measurement process that must be characterized and reported alongside any efficiency claim.

The choice of a 10% relative-difference threshold in the audit reflects a pragmatic middle ground in performance reporting. Smaller thresholds, such as 5%, would demand even more observations per group under the observed variability, potentially pushing sample-size requirements well beyond what typical benchmarking studies can afford. Larger thresholds, such as 20%, would be more forgiving but would risk masking genuine efficiency differences that matter in deployment decisions, particularly when aggregated across many inference calls or training steps. The 10% level also aligns with informal conventions in systems literature where differences below roughly a tenth are often treated as indistinguishable without strong statistical support. By reporting exceedance rates at this threshold, the audit provides a concrete, interpretable quantity: roughly three out of four same-condition comparisons exceeded it when only a single timing observation was used, which means a practitioner reporting one number has a high probability of presenting a difference that the measurement process itself could have manufactured.

The reduction in exceedance rates when moving from a single observation to the mean of ten observations, from the high seventies to the high sixties and low seventies across the five representative single-thread scenarios, illustrates a fundamental statistical principle with practical consequences. Averaging reduces the variance of the estimate by approximately the number of independent observations, so the standard error of a ten-observation mean is roughly one-third that of a single observation. Yet the audit shows that even this tenfold increase in measurement effort does not bring exceedance rates below two-thirds in most scenarios. This persistence occurs because the underlying timing distributions in CPU-only workloads are not always well-behaved: they can exhibit heavy tails from occasional operating-system preemptions, cache evictions from background processes, and frequency scaling transitions. When distributions are heavy-tailed, the mean itself becomes an unstable estimator, and the effective reduction in variability from averaging is smaller than the ideal statistical prediction. This observation reinforces the paper’s argument that sample-size planning must be grounded in the empirically observed variability of the specific workload and context rather than in generic assumptions about measurement noise.

The estimated requirement of 81 to 110 timing observations per group to support a 10% target relative difference carries significant implications for how CPU-only efficiency studies should be designed and budgeted. In many published comparisons, particularly those conducted in resource-constrained settings such as shared cloud allocations or personal machines with competing workloads, the number of repeated measurements per condition is far smaller, often between three and ten. The audit’s sample-size estimates suggest that such studies operate in a regime where the measurement process itself produces differences exceeding practical thresholds in the majority of same-condition comparisons. This does not mean that every reported difference in such studies is spurious, but it does mean that the evidence supporting those differences is substantially weaker than the precision of the reported numbers would suggest. A study reporting a difference to two decimal places based on five observations is presenting a level of precision that the measurement design cannot justify.

The structure of the audit as an A/A comparison, in which the same documented condition is measured twice and compared against itself, deserves particular attention because it inverts the usual logic of benchmarking. Traditional benchmark comparisons ask whether condition A is faster than condition B, and statistical tests are applied to detect differences between them. The A/A approach instead asks how often the measurement process, applied to identical conditions, produces differences that would be interpreted as meaningful if they had appeared between two genuinely different conditions. This framing connects to practices established in online controlled experimentation, where A/A tests are used to validate that the experimental apparatus does not manufacture spurious effects. In the context of CPU-only efficiency measurement, the A/A audit serves as a calibration of the measurement instrument itself: if the instrument produces threshold-exceeding differences in 77.6% to 80.6% of same-condition cases under single-observation reporting, then any between-condition comparison conducted with the same reporting rule inherits that baseline risk of false interpretation.

The dependence of timing dispersion on workload role is one of the audit’s most practically useful findings, because it implies that sample-size requirements and reporting caution cannot be standardized across all CPU-only workloads. Dense numerical computation, which is dominated by sustained arithmetic in cache-resident data, tends to exhibit different variability characteristics than memory-movement workloads, which are sensitive to cache hierarchy behavior, prefetching effectiveness, and memory controller contention. Framework-level inference adds another layer, since the framework’s internal scheduling, memory allocation patterns, and operator dispatch introduce variability sources that pure numerical kernels do not have. Single-step training scenarios combine these effects with the additional overhead of optimizer state updates and gradient computation. The audit’s finding that dispersion depends strongly on workload role means that a sample size adequate for one workload type may be inadequate for another, even within the same execution context and phase policy. Practitioners who benchmark a heterogeneous set of workloads cannot assume that a single measurement protocol will provide uniform evidential support across all of them.

The role of phase policy, which governs whether warm-up measurements are included or excluded from the reported summary, interacts with the audit’s findings in ways that merit elaboration. Early executions of a workload often run slower because instruction and data caches are cold, branch predictors are untrained, dynamic libraries have not been fully loaded, and just-in-time compilation or lazy initialization may occur. Including these warm-up phases in the reported summary inflates both the central tendency and the dispersion of the timing distribution, because the warm-up observations are drawn from a different regime than the steady-state observations. Excluding them reduces dispersion but introduces a researcher degree of freedom: the choice of how many observations to discard affects the reported result. The audit’s insistence that phase policy be explicitly documented addresses this concern by making the choice visible and comparable across studies. Without such documentation, two studies measuring the same workload on the same hardware could report substantially different efficiency values solely because of different warm-up handling, and readers would have no way to reconcile the discrepancy.

The three execution contexts examined in the study, two local Windows environments and one managed Google Colab CPU allocation, represent a meaningful spread of the conditions under which CPU-only machine-learning work is actually performed. Local Windows machines are subject to background processes, antivirus scanning, automatic updates, and power-management policies that can inject variability at unpredictable moments. Managed cloud allocations, while more standardized in some respects, introduce their own sources of variability through virtualization overhead, noisy-neighbor effects from co-located workloads, and the abstraction layers between the guest operating system and the physical hardware. The fact that the audit was conducted across all three contexts strengthens its external relevance, because it demonstrates that the measurement-evidence problem is not confined to any single platform type. A practitioner moving between a personal workstation and a cloud notebook environment cannot assume that measurement practices validated in one context transfer to the other without re-establishing the variability baseline.

The propagation of timing uncertainty into derived metrics such as estimated GFLOP/s and effective bandwidth is worth examining in more detail, because these derived quantities are often the headline numbers in efficiency reports. When throughput or GFLOP/s is computed by dividing a fixed workload size by measured wall-clock time, the relative uncertainty in the derived metric is approximately the relative uncertainty in the timing measurement, since the workload size is typically known exactly. This means that a timing measurement with 15% run-to-run variability yields a GFLOP/s estimate with comparable instability, and any comparison between two conditions based on single observations of each inherits the combined variability of both. The audit’s focus on wall-clock time as the primary measured quantity is therefore well-founded: controlling the evidential quality of the timing measurement is the necessary first step toward trustworthy derived metrics. Speedup ratios, which divide one timing by another, compound the problem further, because the relative uncertainty of a ratio depends on the uncertainties of both numerator and denominator.

The audit’s framing of the reporting decision as a choice between strong, conditional, and cautious wording offers a constructive path forward for the field. Rather than treating measurement variability as a reason to abandon efficiency reporting altogether, the framework acknowledges that reports must be written and provides a principled basis for calibrating the strength of the claims they contain. A difference supported by 100 observations per group with low same-condition exceedance can be stated strongly; a difference based on ten observations with moderate exceedance should be stated conditionally; a difference based on a single observation in a high-variability context warrants cautious wording that acknowledges the possibility that the measurement process itself produced the observed gap. This graduated approach mirrors how other measurement-intensive disciplines have evolved, replacing binary significant-or-not judgments with explicit statements about the evidential basis of each claim.

For the broader machine-learning community, the audit’s findings arrive at a moment when CPU-only evaluation retains genuine practical importance. Edge deployment, educational settings, privacy-constrained environments, and the preprocessing stages of accelerator-based pipelines all rely on CPU execution, and efficiency decisions in these settings affect real resource consumption and user experience. The audit does not argue that CPU-only efficiency measurement is futile; it argues that the current common practice of reporting single timing observations without context is insufficient for the decisions those numbers are meant to support. By providing a concrete procedure, quantified exceedance rates, and sample-size guidance grounded in observed variability, the study gives practitioners a template for strengthening their own measurement practices. The broader lesson extends beyond CPU-only settings: any efficiency claim, on any hardware, is ultimately a claim about a measurement process, and the credibility of the claim is bounded by the quality of the repeated-measurement evidence behind it.

Subject of Research: computer science

Article Title: A same-condition decision-risk audit for CPU-only machine-learning efficiency measurement

Article References: A same-condition decision-risk audit for CPU-only machine-learning efficiency measurement. (n.d.). https://doi.org/10.3389/fcomp.2026.1928487

Image Credits: AI Generated

DOI: 10.3389/fcomp.2026.1928487

Keywords: machine learning, benchmarking, CPU performance, measurement variability, reproducibility, same-condition, decision-risk, audit, CPU-only, machine-learning, efficiency, measurement