<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>confidence intervals &#8211; Science</title>
	<atom:link href="https://scienmag.com/tag/confidence-intervals/feed/" rel="self" type="application/rss+xml" />
	<link>https://scienmag.com</link>
	<description></description>
	<lastBuildDate>Fri, 16 Oct 2026 17:02:33 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.3</generator>

<image>
	<url>https://scienmag.com/wp-content/uploads/2024/07/cropped-scienmag_ico-32x32.jpg</url>
	<title>confidence intervals &#8211; Science</title>
	<link>https://scienmag.com</link>
	<width>32</width>
	<height>32</height>
</image> 
<site xmlns="com-wordpress:feed-additions:1">73899611</site>	<item>
		<title>New Statistical Framework Helps Researchers Judge When Effects Actually Matter</title>
		<link>https://scienmag.com/new-statistical-framework-helps-researchers-judge-when-effects-actually-matter/</link>
		
		<dc:creator><![CDATA[Glenn Wilkins]]></dc:creator>
		<pubDate>Fri, 16 Oct 2026 17:02:33 +0000</pubDate>
				<category><![CDATA[Psychology & Psychiatry]]></category>
		<category><![CDATA[Behavior Research Methods]]></category>
		<category><![CDATA[confidence interval analysis]]></category>
		<category><![CDATA[confidence intervals]]></category>
		<category><![CDATA[decision-making under uncertainty]]></category>
		<category><![CDATA[effect size]]></category>
		<category><![CDATA[effect size evaluation]]></category>
		<category><![CDATA[equivalence testing]]></category>
		<category><![CDATA[interval-based hypothesis testing]]></category>
		<category><![CDATA[meaningfulness of research effects]]></category>
		<category><![CDATA[minimum-effect test]]></category>
		<category><![CDATA[practical relevance]]></category>
		<category><![CDATA[practical significance in psychology]]></category>
		<category><![CDATA[quantitative psychology]]></category>
		<category><![CDATA[quantitative psychology reform]]></category>
		<category><![CDATA[research effect interpretation]]></category>
		<category><![CDATA[research methods]]></category>
		<category><![CDATA[significance testing limitations]]></category>
		<category><![CDATA[smallest effect size of interest]]></category>
		<category><![CDATA[smallest effect size of interest (SESOI)]]></category>
		<category><![CDATA[statistical significance]]></category>
		<category><![CDATA[statistical vs. practical significance]]></category>
		<guid isPermaLink="false">https://scienmag.com/?p=287102</guid>

					<description><![CDATA[A new framework in Behavior Research Methods gives researchers structured tools for interpreting inconclusive results from interval-based hypothesis tests that use a smallest effect size of interest.]]></description>
										<content:encoded><![CDATA[<p>For decades, psychology has been obsessed with one question: is the effect different from zero? A study produces a p-value below 0.05, the result is declared significant, and the finding enters the literature. But a growing movement within quantitative psychology argues that this binary ritual answers the wrong question. An effect can be statistically significant yet so tiny that it changes nothing in the real world, while a nonsignificant result can mask an effect that genuinely matters. A new paper published in Behavior Research Methods by Paul Riesthuis of KU Leuven, together with Robert A. Cribbie and Victoria Celio of York University and Nataly Beribisky of Davidson College, tackles a stubborn gap in this reform effort: what should researchers do when even the most modern interval-based tools cannot tell them whether an effect is practically meaningful or practically negligible?</p>
<p>The framework builds on a practice that has been gaining ground in psychological methods: interval-based hypothesis testing. Instead of testing whether an effect differs from zero, researchers first specify a smallest effect size of interest, often abbreviated as SESOI, which is the boundary below which an effect is considered too small to care about. They then examine a confidence interval around the observed effect. If the entire interval falls outside the SESOI boundaries, the effect is deemed practically meaningful, a procedure known as a minimum-effect test. If the entire interval falls inside the boundaries, the effect is deemed negligible, the logic behind equivalence testing. These approaches, which trace back to work in pharmacology by Westlake in 1972 and to minimum-effect tests proposed by Murphy and Myors in 1999, force researchers to commit to what counts as a meaningful result before they see the data.</p>
<p>But there is a catch that the new paper puts front and center. Confidence intervals frequently straddle the SESOI boundaries, overlapping both the meaningful and the negligible regions at the same time. When that happens, the interval-based test is inconclusive: the data are compatible with an effect that matters and with an effect that does not. The authors illustrate the problem with a running example throughout the article. Suppose an observed effect of 5.5 on some scale, with a standard error of 0.51, is tested against SESOI boundaries of minus 5 and plus 5 using a two-tailed minimum-effect test. The 90 percent confidence interval around the observed effect extends beyond the upper boundary, so the researcher cannot confidently claim the effect is practically meaningful, yet the point estimate sits above the threshold. The result is neither a clean win nor a clean loss, and standard practice offers little guidance on what to say next.</p>
<p>This inconclusive zone is not a rare curiosity. Because confidence intervals widen with smaller samples and noisier measures, underpowered studies land in the uncertainty zone routinely. The authors argue that the problem has been underappreciated precisely because the methodological reform conversation has focused on getting researchers to adopt SESOI-based tests in the first place, without equipping them for the awkward outcomes those tests can produce. A researcher who runs an equivalence test and finds the confidence interval crossing the equivalence margin is left with a result that looks like failure but is really just ambiguity. Worse, the pressure to publish can tempt researchers to interpret ambiguous intervals in whichever direction favors their hypothesis, reintroducing the very flexibility that interval-based testing was meant to eliminate.</p>
<p>To address this, Riesthuis and colleagues propose a set of complementary decision tools, packaged in a freely available Web application that accompanies the paper. The tools are designed to be used after an interval-based test yields an inconclusive result, helping researchers characterize just how uncertain they are and what kinds of effects remain plausible given the data. Rather than forcing a binary verdict, the framework encourages researchers to describe the position of the confidence interval relative to the SESOI boundaries, quantify how much of the interval lies in the meaningful versus negligible region, and consider what additional data would be needed to reach a decisive conclusion. The goal, the authors write, is to support more informed decision-making under uncertainty and to help researchers draw more nuanced statistical and practical conclusions from their results.</p>
<p>The framework also engages with a broader debate about how uncertainty should be quantified. Notably, the authors use a 90 percent confidence interval for their interval-based tests rather than the conventional 95 percent interval, citing a rationale discussed by Alter and Counsell in 2023. Because a two-sided interval-based test at significance level alpha effectively involves checking whether the interval excludes the SESOI boundary, a 1 minus 2 alpha confidence level is the coherent choice, and this detail matters for researchers implementing the procedures correctly. The paper also distinguishes itself from magnitude-based inference, a controversial approach used in sports science that was systematically reviewed in 2020 and criticized in 2019 as neither Bayesian nor a valid method of inference. The new framework stays firmly within frequentist interval logic while adding structured tools for interpreting ambiguous outcomes.</p>
<p>Specifying the SESOI in the first place is itself a thorny problem, and the paper situates its contribution within an existing literature on how to do it well. Anchor-based methods, described by Anvari and Lakens in 2021, derive thresholds from external benchmarks, such as the minimal clinically important difference concept introduced in health-status measurement by Jaeschke, Singer and Guyatt in 1989. Other work has surveyed experts to elicit the smallest effect that would matter in a given field, including studies on false memory research and on expert witness testimony concerning alcohol and memory. A 2025 literature review by Riesthuis and Cribbie examined when scientific findings are deemed practically or theoretically relevant in minimum-effect testing applications. The new paper assumes such a threshold has been chosen and focuses on the downstream problem: what to conclude when the interval will not cooperate.</p>
<p>The practical implications reach beyond academic statistics. Recent work has documented how frequently nonsignificance is misinterpreted as evidence of an effect&#8217;s absence in psychology, with a 2025 analysis in Royal Society Open Science documenting the prevalence of this error. Interval-based testing was supposed to fix such misinterpretations by making the role of effect magnitude explicit. But if inconclusive results are handled ad hoc, the reform risks trading one form of confusion for another. The authors also point to connections with replication research, noting in prior work that statistical nonsignificance does not equal an unsuccessful replication when the smallest effect size of interest is taken into account. A replication that lands in the uncertainty zone may simply reflect insufficient precision, not a failed replication, and the framework&#8217;s tools help make that distinction visible.</p>
<p>For researchers who want to apply the framework, the entry points are concrete. The Web application, hosted at paulriesthuis.github.io/SESOI-Decisions/, implements the complementary tools, and a mirror is available on ShinyApps. The code is openly available on the Open Science Framework, and supplementary materials accompany the article, including an HTML file and a PDF. The authors even provide exact parameters for reproducing their leading example: an observed effect of 5.5, a standard error of 0.51, SESOI bounds of minus 5 and 5, and a two-tailed minimum-effect test. The manuscript was supported by a FWO post-doctoral fellowship grant awarded to the first author, and the authors declared no conflicts of interest.</p>
<p>The paper lands at a moment when quantitative psychology is actively rethinking inference, with parallel efforts ranging from unified null-region frameworks to practical guides for drawing statistical conclusions that bridge null hypothesis testing and estimation. What distinguishes this contribution is its honesty about the limits of the reform it supports. Specifying a smallest effect size of interest and testing against it is a genuine improvement over worshiping the p-value threshold against zero, but no statistical procedure eliminates uncertainty; it can only make uncertainty explicit and manageable. By giving researchers a structured way to reason through inconclusive interval-based tests, the framework aims to make the ambiguity itself informative, showing which conclusions the data can support, which they cannot, and what it would take to find out. In a field still learning to think in intervals rather than verdicts, that may be the most practical result of all.</p>
<p><strong>Subject of Research:</strong> A statistical framework for evaluating practical relevance in interval-based hypothesis testing using a smallest effect size of interest</p>
<p><strong>Article Title:</strong> Decisions under uncertainty: A statistical framework for evaluating practical relevance in interval-based hypothesis testing</p>
<p><strong>Article References:</strong> Riesthuis, P., Cribbie, R. A., Celio, V., &amp; Beribisky, N. (2026). Decisions under uncertainty: A statistical framework for evaluating practical relevance in interval-based hypothesis testing. <em>Behavior Research Methods, 58</em>(11), Article 323. <a href="https://doi.org/10.3758/s13428-026-03204-9" rel="noopener noreferrer">https://doi.org/10.3758/s13428-026-03204-9</a></p>
<p><strong>Image Credits:</strong> AI Generated</p>
<p><strong>DOI:</strong> <a href="https://doi.org/10.3758/s13428-026-03204-9" rel="noopener noreferrer">10.3758/s13428-026-03204-9</a></p>
<p><strong>Keywords:</strong> interval-based hypothesis testing, smallest effect size of interest, equivalence testing, minimum-effect test, effect size, statistical significance, practical relevance, confidence intervals, quantitative psychology, research methods, decision making under uncertainty, Behavior Research Methods</p>
]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">287102</post-id>	</item>
		<item>
		<title>New R Package Brings the Sampling Algorithms Behind Human Thinking to Every Lab</title>
		<link>https://scienmag.com/new-r-package-brings-the-sampling-algorithms-behind-human-thinking-to-every-lab/</link>
		
		<dc:creator><![CDATA[Glenn Wilkins]]></dc:creator>
		<pubDate>Thu, 15 Oct 2026 06:41:29 +0000</pubDate>
				<category><![CDATA[Psychology & Psychiatry]]></category>
		<category><![CDATA[anchoring effect]]></category>
		<category><![CDATA[approximate Bayesian computation]]></category>
		<category><![CDATA[autocorrelated Bayesian sampler]]></category>
		<category><![CDATA[autocorrelation]]></category>
		<category><![CDATA[Bayesian modeling in psychology]]></category>
		<category><![CDATA[Bayesian sampler]]></category>
		<category><![CDATA[Behavior Research Methods]]></category>
		<category><![CDATA[Cognitive modeling]]></category>
		<category><![CDATA[computational models of overconfidence]]></category>
		<category><![CDATA[confidence intervals]]></category>
		<category><![CDATA[diagnostics for autocorrelation in sampling]]></category>
		<category><![CDATA[fast R tools for Bayesian cognitive models]]></category>
		<category><![CDATA[Hamiltonian Monte Carlo for psychology research]]></category>
		<category><![CDATA[human decision-making sampling algorithms]]></category>
		<category><![CDATA[Markov chain Monte Carlo]]></category>
		<category><![CDATA[Markov chain Monte Carlo methods for cognitive science]]></category>
		<category><![CDATA[probability estimation drift analysis]]></category>
		<category><![CDATA[probability judgment biases]]></category>
		<category><![CDATA[probability judgments]]></category>
		<category><![CDATA[R package]]></category>
		<category><![CDATA[R package for cognitive sampling models]]></category>
		<category><![CDATA[sampling algorithms]]></category>
		<category><![CDATA[sampling-based explanations of human biases]]></category>
		<category><![CDATA[samplr]]></category>
		<guid isPermaLink="false">https://scienmag.com/?p=273003</guid>

					<description><![CDATA[Researchers at the University of Warwick have released samplr, an open-source R package that makes Markov chain Monte Carlo sampling algorithms and sampling-based cognitive models accessible for modeling human probability judgments, estimates, choices, and confidence.]]></description>
										<content:encoded><![CDATA[<p>Why do people make systematic mistakes when judging probabilities? Why do repeated estimates of the same quantity drift in strange, long-range patterns? Why are we overconfident, and why does a choice we just made bias the very next number we produce? A growing body of research argues that all of these quirks share a single root cause: the human mind does not compute exact answers, but instead draws samples from internal probability distributions. Now, a team at the University of Warwick has released a software package designed to make that idea testable by anyone who can run R.</p>
<p>The package, called samplr, was introduced by Lucas Castillo, Yun-Xiao Li, and Adam N. Sanborn in the journal Behavior Research Methods. It bundles together a family of Markov chain Monte Carlo algorithms, including Metropolis–Hastings and Hamiltonian Monte Carlo, alongside cognitive models built on top of them, such as the Bayesian sampler and the autocorrelated Bayesian sampler. The core routines are written in C++ for speed, running up to four times faster than equivalent MATLAB code and far faster than base R. Diagnostic functions for analyzing autocorrelation and power spectral density round out the toolkit. The authors hope it will do for sampling-based models of cognition what packages for drift diffusion models did for sequential sampling: turn a technically demanding modeling approach into something ordinary labs can adopt.</p>
<p>The intellectual backdrop is a decades-long convergence between statistics and psychology. Statisticians use sampling algorithms for a practical reason: approximating complex probability distributions that cannot be computed exactly. Calculating a posterior distribution requires dividing by a normalizing constant, the sum of prior-times-likelihood across every possible event, which quickly becomes intractable. Markov chain Monte Carlo sidesteps this by requiring only ratios of probabilities, in which the constants cancel out. The price is autocorrelation: successive samples are dependent, and results depend on the starting point. Strikingly, those same dependencies appear in human behavior, in repeated judgments and in the anchoring effect, which is precisely why psychologists began proposing that the brain itself might run something like MCMC.</p>
<p>The package demonstrates its power on classic findings. Consider probability judgments, where people famously commit the conjunction fallacy, rating the probability of event A as smaller than that of A and B together, and violate binary complementarity, with estimates of A and not A summing to more than one. Two leading models, probability theory plus noise and the Bayesian sampler, make identical predictions for the mean of people&#8217;s judgments but differ in how judgment variance relates to the mean. In the Bayesian sampler, fewer samples pull the endpoints of the mean-variance curve toward the center, while in probability theory plus noise the endpoints stay fixed. samplr provides functions to compute the diagnostic Z identities and to simulate the Bayesian sampler directly, letting researchers distinguish the two accounts with their own data.</p>
<p>Another striking phenomenon is 1/f noise in cognition. In the 1990s, researchers found that when people repeatedly reproduce a duration, their errors are not independent but show long-range autocorrelations, with a power spectral exponent between white noise and Brownian noise. Within a sampling framework, these repeated estimates can be seen as raw samples drawn without aggregation into a decision. Work by the Warwick group compared Metropolis–Hastings, Metropolis-coupled MCMC, and Hamiltonian Monte Carlo and found that the multiple-chain algorithm best reproduced the long-range structure of human estimates. samplr implements all of these algorithms, so users can generate sample sequences and run the same spectral analyses on them.</p>
<p>Random generation offers a third case study. Asked to produce random heights as if drawing from a bag, people deviate systematically: distances between consecutive items are too small, and repetitions almost never occur. In recent experiments, participants generated random heights under Gaussian and Uniform conditions, and six sampling algorithms were pitted against the data. The winner was MCREC, an algorithm combining the gradient-based proposals of Hamiltonian Monte Carlo with the multiple chains of Metropolis-coupled MCMC and a sense of momentum that adds autocorrelation to the direction of proposals. All six algorithms are available in samplr through functions beginning with sampler_, and they can target mixtures or fully custom distributions.</p>
<p>The package is not limited to the authors&#8217; own work. The article walks through a case study from research on learning across the lifespan, in which participants placed a bucket to catch bags dropped by a helicopter hidden in clouds, never knowing whether an odd drop reflected ordinary fluctuation or a genuine change in the helicopter&#8217;s position. A model from that study explained perseveration and anchoring biases by assuming people draw posterior samples with Metropolis–Hastings in resource-limited chunks, stopping early when the samples do not change the current belief. The authors show that samplr&#8217;s sampler_mh function could implement the same model, and on their machine it ran 31 times faster than the original code, illustrating how a shared, optimized code base can accelerate modeling across research groups.</p>
<p>Perhaps the most ambitious component is the autocorrelated Bayesian sampler, or ABS, which uses a single sampling process to explain probability judgments, point estimates, confidence intervals, choices, confidence judgments, and response times. For two-alternative choices, sampling continues until the accumulated difference in evidence between options crosses a threshold; raising the threshold yields the classic speed-accuracy trade-off, with slower but more accurate responses. The same framework predicts over-precise confidence intervals, because limited samples underrepresent distributional tails, and the resolution of confidence, because errors tend to arise when evidence stays balanced for a long time, inflating the sample count and lowering confidence. It even explains how choices bias later estimates: when the chosen value sits near the stimulus distribution, estimates are pushed away from it, the repulsion effect, while a distant value pulls estimates toward it, the anchoring effect. Both emerge from the same autocorrelated chain starting at the choice value.</p>
<p>Qualitative matches are only the beginning, and samplr also supports quantitative model comparison through likelihood-free inference. Because these models have no tractable likelihood function, the package leans on approximate Bayesian computation: simulate data from candidate models, compress each dataset into summary statistics, and keep only simulations close enough to the observed data. In the random generation example, rejection ABC with 10,000 simulations per model gave overwhelming support to MCREC, with a Bayes factor above 400 over the next best model. A more sophisticated variant using random forests, implemented in the external abcrf package, weighs summary statistics by their predictive value and needs fewer simulations; it pushed the Bayes factor above a thousand. The authors stress the accompanying Bayesian workflow: checking that priors produce sensible data and that the procedure can recover known models before trusting its verdicts.</p>
<p>The broader promise is unification. Sampling algorithms have already been applied to phenomena from low-level perception to causal reasoning, and they are not confined to simple representations; text descriptions could be mapped into semantic vector spaces and sampled using probabilities from a neural network. If a single sampling process, as ABS proposes, can quantitatively account for judgments, choices, response times, and confidence across tasks, psychologists will gain far more insight than separate models for separate response modes ever could. samplr is free, open-source software available on CRAN, with datasets from the group&#8217;s experiments in a companion package, and the authors are actively inviting feature requests and contributions from the community.</p>
<p><strong>Subject of Research:</strong> An R software package implementing statistical sampling algorithms and sampling-based cognitive models of human judgment and decision-making</p>
<p><strong>Article Title:</strong> The samplr package: A tool for modeling human cognition with sampling algorithms</p>
<p><strong>Article References:</strong> Castillo, L., Li, Y.-X., &amp; Sanborn, A. N. (2026). The samplr package: A tool for modeling human cognition with sampling algorithms. <em>Behavior Research Methods, 58</em>(11), Article 319. <a href="https://doi.org/10.3758/s13428-026-03188-6" rel="noopener noreferrer">https://doi.org/10.3758/s13428-026-03188-6</a></p>
<p><strong>Image Credits:</strong> AI Generated</p>
<p><strong>DOI:</strong> <a href="https://doi.org/10.3758/s13428-026-03188-6" rel="noopener noreferrer">10.3758/s13428-026-03188-6</a></p>
<p><strong>Keywords:</strong> samplr, sampling algorithms, Markov chain Monte Carlo, Bayesian sampler, cognitive modeling, probability judgments, autocorrelation, anchoring effect, confidence intervals, approximate Bayesian computation, R package, Behavior Research Methods</p>
]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">273003</post-id>	</item>
		<item>
		<title>Stop trusting single accuracy scores: why ML models need distributional evaluation</title>
		<link>https://scienmag.com/stop-trusting-single-accuracy-scores-why-ml-models-need-distributional-evaluation/</link>
		
		<dc:creator><![CDATA[Blake Davidson]]></dc:creator>
		<pubDate>Sun, 11 Oct 2026 12:08:38 +0000</pubDate>
				<category><![CDATA[Technology and Engineering]]></category>
		<category><![CDATA[benchmarking]]></category>
		<category><![CDATA[best practices for model validation]]></category>
		<category><![CDATA[confidence intervals]]></category>
		<category><![CDATA[deep learning]]></category>
		<category><![CDATA[distributional performance metrics]]></category>
		<category><![CDATA[experimental design]]></category>
		<category><![CDATA[hyperparameter optimization]]></category>
		<category><![CDATA[hyperparameter optimization impact]]></category>
		<category><![CDATA[importance of multiple model assessments]]></category>
		<category><![CDATA[interpreting model performance distributions]]></category>
		<category><![CDATA[limitations of single accuracy scores]]></category>
		<category><![CDATA[Machine learning]]></category>
		<category><![CDATA[machine learning evaluation]]></category>
		<category><![CDATA[model evaluation]]></category>
		<category><![CDATA[model robustness and variability]]></category>
		<category><![CDATA[quantiles]]></category>
		<category><![CDATA[randomized data augmentation]]></category>
		<category><![CDATA[reproducibility]]></category>
		<category><![CDATA[risk assessment]]></category>
		<category><![CDATA[statistical analysis in ML]]></category>
		<category><![CDATA[statistics]]></category>
		<category><![CDATA[stochastic nature of ML training]]></category>
		<category><![CDATA[train-test split effects]]></category>
		<category><![CDATA[uncertainty quantification]]></category>
		<guid isPermaLink="false">https://scienmag.com/?p=262126</guid>

					<description><![CDATA[A large-scale study argues that machine learning performance should be reported as distributions with confidence intervals on quantiles rather than single point estimates, revealing model weaknesses that average metrics conceal.]]></description>
										<content:encoded><![CDATA[<p>Every machine learning practitioner knows the ritual: train a model, run it on a held-out test set, and report a single number — an accuracy of 0.91, a root mean squared error of 10.4. That number then travels through papers, dashboards, and boardroom slides as if it were a definitive statement about the model. A new study argues that this ritual is quietly misleading us. Machine learning training is a stochastic process, shaped by random weight initializations, random data splits, random augmentation, and randomized hyperparameter search, and a single performance figure is just one draw from an unknown underlying distribution. Treating it as the whole story, the authors contend, is like judging a drug trial from a single patient.</p>
<p>The research, published in Machine Learning with Applications by Christoph Lehmann and Yahor Paromau, proposes a deceptively simple reframing: view ML training as a controlled scientific experiment with repeated measurements. By varying one confounding factor at a time — the train–test split, the initial weights, the data augmentation scheme, the dropout configuration, or the hyperparameter optimization procedure — while holding everything else fixed via random seed control, each training run becomes a sample from the distribution of a target metric of interest. Repeating the run with different seeds yields repeated measurements, and with them the tools of classical inferential statistics become applicable: quantiles, confidence intervals, and explicit uncertainty quantification.</p>
<p>The scale of the empirical work is unusual for this kind of methodological study. The authors ran roughly 500 to 1,000 seed-controlled training repetitions for each source of variation across three real-world use cases: image classification of Simpsons characters with convolutional neural networks, the standard CIFAR10 benchmark with a VGG11 architecture, and a regression task predicting the critical temperature of superconductors using deep neural networks and gradient boosting trees. The effort consumed approximately 19,000 GPU hours on NVIDIA A100 and H100 accelerators, producing empirical performance distributions dense enough to serve as near-ground-truth references against which small-sample statistical methods could be validated.</p>
<p>The core statistical move is a shift from means to quantiles. A quantile answers a risk-oriented question that a mean cannot: the 25% quantile of accuracy tells you the accuracy falls below that threshold in only a quarter of runs, while the 90% quantile of RMSE tells you the error exceeds that value in just one run out of ten. The authors draw an explicit parallel to Value-at-Risk in financial regulation, where a quantile of the loss distribution defines the threshold exceeded only with a small pre-specified probability. For high-stakes applications — medicine, banking, criminal justice — this threshold language maps naturally onto acceptance criteria: a model can be required, in advance, to keep its error below a bound with a stated probability, something a mean-based evaluation cannot certify.</p>
<p>Quantiles, however, are harder to estimate than means. They are sensitive to the shape of the distribution, and the more extreme the quantile level, the more data is needed. The study characterizes this trade-off precisely. Exact nonparametric confidence intervals, built from order statistics and the binomial distribution, exist only when the sample size clears a minimum that grows steeply toward the distribution&#8217;s tails: estimating a 10% quantile with 90% confidence requires at least 22 runs, while a 1% quantile demands 230. Asymptotic intervals based on the normal approximation of sample quantiles have their own requirements, and the authors show a clever sign transformation — analyzing the negated metric — that converts a costly lower-tail quantile into a cheaper upper-tail one, cutting the required sample size for a 10% quantile from 42 to 25 runs.</p>
<p>For situations where even those minimums cannot be met, the authors evaluate a semiparametric bootstrap that extrapolates into the distribution&#8217;s tails using the spacing of the outermost order statistics. This method remains applicable where the closed-form intervals are undefined, at the price of extrapolation that may not match the true tail behavior. Across 42 experimental settings and a simulation study spanning skewed Beta distributions, normal mixtures, and Gumbel-distributed errors, the asymptotic nonparametric interval emerged as the practical default, offering a favorable balance of coverage, interval length, and simplicity. Bootstrap intervals remained a usable fallback for very small samples, though with coverage dropping to around 0.85 and noticeably wider intervals.</p>
<p>The practical payoff is demonstrated with two worked examples. In the Simpsons classification task, two training configurations — one varying hyperparameter optimization, one varying data augmentation — had nearly identical mean accuracies of 0.874 and 0.872, indistinguishable by any mean-based comparison. But their 10% quantile confidence intervals, estimated from just 25 runs each, did not overlap at all: [0.870, 0.873] for hyperparameter optimization versus [0.860, 0.868] for data augmentation. The hyperparameter-optimized configuration was substantially more resistant to producing poor runs, a fact invisible in the means. Against a deployment requirement that accuracy not drop below 0.87 in more than 10% of cases, only one configuration qualified — a statistically grounded model selection decision that point estimates could never support.</p>
<p>The regression example on superconductors showed the complementary value of the framework. A deep neural network and a gradient boosting tree differed clearly in mean RMSE, 10.47 versus 9.59, and their confidence intervals for the mean, the 10% quantile, and the 90% quantile were all cleanly separated with similar widths. That uniform pattern, the authors note, is the empirical signature of a simple location shift between the two distributions — and its absence of tail-specific effects is itself informative, justifying a mean-based comparison in this case. Across 2,000 repeated paired draws, the quantile intervals separated the two models in 92 to 99 percent of draws at n=25, while the mean separated them essentially always.</p>
<p>The study also distills concrete guidance for practitioners working under realistic compute budgets. With fewer than 10 repetitions, only a t-interval for the mean is advisable. Between 10 and 15 runs, exact nonparametric intervals for the quartiles become feasible, with a maximum recommended confidence level of 0.9. At 25 runs, asymptotic intervals with the sign transformation extend coverage to the 10% and 90% quantile levels, and at 50 or more runs, extreme quantiles such as the 5% and 95% levels come within reach. Interval lengths themselves carry diagnostic value: a median interval markedly longer than the mean&#8217;s t-interval signals low density near the center and hints at bimodality, while asymmetric tail intervals reveal skewness. The authors also caution that metrics like accuracy are formally discrete, computed on a grid set by test-set size, and recommend inspecting the fraction of distinct values in a sample before trusting quantile estimates.</p>
<p>What emerges is a bridge between two communities that rarely speak. Statisticians have long treated experimental outcomes as random variables deserving of distributional analysis; machine learning practice, optimized for leaderboard positions and single benchmark numbers, has largely resisted that view. This work shows that the statistical machinery is not merely compatible with modern deep learning — it is affordable. Even 10 to 25 seed-controlled runs, a modest addition to any training budget, unlock tail behavior, stability assessment, and requirement-driven evaluation that no point estimate can provide. As models move into domains where a single bad run can mean a misdiagnosis or a denied loan, the authors&#8217; message lands with force: the average is not the answer, and the distribution is the evidence.</p>
<p><strong>Subject of Research:</strong> Distributional uncertainty quantification in machine learning performance evaluation using quantile estimation and confidence intervals</p>
<p><strong>Article Title:</strong> Beyond point estimates: Distributional uncertainty in machine learning performance evaluation</p>
<p><strong>Article References:</strong> Lehmann, C., &amp; Paromau, Y. (2026). Beyond point estimates: Distributional uncertainty in machine learning performance evaluation. <em>Machine Learning with Applications, 26</em>, Article 101033. <a href="https://doi.org/10.1016/j.mlwa.2026.101033" rel="noopener noreferrer">https://doi.org/10.1016/j.mlwa.2026.101033</a></p>
<p><strong>Image Credits:</strong> AI Generated</p>
<p><strong>DOI:</strong> <a href="https://doi.org/10.1016/j.mlwa.2026.101033" rel="noopener noreferrer">10.1016/j.mlwa.2026.101033</a></p>
<p><strong>Keywords:</strong> machine learning, model evaluation, uncertainty quantification, confidence intervals, quantiles, statistics, reproducibility, deep learning, risk assessment, benchmarking, hyperparameter optimization, experimental design</p>
]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">262126</post-id>	</item>
		<item>
		<title>New Browser Tool Turns Regression Math Into Something You Can Actually See</title>
		<link>https://scienmag.com/new-browser-tool-turns-regression-math-into-something-you-can-actually-see/</link>
		
		<dc:creator><![CDATA[Denise Maddox]]></dc:creator>
		<pubDate>Fri, 02 Oct 2026 09:05:12 +0000</pubDate>
				<category><![CDATA[Technology and Engineering]]></category>
		<category><![CDATA[confidence intervals]]></category>
		<category><![CDATA[D3.js]]></category>
		<category><![CDATA[data visualization]]></category>
		<category><![CDATA[educational tools for data science students]]></category>
		<category><![CDATA[enhancing understanding of confidence intervals]]></category>
		<category><![CDATA[geometric interpretation of linear regression]]></category>
		<category><![CDATA[interactive learning]]></category>
		<category><![CDATA[interactive statistical software for learning regression]]></category>
		<category><![CDATA[limitations of traditional statistical software]]></category>
		<category><![CDATA[linear regression]]></category>
		<category><![CDATA[open-source data science tools]]></category>
		<category><![CDATA[open-source software]]></category>
		<category><![CDATA[R-squared]]></category>
		<category><![CDATA[real-time regression visualization in web browsers]]></category>
		<category><![CDATA[regression analysis visualization]]></category>
		<category><![CDATA[regression coefficient interpretation]]></category>
		<category><![CDATA[regression diagnostics and error decomposition]]></category>
		<category><![CDATA[residual diagnostics]]></category>
		<category><![CDATA[sampling variability]]></category>
		<category><![CDATA[sampling-based interval exploration]]></category>
		<category><![CDATA[statistics education]]></category>
		<category><![CDATA[variance decomposition]]></category>
		<category><![CDATA[visualization of statistical concepts]]></category>
		<category><![CDATA[web application]]></category>
		<guid isPermaLink="false">https://scienmag.com/?p=226742</guid>

					<description><![CDATA[A new open-source web application called RegVis makes the geometry of linear regression and the meaning of confidence intervals directly visible through interactive, real-time visualization.]]></description>
										<content:encoded><![CDATA[<p>Linear regression is one of the most widely taught and widely used methods in all of statistics, data science, and machine learning, yet the mathematics at its core has long remained stubbornly invisible. Students learn to compute a coefficient of determination, R², from a formula, and they learn that confidence intervals should contain a true parameter a certain percentage of the time, but the geometric and probabilistic meaning behind those numbers rarely becomes tangible. A new open-source software package called RegVis, described in the journal SoftwareX by Uzair Ahmad and Hamna Uzair, aims to change that by turning the abstract machinery of regression into something users can watch, poke, and dismantle in real time, directly inside a web browser.</p>
<p>The motivation for the tool stems from a persistent gap in how statistical software presents regression. According to the authors, mainstream packages such as SPSS, Minitab, Tableau, and JMP present regression results through disconnected numerical tables and static plots, obscuring the geometric relationships that are essential for deep conceptual understanding. A systematic survey of ten widely used systems conducted by the developers found that none of them provides geometric error decomposition or sampling-based interval exploration, while coordinated diagnostic views appear only in limited or partial form in a small number of systems. RegVis was built specifically to close those three gaps within a single, installation-free web application.</p>
<p>The first and perhaps most striking feature is the geometric decomposition of variance. For every data point, RegVis draws a red translucent square whose side length equals the absolute residual, the distance between the observed value and the fitted value predicted by the regression line. It also draws a blue translucent square whose side length equals the distance between the observed value and the mean of the response. The aggregate area of all red squares corresponds to the sum of squared errors, SSE, while the aggregate area of all blue squares corresponds to the total sum of squares, TSS. The identity R² = 1 − SSE/TSS, usually presented as an abstract algebraic formula, thus becomes a visible contrast between two fields of colored area. When a user drags a point or adds an outlier, the squares animate smoothly, and the shift in the ratio of red to blue area makes the change in explained variance perceptually apparent rather than merely calculable.</p>
<p>The developers are careful to note that this area-based rendering is intended as a conceptual aid rather than a precise measurement instrument. Research in graphical perception going back to the classic work of Cleveland and McGill shows that area encodings are less accurate than position or length for exact quantitative comparison. RegVis embraces this limitation deliberately: the squares are meant to convey a holistic, intuitive sense of how total variance splits into explained and unexplained components, while the exact numeric value of R² remains available in a live readout at all times. Users can toggle each layer independently and limit the maximum number of displayed squares to reduce visual clutter, supporting a progressive disclosure approach that starts with a simple scatterplot and builds toward the full decomposition.</p>
<p>The second major innovation tackles one of the most misunderstood concepts in statistics: the confidence interval. Studies have shown that even professional researchers harbor robust misinterpretations of what a confidence interval actually means. RegVis addresses this with an interactive sampling demonstration. Clicking the Sample button draws a random subsample of a user-configured proportion from the current dataset, fits a least-squares line to that subset, and overlays the result as a persistent semi-transparent grey trace. Each click produces a fresh, independent sample, and the accumulating cloud of grey lines gives learners a visceral sense of sampling variability. The displayed confidence band, computed once from the complete dataset, serves as the reference against which the repeated sample lines can be judged.</p>
<p>Because the dataset is treated as a finite population, the behavior of the containment rate follows finite-population sampling theory rather than the textbook infinite-population ideal. A grey trace counts as contained only if it lies within the confidence band at every value of x. As the sampling fraction increases, sampling variance shrinks through the finite-population correction, and containment rises monotonically toward certainty at 100 percent, where the subsample coincides with the population. To make this relationship observable rather than merely asserted, a Simulate function draws 100 independent samples at each of eleven sampling fractions from 0 to 100 percent, fits a line to each, and plots the resulting containment rate against the sampling fraction. In one illustrative workflow described in the paper, a 50-point dataset at the 95 percent confidence level produced a curve rising from roughly 10 percent containment at a 10 percent sampling fraction to full containment at 100 percent.</p>
<p>Underlying both features is a coordinated three-view architecture designed around Shneiderman&#8217;s direct-manipulation principle. The primary scatterplot displays the data, the red regression line, the toggleable error squares, adjustable confidence and prediction intervals, and a live regression equation with R² and sample size. A residuals-versus-fitted-values panel reveals heteroscedasticity as cone-shaped patterns, non-linearity as systematic curves, and outliers as extreme vertical displacements. A Q-Q plot maps empirical residual quantiles against theoretical normal quantiles, exposing heavy tails, skewness, and individual aberrant points. Hovering over any observation in any view triggers synchronized highlighting across all three panels, allowing users to trace a single data point through data space, error space, and distributional space simultaneously. Clicking anywhere adds a point; clicking an existing point removes it; and every such action triggers the full pipeline of data update, statistical recalculation, and rendering across all three views.</p>
<p>That pipeline is fast. The developers instrumented each architectural layer with performance timers and ran 30 repetitions at six dataset sizes across Microsoft Edge, Firefox, and Chrome on a 12-core Windows desktop. The computation layer, which recalculates least-squares coefficients, residuals, SSE, TSS, R², and interval bounds from scratch after every change, stayed below 1.2 milliseconds on average across all tested conditions. End-to-end latency rose gradually from under 2.3 milliseconds at ten points to a maximum of 17.93 milliseconds at 1,000 points in Firefox, comfortably below the roughly 100-millisecond threshold at which interactions are typically perceived as instantaneous. Rendering dominates the total, contributing an order of magnitude more time than computation even at the largest tested size, and the authors note that performance was not evaluated beyond 1,000 points. A benchmark script ships with the repository so that others can verify or extend the measurements.</p>
<p>The system is implemented entirely in client-side JavaScript using D3.js version 7, with HTML5 and CSS3, and requires no server-side installation; it can be deployed as a static file on any web host or opened directly from a local filesystem. The choice of D3.js over higher-level charting libraries was deliberate, since the custom geometric error squares, the sampling accumulation overlay, and the synchronized cross-view highlighting all demand low-level control over SVG elements that pre-built statistical libraries do not expose. The software is released under the MIT license on GitHub, includes a guided five-step tutorial covering dataset construction, variance decomposition, interval sampling, and diagnostic coordination, and supports CSV import and export with basic validation for reproducibility.</p>
<p>An exploratory usability assessment with eighteen expert participants, each holding at least one graduate-level statistics course, yielded a mean System Usability Scale score of 79.2 with a standard deviation of 12.8, a figure the authors describe as indicating high perceived usability, though they caution that the study lacked a controlled baseline and formal hypothesis testing. The developers also disclose that Ahmad is the founder of SpimeLab Inc., which hosts the platform, while stating that the reported results were developed independently of commercial considerations. Looking forward, the design principles of geometric correspondence, observable sampling, and coordinated multi-view interaction are described as general enough to extend to multiple regression, generalized linear models, ANOVA, and cross-validation. For now, RegVis positions itself as the first comprehensive visual environment in which regression mathematics is not merely calculable but directly observable, one clickable point at a time.</p>
<p><strong>Subject of Research:</strong> Interactive web-based visualization of linear regression analysis, variance decomposition, and confidence interval sampling for statistics education</p>
<p><strong>Article Title:</strong> RegVis: An interactive web-based visualization system for linear regression analysis and diagnostics</p>
<p><strong>Article References:</strong> Ahmad, U., &amp; Uzair, H. (2026). RegVis: An interactive web-based visualization system for linear regression analysis and diagnostics. <em>SoftwareX, 36</em>, Article 103045. <a href="https://doi.org/10.1016/j.softx.2026.103045" rel="noopener noreferrer">https://doi.org/10.1016/j.softx.2026.103045</a></p>
<p><strong>Image Credits:</strong> AI Generated</p>
<p><strong>DOI:</strong> <a href="https://doi.org/10.1016/j.softx.2026.103045" rel="noopener noreferrer">10.1016/j.softx.2026.103045</a></p>
<p><strong>Keywords:</strong> linear regression, data visualization, statistics education, confidence intervals, D3.js, variance decomposition, open-source software, residual diagnostics, interactive learning, R-squared, sampling variability, web application</p>
]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">226742</post-id>	</item>
	</channel>
</rss>
