Bayesian adaptive testing with variable lengths and new stopping rules

Computerized adaptive testing has become one of the quiet success stories of modern measurement, powering everything from nursing licensure examinations to large-scale educational assessments. Rather than presenting every examinee with the same fixed set of questions, adaptive systems select each subsequent item based on the examinee’s evolving ability estimate, yielding precise scores with far fewer items than traditional tests. Yet a new study published in Behavior Research Methods reveals a hidden weakness at the heart of many of these systems—and offers a fully Bayesian solution that could make adaptive testing both more accurate and more honest about its own uncertainty.

The research, conducted by Luping Niu of the American Board of Pediatrics and Seung W. Choi of the University of Texas at Austin, addresses a problem that has long been acknowledged but rarely solved in practice: item parameters are never known with perfect precision. In standard computerized adaptive testing, or CAT, items in the pool are calibrated in advance using samples of examinees, and the resulting difficulty and discrimination parameters are treated as fixed, known quantities. This convenient assumption, the authors show, can seriously distort both the accuracy of ability estimates and the moment at which a variable-length test decides it has gathered enough information.

The consequences of ignoring calibration error are not merely theoretical. Earlier work has demonstrated that when item parameter uncertainty is brushed aside, adaptive algorithms can capitalize on estimation error, selecting items whose apparent statistical properties look better than they really are. This “capitalization on chance” inflates confidence in ability estimates that do not deserve it, producing standard errors that are too small and confidence intervals that are too narrow. In high-stakes settings—where pass-fail decisions about professional certification may ride on these numbers—such overconfidence is a genuine concern.

Niu and Choi’s answer is a fully Bayesian CAT algorithm that incorporates item parameter uncertainty directly into every stage of the testing process. Instead of relying on point estimates of item parameters, the fully Bayesian approach maintains a full posterior distribution for each item’s characteristics, propagating that uncertainty into both the estimation of the examinee’s ability and the selection of the next item. The result is a measurement system that knows what it does not know: ability estimates come with honest interval estimates that reflect not only the randomness of the examinee’s responses but also the imperfect knowledge of the items themselves.

To evaluate this approach, the researchers turned to simulation. They varied the size of the calibration samples used to estimate item parameters and the size of the item pools available to the adaptive algorithm, creating a grid of conditions that mirrors the practical constraints testing organizations actually face. Small calibration samples and small item pools—common in operational programs—are precisely the conditions under which parameter uncertainty is most severe. Under each condition, the fully Bayesian algorithm was pitted against a conventional CAT algorithm that treated item parameters as known.

Three stopping rules formed the second axis of the investigation. In variable-length CAT, the test does not have a predetermined number of items; instead, it continues until a criterion is met. The most common criterion is the standard error rule: keep administering items until the standard error of the ability estimate drops below a specified threshold. The second is the change-in-theta, or CIT, rule, which stops when the ability estimate has stabilized—when successive estimates change by less than a small amount. The third, a combined CIT plus SE rule, requires both that the estimate has stabilized and that the standard error criterion is satisfied. The researchers compared how the fully Bayesian and conventional algorithms behaved under each of these three rules.

The findings were clear on the accuracy front. Across the simulated conditions, the fully Bayesian algorithm generally improved estimation accuracy and, crucially, produced interval coverage rates closer to their nominal levels. In plain terms, when the fully Bayesian system reported a 95 percent confidence interval around an ability estimate, that interval actually contained the true ability about 95 percent of the time—something conventional CAT failed to achieve, particularly when the calibration sample size was small. The advantage was strongest exactly where it matters most: in the data-poor conditions that characterize many real testing programs, including those with small item pools or limited pretesting resources.

The story with stopping rules was more nuanced. The standard error rule alone, long the workhorse of variable-length CAT, showed a troubling tendency in certain regions of the ability scale: it could keep tests running far longer than necessary. The authors found this happened particularly at ability levels for which the remaining item pool offers limited additional information. When the pool simply contains no items that can meaningfully reduce the standard error further, insisting on reaching the SE threshold forces examinees to answer a stream of questions that accomplish almost nothing—a waste of testing time, examinee patience, and operational resources.

The combined CIT plus SE rule proved to be the practical remedy. By requiring both stabilization of the ability estimate and satisfaction of the precision criterion, the combined rule trimmed those unnecessarily long tests, particularly at ability levels where the pool’s information was exhausted. The result, the authors report, is a practical balance between measurement precision and testing efficiency: examinees are not released prematurely with unreliable scores, but neither are they held hostage to a threshold the item pool can no longer help them reach. For testing organizations, that balance translates directly into shorter tests, reduced costs, and improved examinee experience without sacrificing score quality.

The implications extend well beyond the simulation laboratory. Adaptive testing underpins the NCLEX examinations used for nursing licensure, certification programs in emergency medicine, and countless educational and psychological assessments. Any of these programs that operate with modest calibration samples—especially smaller organizations or newer testing programs—stand to benefit from a framework that quantifies uncertainty more honestly. Moreover, as testing programs increasingly grapple with item exposure, security, and the need to refresh item pools frequently, the assumption of perfectly known item parameters becomes ever harder to defend. A Bayesian treatment offers a principled way to live with that uncertainty rather than pretend it away.

The methodological lineage of the study is worth noting. Fully Bayesian adaptive testing builds on a growing body of work showing that Bayesian approaches can handle the joint uncertainty of examinee ability and item parameters, including fast algorithms developed in recent years that make Bayesian item selection computationally feasible in real time. What distinguishes the present contribution is its focus on the variable-length setting, where the stopping decision itself depends on uncertainty quantification. If the standard errors are wrong, the test ends at the wrong time—and the fully Bayesian framework corrects those standard errors at the source.

The researchers have also made their work transparent and reproducible. Both the simulated data and the R program code used for the CAT simulations are publicly available through the Open Science Framework, allowing other measurement specialists to replicate the conditions, adapt the simulation engine to their own item pools, or extend the stopping-rule comparisons to additional models. The work builds on Niu’s doctoral dissertation at the University of Texas at Austin, adding new investigations and refined approaches to the fully Bayesian stopping framework.

For the field of psychometrics, the message is twofold. First, item parameter uncertainty is not a technical footnote but a first-class citizen that belongs in the estimation machinery of adaptive tests; ignoring it yields scores that are silently overconfident. Second, when it comes to deciding when to stop, no single rule is optimal in all circumstances, but the combination of a stabilization criterion and a precision criterion offers a robust compromise that adapts gracefully to the realities of finite item pools. As adaptive testing continues to expand into new domains—from health measurement to language assessment—the fully Bayesian framework, paired with sensible stopping rules, may well become the standard against which future systems are judged.