New BUDS Framework Controls Trial Error Rates Amid Benchmark Uncertainty

Researchers have introduced a statistical framework called Benchmark Uncertainty Design Selection, or BUDS, to address a critical flaw in the planning of two-stage single-arm phase II clinical trials. Published in BMC Medical Research Methodology, the study by Rebecca Irlmeier, Zhuoli Jin, and Fei Ye proposes a method for selecting trial designs that accounts for the inherent uncertainty in historical benchmarks. Traditional approaches often rely on a single fixed value for historical performance, but in practice, these benchmarks are frequently imprecise due to small sample sizes, population heterogeneity, changing eligibility criteria, and evolving standards of care. The new framework aims to prevent the costly advancement of ineffective treatments by ensuring that Type I error rates are controlled even when the true historical benchmark differs from the value used to calibrate the design.

The background of the research highlights a significant risk in current trial methodologies. Simon two-stage designs for binary endpoints and their time-to-event analogues, such as the restricted-Kwak and Jung method, typically depend on a fixed historical benchmark. When the clinically relevant benchmark exceeds the value used for calibration, the Type I error rate can be substantially inflated. This inflation can lead to the costly advancement of ineffective treatments. Existing design-selection criteria generally optimize efficiency at a single benchmark without accounting for this variability, leaving trials vulnerable to statistical errors when real-world data deviates from planning assumptions.

To solve this problem, the authors propose BUDS, a framework applicable to both binary and time-to-event endpoints. Rather than relying on a single planning benchmark, BUDS addresses benchmark uncertainty by specifying a plausible range of response rates for binary endpoints or survival probabilities at a prespecified timepoint for time-to-event endpoints.

The BUDS framework offers two specific objectives for selecting designs among those that satisfy both Type I error and Type II error constraints. The first, BUDS-Least-Regret, minimizes the maximum difference in expected sample size from the benchmark-specific optimal design. The second, BUDS-Avg-EN, minimizes the average expected sample size across the range. This objective focuses on overall efficiency. Through monotonicity, the framework ensures that the Type I error rate is controlled at the upper bound of the benchmark range, providing a statistical guarantee of error control across the specified uncertainty interval.

The study presents results from representative scenarios to demonstrate the efficacy of the BUDS framework. Using single-benchmark objectives, the Type I error rate increases substantially when true benchmarks exceed their planning values. For instance, the Simon Optimal design, with a planning benchmark of 0.100 and a true benchmark of 0.250, exhibited a Type I error rate of 0.407, far exceeding the nominal alpha level of 0.10. Similarly, the restricted-Kwak and Jung design, with a planning benchmark of 0.500 and a true benchmark of 0.673, showed a Type I error rate of 0.165, which is more than three times the nominal alpha level of 0.05.

In contrast, designs selected using the BUDS framework maintain Type I error rates at or below the nominal level across the entire benchmark range. This robustness is achieved without sacrificing the ability to prioritize either regret or efficiency, as the proposed objectives provide explicit criteria for this trade-off. The framework makes the trade-off between robustness and efficiency explicit. By controlling Type I error across the plausible benchmark range, BUDS offers a transparent and reliable method for trial planning.

The framework applies to both binary and time-to-event endpoints in two-stage single-arm Phase II trial planning. The framework is implemented in the open-source BUDS R package and interactive Shiny app.

The study concludes that BUDS provides a transparent framework for selecting designs when historical benchmarks are uncertain. It addresses a significant limitation in current practice by accounting for the variability in historical data. By making the trade-off between robustness and efficiency explicit, the framework enables more informed decision-making in trial planning.

Subject of Research: BUDS: Benchmark Uncertainty Design Selection for two-stage single-arm phase II trials

Article Title: BUDS: Benchmark Uncertainty Design Selection for two-stage single-arm phase II trials

Article References: Irlmeier, R., Jin, Z., & Ye, F. (2026). BUDS: Benchmark Uncertainty Design Selection for two-stage single-arm phase II trials. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-026-02994-y

Image Credits: AI Generated

DOI: 10.1186/s12874-026-02994-y

Keywords: BUDS, Benchmark, Uncertainty, Design, Selection, two-stage, single-arm, phase, trials, scientific research