The study, led by Alyssa A. Pradarelli of the University of Michigan and colleagues, addressed a persistent gap in medical education. While workplace-based assessments have become the dominant approach to evaluating clinical performance in authentic settings over the past two decades, there has been limited guidance on how to translate the large volumes of performance data they generate into defensible, high-stakes decisions about trainee competence. Existing standard-setting efforts in medical and surgical education have largely centered on single tasks, single procedures, or written examinations rather than on longitudinal, multi-procedure performance data derived from WBAs.

To fill this gap, the researchers convened a panel of 10 surgical educators for a two-hour virtual Benchmark Panel. Panel size was informed by existing guidance for standard-setting panels in health professions education, which recommends approximately 8 to 12 well-trained panelists with diversity of training context and expertise prioritized over additional size. All panelists were practicing surgeons or surgical fellows at the time of the session, recruited to represent diverse program types and subspecialties. The panel was tasked with defining a cut score corresponding to a practice-ready graduation standard for high-value, high-frequency procedures in general surgery.

The test score at the heart of the exercise was the SIMPL Operative Performance (OP) score, a novel metric developed by the Society for Improving Medical Professional Learning (SIMPL). The OP score summarizes cumulative workplace-based assessments using methods adapted from computer adaptive testing and Bayesian inference networks. The scoring pipeline involves two distinct modeling steps: first, a two-parameter Item Response Theory (2PL IRT) model was fit to existing SIMPL OR performance ratings across a pool of 165 surgical procedures to estimate procedure-specific difficulty and discrimination parameters; second, a Bayesian network combined those parameters with an individual trainee’s accumulated ratings to produce an estimated overall ability on the OP score scale, which ranges from 372 to 628 and is organized into five levels. The underlying data represented approximately 3,000 trainees across 80 distinct procedures.

Three standard-setting methods were adapted to this longitudinal data. The Angoff method, a mainstay of medical education assessment, asked panelists to estimate the probability that a minimally competent graduating trainee would meet a practice-ready standard on each specified procedure. The Bookmark method presented procedures ordered by predicted probability of practice-ready performance and asked panelists to identify the point beyond which a minimally competent trainee would have less than a 75 percent chance of earning a practice-ready rating. Construct Mapping, the third approach, used a heat map representation of predicted probabilities across the OP score range, allowing panelists to identify the score that separates an intermediate from a practice-ready trainee while incorporating both criterion- and norm-referencing.

The results were striking in their consistency. Mean participant-level OP cut scores were 577 (standard deviation 30) for the Angoff method, 588 (17) for the Bookmark method, and 582 (19) for Construct Mapping. A one-way repeated-measures analysis of variance demonstrated no significant association between standard-setting method and panelists’ OP cut scores, with Holm-adjusted pairwise comparisons also non-significant. This convergence across methodologically independent approaches stands in sharp contrast to prior surgical standard-setting work based on single observed performances, in which different methods produced cut scores translating into failure rates ranging from 0 to 50 percent among experienced surgeons. The researchers suggest this difference may stem from the OP score aggregating many ratings across multiple procedures rather than measuring performance on a single task.

The panel ultimately aligned on a proposed graduation standard of 582 on the SIMPL OP scale, anchored in the Construct Mapping method based on qualitative preference. This cut score corresponds to the 86th percentile in the sample, meaning roughly 14 percent of trainees exceeded it. At this threshold, a graduating resident would have an 88 to 90 percent predicted probability of earning a practice-ready rating for laparoscopic appendectomy, 83 to 85 percent for laparoscopic cholecystectomy, and 76 to 80 percent for open partial colectomy. Lower probabilities were associated with more complex procedures, including 67 to 71 percent for laparoscopic partial colectomy and 53 to 55 percent for laparoscopic splenectomy.

Qualitative analysis of the panel session, conducted using interpretive description methodology, surfaced important process considerations for future standard-setting efforts. Panelists found the Angoff method cognitively demanding, in part because it embedded the challenge of aligning on a shared mental model of the minimally competent graduating trainee within the method itself. The Bookmark method suffered from panelists’ lack of agreement with a pre-determined, statistically derived ordering of procedures, which reduced their confidence in the results. Construct Mapping was perceived as the most useful approach, praised for its efficiency and its ability to translate a single cut score across multiple procedures in a visually intuitive way, though panelists noted difficulty interpreting heat map data that pooled trainees across all postgraduate year levels rather than focusing on graduating residents.

The findings also raise deeper conceptual questions about what surgical competence means. Panelists wrestled with the distinction between competency in managing a specific disease and competency in a specific procedure, noting that a trainee who can safely manage a condition by one approach may not need to be equally proficient in an alternative technique. They also grappled with how to incorporate variation in case complexity, ultimately aligning on a standard defined for straightforward cases. The tension between a minimally competent standard and a mastery learning standard remains open, particularly for the highest-value, highest-frequency procedures, with recent meta-analytic evidence indicating that mastery-based variants yield substantially higher pass criteria and meaningfully lower pass rates than minimum-competency approaches.

The authors acknowledge limitations, including the small single-panel sample, the focus on a circumscribed set of procedures and straightforward case complexity, and the fact that the detailed OP score methodology has not yet been published. The OP score is also specific to the SIMPL OR platform and cannot be directly transferred to other WBA systems without additional calibration. Nevertheless, the study provides an initial framework for setting defensible, data-informed competency standards in the era of workplace-based assessment. Future directions include expanding panels to include board representatives, recently graduated surgeons, and patient representatives, and evaluating the consequence validity of the resulting cut scores on national pass rates and remediation decisions.

Subject of Research: Standard-setting methods for defining surgical graduation competency using longitudinal workplace-based assessment data

Article Title: Standard setting in the era of workplace-based assessment: a pilot exploration

Article References: Pradarelli, A. A., Marcotte, K. M., George, B. C., Loftus, T. J., Korndorffer, J. R., Jr., Hughes, D. T., Kwakye, G., McKinley, S. K., White, E. M., Lau, J. N., Bohnen, J. D., & Krumm, A. E. (2026). Standard setting in the era of workplace-based assessment: a pilot exploration. Global Surgical Education – Journal of the Association for Surgical Education, 5(1), Article 200. https://doi.org/10.1007/s44186-026-00598-0

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00598-0

Keywords: standard setting, workplace-based assessment, competency-based medical education, surgical education, operative performance, SIMPL, Angoff method, Bookmark method, construct mapping, item response theory, Bayesian networks, graduation standards