The study, led by Alyssa A. Pradarelli of the University of Michigan and colleagues, addressed a persistent gap in medical education. While workplace-based assessments have become the dominant approach to evaluating clinical performance in authentic settings over the past two decades, there has been limited guidance on how to translate the large volumes of performance data they generate into defensible, high-stakes decisions about trainee competence. Existing standard-setting efforts in medical and surgical education have largely centered on single tasks, single procedures, or written examinations rather than on longitudinal, multi-procedure performance data derived from WBAs.

To fill this gap, the researchers convened a panel of 10 surgical educators for a two-hour virtual Benchmark Panel. Panel size was informed by existing guidance for standard-setting panels in health professions education, which recommends approximately 8 to 12 well-trained panelists with diversity of training context and expertise prioritized over additional size. All panelists were practicing surgeons or surgical fellows at the time of the session, recruited to represent diverse program types and subspecialties. The panel was tasked with defining a cut score corresponding to a practice-ready graduation standard for high-value, high-frequency procedures in general surgery.

The test score at the heart of the exercise was the SIMPL Operative Performance (OP) score, a novel metric developed by the Society for Improving Medical Professional Learning (SIMPL). The OP score summarizes cumulative workplace-based assessments using methods adapted from computer adaptive testing and Bayesian inference networks. The scoring pipeline involves two distinct modeling steps: first, a two-parameter Item Response Theory (2PL IRT) model was fit to existing SIMPL OR performance ratings across a pool of 165 surgical procedures to estimate procedure-specific difficulty and discrimination parameters; second, a Bayesian network combined those parameters with an individual trainee’s accumulated ratings to produce an estimated overall ability on the OP score scale, which ranges from 372 to 628 and is organized into five levels. The underlying data represented approximately 3,000 trainees across 80 distinct procedures.

Three standard-setting methods were adapted to this longitudinal data. The Angoff method, a mainstay of medical education assessment, asked panelists to estimate the probability that a minimally competent graduating trainee would meet a practice-ready standard on each specified procedure. The Bookmark method presented procedures ordered by predicted probability of practice-ready performance and asked panelists to identify the point beyond which a minimally competent trainee would have less than a 75 percent chance of earning a practice-ready rating. Construct Mapping, the third approach, used a heat map representation of predicted probabilities across the OP score range, allowing panelists to identify the score that separates an intermediate from a practice-ready trainee while incorporating both criterion- and norm-referencing.

The results were striking in their consistency. Mean participant-level OP cut scores were 577 (standard deviation 30) for the Angoff method, 588 (17) for the Bookmark method, and 582 (19) for Construct Mapping. A one-way repeated-measures analysis of variance demonstrated no significant association between standard-setting method and panelists’ OP cut scores, with Holm-adjusted pairwise comparisons also non-significant. This convergence across methodologically independent approaches stands in sharp contrast to prior surgical standard-setting work based on single observed performances, in which different methods produced cut scores translating into failure rates ranging from 0 to 50 percent among experienced surgeons. The researchers suggest this difference may stem from the OP score aggregating many ratings across multiple procedures rather than measuring performance on a single task.

The panel ultimately aligned on a proposed graduation standard of 582 on the SIMPL OP scale, anchored in the Construct Mapping method based on qualitative preference. This cut score corresponds to the 86th percentile in the sample, meaning roughly 14 percent of trainees exceeded it. At this threshold, a graduating resident would have an 88 to 90 percent predicted probability of earning a practice-ready rating for laparoscopic appendectomy, 83 to 85 percent for laparoscopic cholecystectomy, and 76 to 80 percent for open partial colectomy. Lower probabilities were associated with more complex procedures, including 67 to 71 percent for laparoscopic partial colectomy and 53 to 55 percent for laparoscopic splenectomy.

Qualitative analysis of the panel session, conducted using interpretive description methodology, surfaced important process considerations for future standard-setting efforts. Panelists found the Angoff method cognitively demanding, in part because it embedded the challenge of aligning on a shared mental model of the minimally competent graduating trainee within the method itself. The Bookmark method suffered from panelists’ lack of agreement with a pre-determined, statistically derived ordering of procedures, which reduced their confidence in the results. Construct Mapping was perceived as the most useful approach, praised for its efficiency and its ability to translate a single cut score across multiple procedures in a visually intuitive way, though panelists noted difficulty interpreting heat map data that pooled trainees across all postgraduate year levels rather than focusing on graduating residents.

The findings also raise deeper conceptual questions about what surgical competence means. Panelists wrestled with the distinction between competency in managing a specific disease and competency in a specific procedure, noting that a trainee who can safely manage a condition by one approach may not need to be equally proficient in an alternative technique. They also grappled with how to incorporate variation in case complexity, ultimately aligning on a standard defined for straightforward cases. The tension between a minimally competent standard and a mastery learning standard remains open, particularly for the highest-value, highest-frequency procedures, with recent meta-analytic evidence indicating that mastery-based variants yield substantially higher pass criteria and meaningfully lower pass rates than minimum-competency approaches.

The authors acknowledge limitations, including the small single-panel sample, the focus on a circumscribed set of procedures and straightforward case complexity, and the fact that the detailed OP score methodology has not yet been published. The OP score is also specific to the SIMPL OR platform and cannot be directly transferred to other WBA systems without additional calibration. Nevertheless, the study provides an initial framework for setting defensible, data-informed competency standards in the era of workplace-based assessment. Future directions include expanding panels to include board representatives, recently graduated surgeons, and patient representatives, and evaluating the consequence validity of the resulting cut scores on national pass rates and remediation decisions.

Subject of Research: Standard-setting methods for defining surgical graduation competency using longitudinal workplace-based assessment data

Article Title: Standard setting in the era of workplace-based assessment: a pilot exploration

Article References: Pradarelli, A. A., Marcotte, K. M., George, B. C., Loftus, T. J., Korndorffer, J. R., Jr., Hughes, D. T., Kwakye, G., McKinley, S. K., White, E. M., Lau, J. N., Bohnen, J. D., & Krumm, A. E. (2026). Standard setting in the era of workplace-based assessment: a pilot exploration. Global Surgical Education – Journal of the Association for Surgical Education, 5(1), Article 200. https://doi.org/10.1007/s44186-026-00598-0

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00598-0

Keywords: standard setting, workplace-based assessment, competency-based medical education, surgical education, operative performance, SIMPL, Angoff method, Bookmark method, construct mapping, item response theory, Bayesian networks, graduation standards

The study, published in Global Surgical Education, the Journal of the Association for Surgical Education, examined a program struggling with low adoption of the System for Improving and Measuring Procedural Learning, known as SIMPL. The smartphone-based platform was designed to let faculty deliver rapid, structured workplace-based assessments of residents using entrustable professional activities, or EPAs, which describe the discrete tasks a surgeon must be able to perform without supervision. The American Board of Surgery now requires EPA-based assessment for general surgery residency, typically delivered through SIMPL, but programs nationwide have reported inconsistent engagement that undermines the reliability of the resulting data.

Researchers at Sidney Kimmel Medical College at Thomas Jefferson University and Jefferson Einstein Philadelphia Hospital, led by Jandie Posner and Ramsey M. Dallal, approached the problem not as an education question but as an organizational change problem. They turned to ADKAR, a change-management model developed by Jeff Hiatt that breaks successful change into five sequential elements: awareness of the need for change, desire to participate, knowledge of how to change, ability to implement new skills, and reinforcement to sustain the change. Rather than simply asking faculty and residents to use the platform more, the team first diagnosed why they had stopped.

The diagnostic phase took the form of a faculty survey completed by 66 surgeons, whose responses were mapped onto the five ADKAR domains to identify where the implementation was failing. The findings informed a multicomponent intervention delivered between August and December 2024. It included re-education sessions for both faculty and residents on the purpose and mechanics of EPA assessments, a monthly leaderboard that gamified participation by ranking assessment activity, and formal recognition of high performers. The bundle was designed to act on multiple ADKAR domains simultaneously, raising awareness of the national mandate, cultivating desire through competition and recognition, restoring knowledge through refresher training, and building reinforcement into the program’s monthly rhythm.

To measure the effect, the investigators compared matched calendar quarters before and after the intervention: January through March 2024 served as the baseline, and January through March 2025 as the post-intervention period. The analysis included 48 resident-periods representing 31 unique residents, with 24 residents in each period. Because assessment counts are skewed count data, the team used count-data regression adjusted for postgraduate year and resident type, a statistical approach that accounts for the fact that senior residents and categorical residents might naturally accumulate more assessments than junior or preliminary residents. The model also accounted for residents who were present in both periods, allowing a within-person comparison.

The results were striking. Total EPA assessments rose from 36 to 100 across the two periods. Adjusted for training level and resident type, residents completed approximately 2.6 times as many assessments after the intervention, with an incidence-rate ratio of 2.58 and a 95 percent confidence interval of 1.17 to 5.73, reaching statistical significance at p equals 0.019. The confidence interval spans a range from a modest to a large effect, a nuance the authors acknowledged rather than smoothed over. Resident participation rates doubled, climbing from 37.5 percent to 75 percent, also statistically significant. Assessment activity expanded across all phases of surgical care, indicating that the gains were not confined to a single rotation or service line.

The statistical model revealed an additional pattern: activity varied significantly by training level, peaking among mid-level residents, but did not differ between categorical and preliminary residents. This distributional detail matters for program directors, because it suggests that mid-level trainees, who are often entrusted with greater operative autonomy, may be the natural center of gravity for workplace-based assessment, while interventions may need tailoring to activate junior and senior cohorts differently.

The authors were notably candid about the limits of their success. Decomposing the increase, they estimated that roughly 40 percent of the program-level gain reflected resident cohort turnover rather than behavior change among existing faculty and residents, as new interns arrived in a climate of heightened engagement. When the analysis isolated residents present in both periods, the within-resident change was not statistically significant, and a concurrent national trend toward increased EPA use may have contributed to the observed rise. The study was a single-center before-after design reported according to SQUIRE 2.0 quality-improvement standards, without a concurrent control group, so these uncontrolled results warrant caution. The team also emphasized that the study measured only EPA utilization, not the quality of feedback, resident learning, or actual competency outcomes, leaving open the question of whether more assessments translate into better surgeons.

Those caveats, however, do not erase the study’s practical value. The authors frame the intervention as a reproducible bundle: diagnose barriers systematically, map them to a recognized change framework, deliver targeted re-education, and embed reinforcement through gamification and recognition. The gamification element builds on a growing literature showing that leaderboards and competition can increase engagement in health professions education, while the ADKAR mapping gives other programs a vocabulary for figuring out which stage of change their own implementation is stuck at. A program whose faculty are unaware of the EPA mandate needs a different fix than one whose faculty know the mandate but lack the desire to comply, and the ADKAR lens makes that distinction actionable.

The work also extends a lineage of re-implementation studies. A prior BMC Surgery report from a different program, published under the pointed title When the First Try Fails, documented a similar recovery effort, and earlier multi-institutional trials had established SIMPL’s feasibility for real-time intraoperative assessment. What the new study adds is a theoretically grounded, low-cost template that programs facing low EPA engagement can adapt without new technology or funding. The research received no specific grant funding, the institutional review board deemed it exempt, and the authors report no conflicts of interest. As EPA-based assessment becomes a fixed feature of surgical training, the study’s central message resonates beyond surgery: implementing an assessment platform is a change-management exercise, and treating it as one may be the difference between a tool that collects dust and one that transforms how surgical competence is measured.

Subject of Research: ADKAR-guided re-implementation of the SIMPL EPA assessment platform in a general surgery residency program

Article Title: When implementing SIMPL is not so simple: an ADKAR-guided re-implementation and engagement strategy in a general surgery residency

Article References: Posner, J., Kardan, R., Godbole, M., Lei, J., Moran, B., & Dallal, R. M. (2026). When implementing SIMPL is not so simple: an ADKAR-guided re-implementation and engagement strategy in a general surgery residency. Global Surgical Education – Journal of the Association for Surgical Education, 5(1), Article 160. https://doi.org/10.1007/s44186-026-00569-5

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00569-5

Keywords: entrustable professional activities, SIMPL, ADKAR, change management, surgical education, general surgery residency, resident assessment, faculty engagement, gamification, quality improvement, workplace-based assessment, competency-based education

Previous investigations into gender gaps in surgical training have relied largely on summary-level data captured at a single moment: graduation. Those studies consistently found that female trainees finish residency with lower Accreditation Council for Graduate Medical Education (ACGME) case totals and report lower intraoperative autonomy than male colleagues. But such snapshots could not answer a critical question that has vexed educators for years—do these disparities exist from day one, or do they develop over the five-year arc of surgical apprenticeship? The new research, published in Global Surgical Education, the journal of the Association for Surgical Education, set out to trace exactly when and how these differences emerge.

The scale of the dataset is what sets the study apart. Researchers analyzed 569,761 individual ACGME case log entries generated by residents at six programs, matching them to 415 attending surgeons, of whom 141, or 34 percent, were women. Crucially, the team obtained resident and attending gender through self-reported data from the SIMPL platform, a smartphone-based surgical performance feedback system, since gender is not recorded in standard ACGME case logs. Nearly half of the residents, 234 of 473, were women. The programs themselves reflected national demographics, with female resident representation ranging from 42 to 58 percent across sites.

Each case entry captured the resident’s post-graduate year, the operative role recorded, and the identity and gender of both trainee and attending surgeon. ACGME case roles fall into distinct categories: first assistant, in which the resident helps another surgeon; surgeon junior and surgeon chief, which together constitute primary surgeon roles; and teaching assistant, in which a chief resident supervises a junior resident through an operation. These designations matter enormously. The American Board of Surgery and program clinical competency committees use case log data to judge whether residents are ready to graduate and to practice independently.

The headline finding was stark. On average, female residents logged 8.1 percent fewer total cases than male residents—1,153 versus 1,255, a statistically significant difference. Yet the more revealing pattern emerged when the researchers stratified cases by operative role and training year. There were no meaningful differences between male and female residents in how often they logged cases as primary surgeon. The divergence appeared instead in first assistant entries, and it widened dramatically with seniority. Female residents were 1.57 times more likely than male peers to log a case as first assistant in their third year of training, with an odds ratio of 1.57 and a 95 percent confidence interval of 1.14 to 2.16. By the fourth year that odds ratio climbed to 2.76, and by the fifth and final year it reached 3.72—meaning senior female residents were nearly four times as likely to record themselves in the assisting role.

The temporal pattern is what makes the finding so consequential. In the early years of residency, serving as first assistant is expected and appropriate; PGY1 residents in the dataset logged 15,047 first assistant cases, a figure that fell to just 212 among PGY5 residents as trainees assumed the surgeon’s role. But a substantial minority of residents continued logging first assistant cases deep into senior years—52 percent of PGY3 residents, 33 percent of PGY4 residents, and 14 percent of PGY5 residents did so. Among this persistent group, women were markedly overrepresented. The most frequently logged first assistant cases included laparoscopic cholecystectomy, laparoscopic appendectomy, inguinal hernia repair, ventral hernia repair, and colectomy—core general surgery procedures in which senior residents should be operating with increasing independence.

The study also examined whether the gender of the attending surgeon shaped these patterns, a question no prior case log analysis had been able to address. Overall, cases performed with female faculty were less likely to be logged as first assistant, with an odds ratio of 0.79. Yet among senior residents the association reversed: PGY4 residents operating with a female attending were 1.68 times more likely to log first assistant cases, and PGY5 residents were 2.46 times more likely. Notably, the researchers found no significant differences between gender-concordant and gender-discordant resident-attending pairings, suggesting that simple matching of trainee and faculty gender does not explain the observed trends. The authors caution that the mechanisms behind these attending-level effects—whether they reflect differences in teaching style, entrustment behavior, or documentation habits—remain outside the scope of the current data and merit dedicated investigation.

Why would female senior residents log more first assistant cases? The authors emphasize that the dataset cannot establish causation, and they lay out competing explanations that likely operate in combination. Extrinsic factors could include faculty perceptions, institutional culture, or the timing of cases within the academic year, while intrinsic factors might involve self-perception, confidence, and how residents understand the role definitions themselves. Teaching assistant and first assistant categories lack the strict criteria, such as post-graduate year level, that govern primary surgeon designations, making them more vulnerable to misinterpretation. Indeed, since primary surgeon cases showed no gender difference at all, the researchers suggest that some of the disparity may stem from inconsistent application of logging guidelines rather than unequal operative opportunity. At the same time, they are careful to note the well-documented literature on gender-based discrimination in surgical training, acknowledging that the uneven distribution of operative roles may reflect faculty bias or structural inequities that limit women’s chances to operate as lead surgeon.

The study carries important limitations that the authors confront directly. The data contain no information about case complexity beyond procedural codes, which are known to be inaccurately recorded, and the analysis excluded residents whose gender data was missing or reported as other or prefer not to answer, leaving the experiences of non-binary trainees unexamined. Race and ethnicity data were absent entirely, a significant gap for any work on equity. The sample was also confined to large university-based programs, which may limit generalizability, although the consistency of the trends with national-level data lends credibility to the findings. One resident, for example, logged an exploratory laparotomy as a teaching assistant case during the first year—a clear misclassification that illustrates the noise inherent in self-reported logs, even as subtler errors likely permeate the data.

The implications reach well beyond the operating room. As surgical education shifts toward competency-based training and Entrustable Professional Activities, understanding how different trainee subpopulations use measurement tools becomes essential to interpreting quantitative data fairly. The authors call for future research pairing ACGME case logs with faculty EPA assessments and resident self-evaluations, alongside qualitative studies of how residents actually decide which role to record. In the meantime, they offer a pragmatic recommendation: individual programs can apply the same analytical methods to their own case log data, monitoring for disparities by gender, PGY level, and even rotation site in real time. Such local surveillance, the researchers argue, could allow program leadership to identify senior residents who continue logging first assistant cases, clarify the correct application of role definitions, and address any underlying inequities before they calcify into a training record—and perhaps a career trajectory—that understates a surgeon’s true capabilities.

Subject of Research: Gender disparities in operative case logging and surgical training roles among general surgery residents

Article Title: Gender disparities among general surgery residents’ ACGME case logs develop over the course of training

Article References: Gender disparities among general surgery residents’ ACGME case logs develop over the course of training. (n.d.). https://doi.org/10.1007/s44186-026-00575-7

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00575-7

Keywords: general surgery, residency training, ACGME case logs, gender disparities, operative autonomy, surgical education, first assistant, attending surgeon gender, SIMPL, graduate medical education, operative experience, competency-based training