Two Words, Two Meanings: How Vague Exam Language Skews Surgical Testing

Every year, surgical trainees across Canada sit down to a nationwide knowledge examination that helps determine whether they are ready to practice. The questions look straightforward: a clinical vignette, a list of options, and a request to identify the best management. But according to a new study published in Global Surgical Education, the Journal of the Association for Surgical Education, those two innocuous words, best management, may be quietly undermining the very thing the exam is supposed to measure. Researchers at McGill University’s Institute of Health Sciences Education, working with data from the Canadian Association of General Surgeons, found that examinees interpret the phrase in strikingly different ways, and that this ambiguity makes questions harder without making them better at distinguishing strong candidates from weak ones.

The research team, led by Maryam Wagner with colleagues Paola Fata, Carol Wissa, and Carlos Gomez-Garibello, set out to answer a deceptively simple question: what do test-takers actually think best management means? Clinical judgement, clinical reasoning, decision-making, and problem-solving are all terms that describe how health professionals apply knowledge to choose treatments. Because clinical judgement is inherently complex, reasonable experts can disagree about what constitutes the right next step for a patient. That disagreement is expected in the clinic, but on a high-stakes multiple-choice examination it creates a serious measurement problem. If candidates cannot agree on what the question is asking, their scores may reflect their interpretation of the wording rather than their surgical knowledge.

To investigate, the researchers analyzed multiple-choice questions from three consecutive administrations of the national surgical examination, identifying items that contained the phrase best management and comparing them with items that did not. Two psychometric properties were central to the analysis. The first is difficulty, typically expressed as the proportion of examinees who answer an item correctly; a lower value means a harder question. The second is discrimination, which reflects how well an item separates high performers from low performers. A good exam item should be challenging enough to be informative but should reliably reward examinees who genuinely possess the tested knowledge. An item that is hard for the wrong reasons, such as confusing wording, fails that second test.

The results were revealing. Items containing the phrase best management were significantly more difficult than items without it, with mean performance of 0.51 on the flagged items compared with 0.57 on the rest. That difference was statistically significant, with an effect size of eta squared equal to 0.02, a small but meaningful shift across an entire examination. More troubling was what the researchers did not find: the ambiguous items showed no significant improvement in discrimination. In other words, the extra difficulty did not come from deeper clinical reasoning being tested. It appeared to come from examinees wrestling with what the question meant, an effect measurement specialists call construct-irrelevant variance, noise introduced by something other than the skill the test intends to measure.

The second half of the study went straight to the source. In one examination cycle, 432 examinees completed a post-examination survey. Two multiple-choice questions probed how they interpreted and applied the term best management when answering questions, and a third open-ended question invited their perceptions and opinions. The survey revealed substantial variability in interpretation. The two most common understandings were nearly tied: 35.5 percent of examinees read the phrase as asking for the next most appropriate investigation or intervention, while 33.1 percent understood it as requesting the definitive overall clinical decision. Those are not trivially different readings. A question about a patient with abdominal pain could reasonably demand an immediate diagnostic test under one interpretation and an operative plan under the other.

The open-ended responses, analyzed using constant comparative analysis, a qualitative method in which researchers iteratively compare responses to develop and refine categories, sharpened the picture. Examinees consistently pointed to a lack of specificity in three dimensions: clinical context, timing, and decision-making. Without knowing whether the question concerns an emergency department at two in the morning or a clinic follow-up weeks later, whether the patient is stable or deteriorating, or whether the exam wants the immediate next step or the ultimate disposition, candidates are left guessing. Some may default to the safest incremental action; others may leap to the definitive answer. Both strategies can be defensible clinical habits, yet only one will match the answer key.

This finding lands at the intersection of assessment science and everyday test-writing practice. Multiple-choice questions remain a workhorse of medical education because they are efficient, objective, and capable of sampling broad content. Organizations such as the National Board of Medical Examiners publish detailed item-writing guides precisely because flawed questions are a known hazard. Decades of research, including influential work by Haladyna, Downing, and Rodriguez on item-writing guidelines and by Downing on the consequences of violating standard item-writing principles, have shown that ambiguous or technically flawed items distort both test scores and the inferences drawn from them. The McGill study adds a specific, widely used phrase to the list of culprits, and it does so with national-scale data rather than a single institution’s question bank.

The stakes extend beyond individual scores. Examinations like the Canadian Association of General Surgeons exam, and analogues such as the American Board of Surgery In-Training Examination, serve as quality-improvement tools for training programs and as signals of readiness for independent practice. When a subset of items measures interpretation of wording rather than surgical knowledge, the validity of score interpretations weakens. Trainees who happen to share the item writer’s reading of best management gain an advantage unrelated to competence, while equally knowledgeable peers may be penalized. Over a full examination, such noise can shift rankings, influence program-level evaluations, and erode trust in the assessment system among the very surgeons it is meant to evaluate.

The study’s authors frame their conclusion as a call to item writers: use more specific, unambiguous language to reduce unnecessary interpretation and thereby improve the quality and validity of written assessments of clinical judgement. In practice, that could mean replacing best management with explicit instructions, such as asking for the most appropriate next step at this moment, or the definitive treatment of choice, or the first investigation to order. It could also mean specifying the clinical timeframe and context within the vignette so that only one answer is defensible. The researchers acknowledge that the study received no dedicated funding and was conducted as part of the examination’s ongoing quality-improvement and program-evaluation initiatives, using anonymized test performance data owned by the Canadian Association of General Surgeons, with ethics exemption granted by McGill’s research ethics board.

There is a broader lesson here for anyone who designs high-stakes tests, in medicine or beyond. Language that seems perfectly clear to the person who wrote it can fragment into multiple meanings in the minds of test-takers, and psychometrics can detect the damage even when it cannot name every cause. The McGill team’s combination of item-level statistics with direct surveys of examinees offers a template: measure the difficulty and discrimination of suspect items, then ask the people who sat the exam what they thought the questions meant. The answer, in this case, was that a phrase intended to evoke sound clinical judgement instead invited a coin flip between next step and final answer. Changing two words on a national exam may seem like a small fix, but for the thousands of surgical trainees whose futures ride on their scores, it could make the difference between being tested on what they know and being tested on how well they guess.

Subject of Research: Ambiguity of the phrase best management in multiple-choice assessment of clinical judgement on a national surgical examination

Article Title: From best to next: improving testing of clinical judgement and knowledge on a nationwide surgical knowledge exam

Article References: From best to next: improving testing of clinical judgement and knowledge on a nationwide surgical knowledge exam. (n.d.). https://doi.org/10.1007/s44186-026-00610-7

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00610-7

Keywords: clinical judgement, surgical education, multiple-choice questions, assessment validity, item writing, psychometrics, item difficulty, discrimination, construct-irrelevant variance, Canadian Association of General Surgeons, medical education, exam ambiguity