Short Answer or Multiple Choice? Head-to-Head Test Reveals What Each Format Really Measures in Medical Students

Every medical student knows the feeling: the multiple-choice exam feels manageable, but the short-answer paper is where confidence crumbles. A new study from Ain Shams University in Cairo suggests that this intuition has a measurable basis, and that the two exam formats are not interchangeable windows into a student’s knowledge. In a carefully controlled comparison of first-year medical students, multiple-choice questions (MCQs) produced significantly higher scores than content-matched short answer questions (SAQs), even though both formats proved equally capable of telling strong students apart from weak ones.

The research, published in BMC Medical Education, set out to answer a deceptively simple question: when the same learning objectives are tested in two different formats, do the formats behave the same way? Psychometricians have long debated this. MCQs are prized for their objectivity, speed of marking, and resistance to scorer bias, while SAQs are widely believed to probe higher-order cognitive skills such as recall, reasoning, and synthesis. Yet rigorous head-to-head comparisons in formative, low-stakes undergraduate settings have remained surprisingly scarce, leaving educators to choose formats largely on tradition and logistics rather than evidence.

To fill that gap, the team ran a descriptive analytical cross-sectional study involving 178 first-year medical students enrolled in the Medical Parasitology course within the Locomotor and Skin module, the first module of the curriculum, during the 2024–2025 academic year. The design’s key strength was content matching: students sat two examinations, one SAQ and one MCQ, each containing exactly 20 items aligned with the same intended learning outcomes. This meant any difference in performance could be attributed to the format itself rather than to one test simply covering harder material. The examinations were delivered electronically with support from the university’s information technology unit, and all data were anonymized with informed consent obtained from participants under ethical approval from the Faculty of Medicine.

The headline result was stark. Students scored an average of 13.5 out of 20 on the MCQ paper but only 11.1 out of 20 on the SAQ paper, a difference that was highly statistically significant. In other words, the same cohort, tested on the same objectives, performed roughly 12 percentage points better when the answer was hidden among four or five options than when it had to be produced from memory. Item-level analysis explained why: MCQ items were significantly easier, with a mean difficulty index of 0.67 compared with 0.55 for SAQs. The difficulty index, conventionally denoted P, reflects the proportion of students answering an item correctly, so higher values indicate easier items.

Difficulty, however, is only half the psychometric story. The other half is discrimination, the ability of an item to separate high achievers from low achievers. Here the two formats were statistically indistinguishable: the mean discrimination index was 0.44 for MCQs and 0.41 for SAQs, a difference the researchers found non-significant. Both values fall in the range typically regarded as good discrimination, meaning that neither format was merely rewarding lucky guessing or penalizing careless wording. Both formats also demonstrated excellent internal consistency, with Cronbach’s alpha of 0.85 for the MCQ examination and 0.83 for the SAQ examination, indicating that the items within each test were measuring a coherent underlying construct rather than a scatter of unrelated facts.

Perhaps the most consequential finding concerned what happened at the pass-fail boundary. Concordance between the two formats was only 68.0 percent, meaning that nearly one in three students received a different verdict depending on which paper was consulted. Significantly more students passed the MCQ examination while failing the SAQ examination, a pattern consistent with the format difference in difficulty. For educators, this is a warning against treating a single format as a complete picture of competence: a student who clears the MCQ threshold may still be harboring gaps that only free-recall testing exposes.

The correlational analysis added further nuance. Item difficulty correlated strongly across the two formats, with a coefficient of 0.71, suggesting that topics that were hard in one format tended to be hard in the other. The underlying content, in other words, has an intrinsic difficulty that survives the change of format. Discrimination, by contrast, showed a weak and non-significant correlation of just 0.01, implying that which items best separate strong from weak students depends substantially on the format used to ask them. A question that discriminates well as an MCQ may discriminate poorly as an SAQ, and vice versa.

The study also probed how question style within the SAQ format shaped results. Clinical SAQs, which require students to apply knowledge to patient-oriented scenarios, were associated with lower performance among low-achieving students. This suggests that application-level questions widen the gap between the strongest and weakest performers, which may be desirable in a formative assessment designed to identify students who need early support. The researchers worked with structured short answer questions and very short answer questions, formats that constrain the scope of acceptable answers and thereby improve marking reliability without sacrificing the free-recall advantage.

For formative assessment specifically, the implications are practical. Because formative tests exist to guide learning rather than to certify competence, the higher difficulty of SAQs can be turned into a diagnostic asset: they reveal precisely where recall and reasoning break down, information that a well-guessed MCQ answer can mask. Meanwhile, the efficiency and objectivity of MCQs make them ideal for covering broad content quickly. The authors conclude that both formats demonstrated sound psychometric properties and that using them together may enhance assessment quality in undergraduate medical education, a recommendation that aligns with the growing consensus that assessment format should be matched deliberately to the cognitive skill being measured.

The study’s limitations are those inherent to a single institution, single module, and single cohort, and the results come from a formative rather than a high-stakes summative context, where student motivation and preparation may differ. Still, the design’s content-matched structure gives the comparison unusual internal validity for this kind of educational research. As medical schools worldwide expand computer-based testing and reconsider the balance between recognition-based and recall-based assessment, this Egyptian dataset offers a rare like-for-like measurement of what each format actually delivers: MCQs that are easier but equally discriminating, SAQs that are harder but psychometrically sound, and a persistent one-in-three chance that the two will disagree about who has mastered the material.

Subject of Research: Comparative psychometric analysis of multiple-choice and short answer question formats in formative assessment of first-year medical students

Article Title: Comparative analysis of short answer questions and multiple-choice questions in formative assessment of first-year medical students

Article References: Ammar, A. M., El Naggar, H. M., Ahmed, M. B., Elsayed, Z., & Tawfeek, G. M. (2026). Comparative analysis of short answer questions and multiple-choice questions in formative assessment of first-year medical students. BMC Medical Education, 26(1), Article 1518. https://doi.org/10.1186/s12909-026-10344-1

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10344-1

Keywords: medical education, multiple-choice questions, short answer questions, formative assessment, psychometrics, item difficulty, item discrimination, Cronbach's alpha, first-year medical students, assessment formats, Ain Shams University, BMC Medical Education