GPT-5 Shows Good Agreement in Medical Undergraduate Portfolio Scoring

In a proof-of-concept study of formative professionalism assessment, GPT-5 showed good agreement with lecturers’ scores for 156 reflectively written portfolios submitted by medical undergraduates. The analysis compared the language model’s rubric-based ratings with lecturer-assigned scores, finding an absolute-agreement, single-measure intraclass correlation coefficient of 0.816, a measure the researchers classified as “good.” The study, reported in BMC Medical Education , examined scores from portfolios previously rated by lecturers using a detailed rubric.

Researchers gave each portfolio to GPT-5 with the same rubric incorporated into a structured prompt developed through a prompt-engineering framework. The portfolio sample represented medical undergraduates, and the assessment addressed professionalism through reflective writing rather than a general test of every form of medical assessment. The design was therefore limited to comparing one AI model’s performance on that specific kind of scored portfolio.

The statistical analysis found a strong linear association between GPT-5 and lecturer scores: Pearson’s correlation was 0.82 and Spearman’s rank correlation was 0.80, with both reported at p < 0.001. The same level of intraclass correlation agreement and strong linear association were observed when individual lecturers’ scores were compared with the corresponding GPT-5 ratings. These results indicate that GPT-5’s scores tracked closely with the lecturer assessments in the analyzed dataset, although they do not show that the model could independently perform every aspect of professionalism assessment.

When GPT-5 evaluated the same portfolios in three separate cycles, its intraclass correlation coefficient was 0.959, classified as “excellent” agreement. Internal consistency across those cycles was also high, with Cronbach’s alpha reaching 0.9862. The repeated-measure analysis describes the stability of outputs under the study’s rubric and structured prompt; it does not establish reliability across other models, prompts, institutions or assessment formats.

Overall, the authors characterized GPT-5 as having potential to complement the scoring of reflectively written portfolios, while stating that its capacity to augment assessment with minimal human intervention should be investigated further.

Subject of Research: Proof-of-concept evaluation of GPT-5 agreement with lecturer ratings for medical undergraduate professionalism portfolios

Article Title: Reliability of ChatGPT in scoring medical undergraduates’ portfolio in professionalism

Article References: Deduwela, R., Kanchuka, R., & Chandratilake, M. (2026). Reliability of ChatGPT in scoring medical undergraduates’ portfolio in professionalism. BMC Medical Education. https://doi.org/10.1186/s12909-026-10343-2

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10343-2

Keywords: GPT-5, medical education, AI assessment, professionalism, reflective writing, medical students, interclass correlation, scoring reliability, generative AI, Reliability, ChatGPT, scoring