New 33-Item Questionnaire Measures How Well Future Health Professionals Understand AI

Artificial intelligence is moving into hospitals and clinics at a pace that has outstripped the tools educators use to judge whether students are actually prepared for it. Diagnostic algorithms, triage assistants and generative chatbots are now part of everyday clinical conversation, yet until recently there has been no rigorously tested way to measure how literate health professions students really are when it comes to AI. A team of researchers from Zhuhai Campus of Zunyi Medical University and The First Affiliated Hospital of Army Medical University in China has now built and validated a dedicated instrument to close that gap, publishing their work in BMC Medical Education.

The new tool, called the AI Literacy Scale, is a 33-item questionnaire organized around four distinct dimensions: Technical Understanding, Ethical Awareness, Practical Ability and Social Impact. Rather than treating AI literacy as a single vague skill, the scale breaks it into measurable components that map onto what students must actually know and do. Technical Understanding probes grasp of how AI systems work, Ethical Awareness covers bias, privacy and responsible use, Practical Ability addresses applying AI tools in learning and clinical contexts, and Social Impact examines the broader consequences of AI for healthcare systems and society.

What makes the study notable to measurement specialists is the theoretical scaffolding behind the item pool. The researchers drew on UNESCO’s AI Competency Framework for Students, the Technological Pedagogical Content Knowledge model and the Kirkpatrick evaluation model, supplementing these frameworks with a systematic literature review and qualitative interviews with students. Items were then refined through a Delphi process, in which expert panels iteratively rate and revise candidate questions, followed by content validation and pilot testing. This layered approach is designed to ensure that every question on the final scale has a defensible theoretical and empirical reason for being there.

To test the instrument, the team collected responses from 473 health professions students and split the sample in two. The first subsample of 241 students was used for exploratory factor analysis, a statistical technique that searches for hidden structure in the data and reveals how many independent dimensions the questionnaire is actually measuring. The second subsample of 232 students was reserved for confirmatory factor analysis, which tests whether the four-factor structure identified in the first half holds up in fresh data, alongside a Rasch rating scale analysis that examines each item individually.

The results were strong by conventional standards. The four-factor solution explained 63.676 percent of the total variance, meaning nearly two-thirds of the differences between students could be captured by the four dimensions. Internal consistency, measured by Cronbach’s alpha, reached 0.979, an exceptionally high value indicating that the items hang together coherently. Confirmatory factor analysis showed acceptable model fit, with a chi-square to degrees of freedom ratio of 2.175, incremental fit index of 0.917, Tucker-Lewis index of 0.908, comparative fit index of 0.916 and root mean square error of approximation of 0.071, all within or near accepted thresholds for a well-fitting measurement model.

The Rasch analysis added a different and arguably more demanding layer of evidence. Unlike classical factor analysis, Rasch modeling converts raw questionnaire responses onto a common interval scale, or logit scale, allowing researchers to compare the ability of each person with the difficulty of each item directly. In this study, person reliability was 0.96 and item reliability was 0.95, both excellent figures that indicate the scale produces highly reproducible measurements of students and stable difficulty estimates for items.

But the Rasch analysis also exposed a limitation that classical statistics would have missed. The person-item map, a visual plot that places students and questions along the same difficulty axis, showed that the average student scored 1.66 logits above the average item difficulty of 0.00 logits. In plain terms, the students were, on average, too skilled for the questions being asked. The scale has relatively few items hard enough to distinguish among the most capable students, a problem known as limited upper-end targeting. The authors suggest several possible explanations: the items may simply not be difficult enough, the sample may have had unusually strong AI backgrounds, students may have accumulated substantial prior AI experience outside the curriculum, or health professions programs may not yet emphasize advanced AI competencies in their teaching.

This finding carries practical consequences for how the scale should be used. The researchers conclude that the AILS is currently better suited to group-level assessment of foundational and intermediate AI literacy than to identifying students with advanced competence. For curriculum designers and education researchers, that means the questionnaire can reliably tell whether a cohort is building baseline AI knowledge and skills, but it cannot yet serve as a fine-grained diagnostic for expertise at the top of the distribution. The team recommends developing more challenging items and validating the scale in independent, multicentre samples before broader deployment.

The study also makes a methodological argument that extends well beyond AI literacy. Combining exploratory and confirmatory factor analysis with Rasch measurement gave the researchers two complementary lenses: factor analysis confirmed the internal structure of the construct, while Rasch analysis revealed how individual items function and whether the difficulty of the test matches the ability of the population. The authors argue that this combined approach illustrates the potential value of Rasch analysis for evaluating item functioning and person-item targeting in health professions education more generally, where many existing questionnaires have never been examined at the item level.

The timing of the work reflects a growing urgency in medical education. As AI reshapes healthcare, new competency requirements are falling on health professionals who must understand, critique and safely deploy these systems, yet assessment instruments tailored to this population have remained scarce and item-level measurement evidence scarcer still. A validated scale gives educators a common yardstick for evaluating whether curricula are working, for benchmarking students across institutions and for tracking progress as AI content is woven into training. The researchers caution that applying the AILS in other professional education domains would require discipline-specific adaptation and revalidation, but within health professions education the instrument now offers one of the most thoroughly tested measures available. The study was conducted in accordance with the Declaration of Helsinki, approved by the Academic Ethics Committee of Zhuhai Campus of Zunyi Medical University, and funded by the university’s Future Distinguished Faculty Development Program.

Subject of Research: Development and validation of an AI literacy questionnaire for health professions students using factor analysis and Rasch measurement

Article Title: Development and validation of the AI literacy questionnaire for health professions students using Rasch analysis

Article References: Development and validation of the AI literacy questionnaire for health professions students using Rasch analysis. (n.d.). https://doi.org/10.1186/s12909-026-10613-z

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10613-z

Keywords: AI literacy, health professions education, Rasch analysis, questionnaire validation, psychometrics, medical education, scale development, confirmatory factor analysis, UNESCO AI Competency Framework, assessment instruments, artificial intelligence, Delphi method