The study proceeded in two phases. Sixteen experts from six Francophone countries—Belgium, Cameroon, Canada, France, Senegal and Switzerland—were invited to a Delphi process based on a 33-item French-language adaptation of the original tool. Thirteen panellists responded in Round 1, an 81% response rate, and 12 responded in Round 2, a 92% response rate. Three consensus rounds examined cross-context comprehension of the French wording, the necessity of each item and its direct observability in a recorded obstetric simulation. An item was considered absolutely necessary when at least 50% of respondents selected that option, while direct observability required at least 60% agreement among the 12 Round 2 respondents.

Nine of the 33 items were reworded during the phrasing round, and none were deleted. The necessity round excluded 15 items, merged three into one leadership item and retained 15 unchanged, producing a 16-item grid across the original tool’s five sections. All items met the direct-observability threshold except the Two-Challenge Rule, which was retained by a documented coordination-team decision. The investigators therefore described the saT-TPOT as a consensus-based Francophone obstetric adaptation whose items were filtered for direct observability in simulation.

In the preliminary evaluation, seven simulation experts independently rated recordings of two postpartum-haemorrhage simulations. They were blinded to instructors’ classifications of the recordings as below expectations or at or above expectations. Ratings used a five-point scale ranging from 1, “far below,” to 5, “far above” expectations, with 0 indicating that an item was not applicable. Rating distributions differed between the two recordings for 11 of the 16 items, in the expected direction for 10, while five items had identical distributions. These findings provide preliminary evidence of discriminant capacity, but the abstract states that the study established content validity only; formal inter-rater reliability estimation and convergent validity against an established obstetric instrument remain necessary next steps.

Subject of Research: Development and preliminary evaluation of a Francophone obstetric simulation team-performance observation tool

Article Title: Obstetric, Francophone adaptation of the TeamSTEPPS team performance observation tool for simulation (saT-TPOT): a Delphi consensus study with a preliminary rater-based evaluation

Article References: Desseauve, D., Daelemans, C., Farin, A., & Ceccaldi, P.-F. (2026). Obstetric, Francophone adaptation of the TeamSTEPPS team performance observation tool for simulation (saT-TPOT): a Delphi consensus study with a preliminary rater-based evaluation. BMC Medical Education. https://doi.org/10.1186/s12909-026-10466-6

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10466-6

Keywords: Francophone obstetric simulation, TeamSTEPPS, saT-TPOT, Delphi consensus, content validity, discriminant capacity, inter-rater reliability, postpartum haemorrhage, simulation assessment, Obstetric, Francophone, adaptation

The answer, published by Volkan Duran, Abdulhakim Tuğluk and Murat Ali Karavelioğlu, is sobering and instructive in equal measure. Of the five content-analytic dimensions the team tracked across the novel’s 44 chapters, only two met a pre-specified reliability criterion. Emotional valence—the dominant positive or negative tone of each chapter—reached an intraclass correlation coefficient of 0.79, with a 95% confidence interval from 0.64 to 0.88. Empirical reasoning, the concrete sensory and bodily mode of thinking, came in at 0.70, with a wider but still acceptable interval of 0.48 to 0.83. The remaining three dimensions—metaphorical reasoning at 0.50, hypothetical reasoning at 0.42 and analogical reasoning at 0.31—fell below the 0.60 threshold and were formally withdrawn from interpretation.

That act of withdrawal is the study’s most distinctive feature. In much AI-assisted content analysis, dimensions that score poorly on reliability are quietly reported anyway, their numbers presented as if they described the text rather than the quirks of a particular scoring procedure. Here, the authors did the opposite. The values for the three failed dimensions are reported only as properties of the adjudicated dataset—the specific set of scores produced by the AI model and then reviewed and finalized by two human experts. They are explicitly not treated as findings about the novel itself, because no independent reader could reproduce them.

The theoretical motivation for the analysis comes from psychological accounts of how affective state couples to mode of reasoning. Pain, according to this framework, biases processing toward the immediate sensory present: the body’s signals crowd out abstraction, and construal becomes concrete, empirical, anchored in what can be felt. Distance from threat, by contrast, supports abstract, comparative and possibility-oriented thinking—the mind is free to imagine alternatives, construct analogies and entertain hypotheticals. An extended first-person illness narrative, the researchers reasoned, offers a complementary source of description for this coupling: if the theory holds, chapters dominated by negative affect should also be chapters dominated by concrete, sensory and bodily representation.

The methodological design was built to test that idea without contaminating it. Each of the 44 chapters was characterized for dominant emotional valence and for four reasoning modes—metaphorical, analogical, hypothetical and empirical—by an AI-assisted first-pass scoring procedure using a GPT-4-class model. All AI-generated candidate scores were then reviewed, adjudicated and finalized by two human experts, so the final dataset reflected human consensus judgments. But because that adjudicating procedure exposes the human readers to the model’s proposals, it risks a subtle form of anchoring: the experts may unconsciously gravitate toward the machine’s candidates, a phenomenon psychologists call automation bias. To guard against it, the team recruited two further readers who rescored all 44 chapters working independently and blind to both the model’s candidates and the adjudicated values.

The results of that blind replication are the study’s core contribution. The association between the two surviving dimensions replicated: emotional valence and empirical, sensory-bodily content co-varied inversely. In the blind scoring, the Spearman correlation was −0.54, with a Pearson coefficient of −0.49 and a value of −0.57 once corrected for the unreliability of the measures. In the adjudicated dataset, the same association was stronger, at −0.78. The direction and rough magnitude held across both scoring procedures; only the strength differed, which is exactly the pattern one expects when the blind scores are noisier but unbiased. The negative affective baseline of the novel was likewise reproduced by every score set: 32, 39 and 37 of the 44 chapters were scored negative across the three procedures, a striking consensus about a text whose narrator lives under the shadow of amputation.

The fate of the three withdrawn dimensions tells the complementary story. Associations involving metaphorical, hypothetical and analogical reasoning differed substantially across scoring procedures: one changed sign, one increased in magnitude, and one approached zero. In other words, whatever relationship the adjudicated dataset suggested for these dimensions was an artifact of the scoring process, not a stable property of the text. Had the team reported only the adjudicated results, readers would have been presented with findings that independent, blind raters could not reproduce—an outcome that mirrors, in miniature, the replication crisis that has reshaped experimental psychology over the past decade.

The authors are careful about what their one secure finding does and does not mean. The inverse coupling of negative affect and concrete sensory-bodily representation in this text is a replicated descriptive pattern, not a measured psychological association, and no causal or generalizing claim follows from it. A single novel, however canonical, cannot establish how pain shapes cognition in general; it can only show that in this particular narrative, written by an author who himself experienced chronic illness, the two features co-occur chapter by chapter in a way that independent readers agree on. The finding is idiographic—about one text, one narrator, one illness trajectory—and the study’s honesty about that scope is part of its methodological message.

The transparency surrounding the use of AI is equally noteworthy. The authors provide a detailed generative AI disclosure: a GPT-4-class model served strictly as a structured first-pass scoring instrument, Claude was used for language editing of author-drafted English text, for organizational assistance in reordering arguments and improving transitions, and for brainstorming and stress-testing the framing of the discussion. No generative AI tool contributed an original scientific idea, hypothesis, theoretical construct or interpretive conclusion, the authors affirm, and all decisions about content and interpretation rest with them. The blind readers scored the chapters without AI assistance of any kind, and their score files are released unmodified as supplementary material, alongside the analysis scripts and the adjudicated dataset. Pre-editing author drafts are retained and available to the editorial office on request, so the scope of AI editing can be inspected directly against the submitted text.

For a field increasingly tempted to let language models do the reading, the study offers a template rather than a warning shot. The wider contribution, the authors write, is methodological: AI-assisted, human-adjudicated annotation produced a coherent coefficient structure of which only one association survived blind rescoring. Reliability, they argue, should be established before substantive interpretation, not after it. Applied to the growing literature in which machines score texts for sentiment, style, metaphor or argument, the lesson is blunt: a coefficient structure that looks impressive in an adjudicated dataset can dissolve under blind replication, and only the findings that survive both stages deserve the name of result. In this case, what survived is a quiet, humanly verifiable pattern in a century-old Turkish novel—that where the pain is sharpest, the language turns toward the body.

Subject of Research: AI-assisted content analysis and blind replication of emotional valence and reasoning styles in a Turkish illness narrative

Article Title: AI-assisted content analysis of a literary illness narrative: a case study in reliability and replication in Peyami Safa’s Dokuzuncu Hariciye Koğuşu (The Ninth External Ward)

Article References: Duran, V., Tuğluk, A., & Karavelioğlu, M. A. (2026). AI-assisted content analysis of a literary illness narrative: a case study in reliability and replication in Peyami Safa’s Dokuzuncu Hariciye Koğuşu (The Ninth External Ward). BMC Psychology. https://doi.org/10.1186/s40359-026-05773-y

Image Credits: AI Generated

DOI: 10.1186/s40359-026-05773-y

Keywords: AI-assisted content analysis, inter-rater reliability, blind replication, emotional valence, reasoning styles, illness narrative, Peyami Safa, Turkish literature, medical humanities, automation bias, intraclass correlation, BMC Psychology

The review, conducted by Hamidah Mabruk Alodayni, Fatimah Alodaini, Alun H. Davies and Pasha Normahani, followed PRISMA-ScR guidelines and searched MEDLINE and Embase from 1946 through November 2025. The result is both encouraging and sobering. Seven studies met the inclusion criteria, all published between 2017 and 2022—and every single one came from the same French vascular research group that developed the system. That single-source evidence base colors everything the review found, even as the individual results are strikingly positive.

The Saint-Bonnet classification was born out of frustration. The French College of Teachers in Vascular Medicine (CEMV) recognized that when clinicians were left to describe waveforms in free text, the vocabulary exploded. In one study, 110 vascular physicians describing 20 anonymized waveforms without any framework produced an average of 92.8 distinct descriptive expressions. With the Saint-Bonnet system, that collapsed to just 6.1. The system itself is elegantly simple: eight ordered morphological types, labeled N, A, B, CD, E, O, FA and U, running from normal triphasic flow all the way to the complete absence of signal, each stage mapped to a defined level of haemodynamic severity.

The performance data are impressive. In a multicentre prospective study of 1,033 Doppler waveforms, the Saint-Bonnet system successfully categorized 98.2 percent of them, significantly outperforming the alternative Spronk (91.3 percent) and Descotes and Cathignol (76.8 percent) classifications. Inter-rater reliability among clinicians was moderate, with a kappa of roughly 0.55—decent but not perfect, and varying considerably across individual categories, from 0.24 to 0.87. The reviewers suggest this variability may reflect the system’s youth and the training it requires rather than any inherent flaw.

More tantalizing still are the clinical correlations. A cross-sectional study of 186 symptomatic patients found that Saint-Bonnet waveform grade was independently associated with maximal walking distance, performing even better than the ankle-brachial index (ABI) in multivariable models—a sign that the shape of the waveform itself carries meaningful functional information. Meanwhile, a prospective cohort of 188 patients showed that using the waveform grade to select which ankle artery to measure for ABI produced near-perfect agreement with guideline-standard measurements, with kappa values reaching 1.00 when the two ankle arteries displayed different waveform grades.

The system is also proving compatible with artificial intelligence. In a machine-learning proof of concept, a deep convolutional neural network trained on Saint-Bonnet-labeled waveform images achieved 83.7 percent accuracy and a kappa of 0.79, approaching the human expert consensus value of 0.83. The reviewers caution, however, that this demonstrates technical reproducibility—teaching a machine to mimic the classification—rather than independent validation of the classification’s diagnostic accuracy, since the training ground truth was the classification itself.

To assess methodological quality, the team applied two validated appraisal instruments, AGREE II and AGREE-REX, to the CEMV guideline document that formally codified the classification. The results reveal a striking split personality. AGREE II scores were excellent for Clarity of Presentation (97 percent) and Scope and Purpose (83 percent), confirming that the system is well-defined and beautifully laid out. But Rigour of Development scored just 44 percent and Applicability 40 percent, pointing to an absence of systematic evidence synthesis, weak implementation planning, and minimal conflict-of-interest reporting. Stakeholder Involvement reached 69 percent and Editorial Independence only 50 percent. Notably, no patient involvement appeared anywhere in the system’s development.

AGREE-REX told the opposite story: high scores for Clinical Applicability (6.0) and Implementability (5.5), with Values and Preferences at a moderate 4.3. Clinicians, in other words, find the system genuinely useful and easy to deploy. The reviewers observe that this combination—low developmental rigour paired with high clinical utility—is a familiar pattern for expert-driven frameworks promoted before independent validation, a phenomenon not unique to vascular medicine.

The central caveat is impossible to ignore. Because all seven supporting studies originated from a single, non-independent French group, the entire evidence base is internally consistent but externally unreplicated. No study has tested the classification in other healthcare systems, among international sonographers, or in longitudinal cohorts tracking whether waveform grade predicts clinical outcomes over time. The reviewers call for multicentre international validation, head-to-head comparisons across different healthcare contexts, prognostic studies, and integration into AI-assisted Doppler platforms as the key research priorities ahead.

The verdict, then, is nuanced: the Saint-Bonnet classification is a clinically logical and remarkably practical framework that tames decades of terminological anarchy and shows genuine promise for both bedside assessment and automated interpretation. Its low AGREE II scores reflect gaps in documentation and governance rather than evidence against its clinical performance. But until independent, multicentre studies replicate the French findings, the reviewers conclude, the system should be regarded as promising rather than established—and broader adoption should wait for that confirmation.

Subject of Research: Evidence appraisal of the Saint-Bonnet Doppler waveform classification for peripheral arterial disease

Article Title: Saint-Bonnet Doppler waveform classification system: a scoping review and evidence-based appraisal using AGREE II and AGREE-REX

Article References: Alodayni, H. M., Alodaini, F., Davies, A. H., & Normahani, P. (2026). Saint-Bonnet Doppler waveform classification system: a scoping review and evidence-based appraisal using AGREE II and AGREE-REX. Frontiers in Cardiovascular Medicine, 13, Article 1870182. https://doi.org/10.3389/fcvm.2026.1870182

Image Credits: AI Generated

DOI: 10.3389/fcvm.2026.1870182

Keywords: Doppler waveform, Saint-Bonnet classification, peripheral arterial disease, duplex ultrasound, AGREE II, AGREE-REX, scoping review, vascular medicine, ankle-brachial index, machine learning, inter-rater reliability, clinical guidelines

The problem centers on two small public corpora that recur as evaluation data for objective ABR detection: the PhysioNet earndb tone-pip set and the Mendeley click set. Both are freely available, both are small, and crucially, neither distributed expert annotations of response presence as part of the primary release. Every published detector evaluation on these corpora relied on clinician annotation that the authors commissioned or produced themselves. A reader of the public releases before 2025 therefore had no annotation to use. The one label that can be computed from the released files alone is an automatic proxy, and the natural proxy is the clinical two-replication reproducibility criterion: two independent sub-averages are recorded at each level, and a response is deemed present when they correlate above a threshold. The proxy is attractive because it is objective, cheap, and rooted in clinical practice—but its behavior as a research target had never been tested.

That test became possible when Liu and colleagues, reporting a multi-center comparison of nine deep models trained on 128,123 clinically annotated waveforms, published as supplementary material the audiologist annotations they had commissioned for both public corpora: one invited rater for earndb and six raters for the Mendeley set. A team led by Seyed-Ali Sadegh-Zadeh of the University of Staffordshire mapped those annotations back onto the original recordings, validating the mapping in five independent ways—including exhaustive level-offset grids from −40 to +20 dB, none of which matched as many records as the identity alignment, and direct confirmation from the annotating authors that the published level column is in peak-equivalent SPL rather than sensation level. On identical data, the team constructed the reproducibility surrogate and four detectors spanning the capacity range: the instrument statistic wnsfmp, logistic regression, gradient boosting, and a multilayer perceptron.

The first result is stark. The surrogate agreed with expert judgment at Cohen’s κ of just 0.167 on earndb and 0.248 on the Mendeley set—against a median κ of 0.691 among the six Mendeley audiologists themselves. Disagreement was strongly asymmetric: on earndb, the surrogate declared a response present where the expert declared it absent in 92 of 204 conditions, the reverse in only 5. The divergence concentrated almost entirely at 1 kHz, where the surrogate labeled 82 percent of conditions response-present while the expert found responses in only 16 percent. Whatever the two quantities register, they are not registering the same thing.

Why does the proxy fail so badly? The answer is structural rather than incidental. The expected correlation between two replicates is, in algebraic terms, SNR divided by one plus SNR—for whatever component the replicates share, response or not. Because any common component, including stimulus-locked residue or slow shared activity, raises the correlation, the surrogate can declare a response present in the complete absence of a brainstem response. A simulation of 2,000 response-free conditions reproduced the effect: at the clinical threshold, the surrogate called 53 percent of these response-free conditions present. On the recordings themselves, wnsfmp—a single-replicate signal-to-noise statistic read from the record header, involving no learning—predicted the surrogate with AUROC 0.789 restricted to the 250 sub-averages in which the expert saw no response. The proxy, in other words, largely leaks the input: it is a thresholded, noisy realization of the very quantity the instrument statistic estimates.

This degeneracy explains a mechanical rank reversal. Under the surrogate label, wnsfmp ranked first with a pooled AUROC of 0.827, ahead of all learned detectors. Under expert annotation on the same recordings, wnsfmp fell to last place at 0.662, while the multilayer perceptron rose from 0.713 to 0.898 and logistic regression from 0.752 to 0.837. The label-by-detector interaction for wnsfmp versus the perceptron was −0.350, resolved by an exact subject-level permutation test at p = 0.031. A ranking produced under the surrogate is therefore not evidence about detectors at all—it is evidence that the surrogate leaks the input, which the authors regard as the article’s central finding about the proxy.

The team then asked through which channel the label changes a study’s conclusion, using a 2 × 2 factorial over training label and evaluation label. The answer is chiefly through training. Detectors trained on the proxy and scored against expert judgment lost 0.139 to 0.304 pooled AUROC—for the perceptron, the loss was positive in every informative listener, with an exact p of 0.031. Within a listener and a carrier frequency, the two trainings ranked records about equally well; the cost arises in comparisons across listeners and frequencies, which is exactly what a fixed clinical decision threshold depends on. For the highest-capacity detector, the two labels defined nearly orthogonal prediction problems: cross-label AUROC values of 0.594 and 0.574 sit close to chance. Scoring fixed predictions against either label, by contrast, did not reorder the detectors at a resolvable magnitude.

The rater analysis extends the point to studies that do use expert annotation. Rerunning the full evaluation against each of the six Mendeley audiologists in turn moved AUROC by 0.065 to 0.082 within a detector—nearly as much as replacing one detector with another, and below the smallest difference the eight-listener design can resolve, which the team’s design analysis puts at 0.096 to 0.136. Resolving a true difference of 0.05 AUROC between two detectors at 80 percent power would require roughly 30 to 59 listeners; a difference of 0.03, the order of many reported gaps, would need 82 to 164. The authors are careful about what this does and does not establish: they claim neither label is correct. Expert visual judgment is itself a proxy for physiological response presence, with documented variability, and the single earndb annotator’s 1-kHz judgments are conservative relative to physiological expectation. The claim is conditional and, they argue, more useful: the choice of target determines the conclusion, the two available targets disagree profoundly, and a reader given only a ranking cannot tell which target produced it.

Two further results sharpen the practical message. Label-free baselines performed disturbingly well against the expert label: stimulus level alone reached an AUROC of 0.931 within 1 kHz, exceeding every waveform-based detector at that frequency, and carrier frequency alone reached 0.747 pooled. A detector evaluated on pooled data has not demonstrated waveform-derived information until it beats the stimulus parameters, which the operator knows without any analysis. Frequency-stratified analysis also exposed a pooling artifact: wnsfmp scored 0.734 within 1 kHz and 0.768 within 4 kHz yet only 0.662 pooled, below either stratum—an instance of Simpson’s paradox driven by the very different prevalence the expert label assigns to the two frequencies. About 37 percent of the headline interaction dissolves into this pooling structure, a property of label construction itself.

The findings echo a broader pattern in machine learning, where label errors in widely used test sets have been shown to change which model ranks first, and where benchmark ambiguity has been found to make progress partly track annotator conventions. What distinguishes the present case is that the substituted target is not a clinician’s judgment at all but an automatic statistic, that the statistic is a function of the model input, and that a published annotation now exists against which it can be checked. The team released the label mappings and a CPU-reproducible pipeline that regenerates every figure and reported number, and they recommend five inexpensive reporting practices: state label provenance and proxy validity, report the cost of proxy training where it can be measured, report performance against each annotator, report label-free baselines, and stratify by acquisition condition or report a covariate-adjusted AUROC. None is expensive; together, they distinguish a ranking that can be interpreted from one that cannot. On these corpora, the authors conclude, measured detector performance depends on how the label is constructed and who provided it, at magnitudes equal to or larger than the differences between detectors—meaning the small benchmarks are better suited to methodological study than to model comparisons.

Subject of Research: Label construction and rater identity in auditory brainstem response detector benchmarking

Article Title: Label construction can determine measured detector performance on small auditory brainstem response benchmarks

Article References: Sadegh-Zadeh, S.-A., Saadat, S., Barati, M. A., Sharvia, S., & Jarchi, D. (2026). Label construction can determine measured detector performance on small auditory brainstem response benchmarks. Frontiers in Audiology and Otology, 4, Article 1952229. https://doi.org/10.3389/fauot.2026.1952229

Image Credits: AI Generated

DOI: 10.3389/fauot.2026.1952229

Keywords: auditory brainstem response, benchmark validity, label quality, objective detection, inter-rater reliability, evaluation methodology, machine learning, audiology, AUROC, reproducibility, signal-to-noise ratio, Cohen's kappa

The research, conducted by Sheetal Thomas of Parul University, Manjeevan Seera of Monash University Malaysia, and Mridula Goel of BITS Pilani, K. K. Birla Goa Campus, drew on data from more than three hundred participants recruited through universities, factories, government organisations and corporate offices across four districts of Gujarat, India. Each participant completed a sixty-item personality questionnaire covering the five major domains of the Big Five framework — Neuroticism, Extraversion, Openness to Experience, Agreeableness and Conscientiousness — and copied a standard passage known as the London Letter, a text deliberately chosen because it contains a broad range of letters, words and punctuation that makes visual characteristics comparable across writers.

The handwriting specimens were then translated into numbers. A coder with roughly twelve years of specialist handwriting-analysis training, working under the Hilliger Assessment Guidelines supplied by the British Institute of Graphologists, recorded between sixteen and twenty-six binary features for each personality domain. Slant, size, spacing, pressure, margins, baseline direction, letter forms and connecting strokes were each scored as present or absent. The personality outcomes, meanwhile, were computed as continuous scores: the mean of twelve five-point Likert items per domain, yielding values on a scale from one to five rather than the binary personality labels that have complicated much of the earlier literature.

What makes this study unusual is not the data itself but the rigor of its evaluation. The researchers first audited the source workbooks, excluding sixty-two pilot-phase records and screening out incomplete, invalid or invariant questionnaires within each trait separately. That produced analytic samples ranging from 320 to 330 participants per domain. They then compared two quite different machine-learning models — elastic-net regression, a regularised linear method that shrinks weak coefficients, and histogram-based gradient boosting, a non-linear ensemble of constrained regression trees — against a deliberately humbling benchmark: a mean-prediction baseline that ignores the handwriting entirely. Every model was evaluated with five-fold cross-validation repeated five times, generating twenty-five out-of-sample test folds for each model and domain combination.

The results were unambiguous. Neither model consistently improved on the mean baseline in mean absolute error, the study’s primary metric. For Neuroticism, the baseline error was 0.3102 score units while elastic net managed 0.3104 — a difference in the fourth decimal place. Extraversion and Openness told the same story. Agreeableness was the single domain where elastic net edged out the baseline, but the advantage amounted to roughly 0.0011 score points, about one-thousandth of a single point on the five-point scale. Gradient boosting actually performed worse than the baseline in every domain. Most tellingly, mean cross-validated R² was negative for every model-domain combination, meaning that in held-out data the models predicted worse than simply predicting the training-set average.

Permutation tests reinforced the conclusion. When the researchers randomly shuffled the outcome values fifty times and re-evaluated each model, none of the observed model performances stood out from what chance would produce. The smallest permutation p-value was 0.078, for elastic net on Agreeableness, and the rest ranged far higher. The authors are careful to note that these exploratory tests used a limited number of shuffles, but taken together with the negative R² values and the negligible error reductions, the pattern points firmly in one direction: the handwriting-feature matrices contained no reliable information about the questionnaire-derived scores in this sample.

Perhaps the most interesting twist lies in what did work. Two independent, practising graphologists, blinded to all questionnaire data and to each other’s ratings, separately coded twenty-six common handwriting samples. They agreed with one another to a striking degree: mean feature-level percentage agreement ranged from 94.1 percent to 97.8 percent across domains, and mean Cohen’s kappa — a statistic that corrects for chance agreement — ranged from 0.803 for Neuroticism to 0.929 for Conscientiousness. In other words, the features could be identified reproducibly. Yet this reliability did nothing to establish validity. As the authors emphasise, inter-rater reliability and criterion validity answer different questions: one asks whether coders see the same thing, the other whether that thing predicts an independent measure of the construct. A feature can be recognised with near-perfect consistency and still carry no psychological meaning.

The study also probes the quality of the reference measures themselves. Internal consistency of the study-specific questionnaire varied substantially, with Cronbach’s alpha of 0.466 for Neuroticism and 0.675 for Openness — below the commonly cited 0.70 threshold — against 0.805 for Conscientiousness. The authors acknowledge that measurement error of this kind can attenuate any true association and cap the achievable predictive performance, so the low reliabilities limit how strongly the negative result can be generalised. But they are equally clear that weak questionnaire reliability does not rescue handwriting-based claims; it simply constrains conclusions in both directions. The study establishes that these specific coded features did not predict these specific scores, not that no conceivable handwriting representation could ever relate to a better-validated personality instrument.

The context of earlier work matters here too. Parts of the same Gujarat dataset had been used in four previous publications, covering image-based detection of Extraversion- and Neuroticism-associated features, sentiment and risk profiling from handwritten text, and questionnaire-based links between personality and mutual-fund investment preferences. The authors stress that this new paper is not a replication or repudiation of those studies but a different analytical task with a stricter estimand: predicting an independently constructed continuous personality score from the complete feature matrix, out of sample. Earlier feature-detection results, they argue, should not be read as demonstrations that handwriting can recover continuous personality scores in unseen individuals without further evidence.

The broader implications reach well beyond graphology. Personality has been linked in the behavioural-finance literature to risk tolerance, investment preferences and planning behaviour, and that association creates commercial temptation to score personality from cheap, easily collected behavioural traces — including handwriting — in finance, employment, insurance and other high-stakes settings. This study demonstrates why such shortcuts are dangerous. A technically reproducible coding pipeline feeding a properly implemented machine-learning model can still fail completely at the criterion it claims to measure, and algorithmic output lends an unwarranted air of objectivity to constructs that lack validated measurement. The authors recommend independent, preregistered, externally validated evidence, subgroup fairness analyses and clear incremental predictive value before any such tool touches consequential decisions. In publishing a transparent negative result with complete data and code — the coded features are archived on Harvard Dataverse and the full Python workflow is released — the team offers a template for how behavioural AI claims should be tested: clear outcome construction, a simple baseline, repeated out-of-sample evaluation and the honesty to report when the signal simply is not there.

Subject of Research: Predictive validity of manually coded handwriting features for questionnaire-derived Big Five personality scores

Article Title: Predictive validity of manually coded handwriting features for questionnaire derived personality scores

Article References: Thomas, S., Seera, M., & Goel, M. (2026). Predictive validity of manually coded handwriting features for questionnaire derived personality scores. Discover Artificial Intelligence, 6(1), Article 1489. https://doi.org/10.1007/s44163-026-02132-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02132-z

Keywords: graphology, handwriting analysis, Big Five personality, predictive validity, machine learning, elastic net, gradient boosting, cross-validation, inter-rater reliability, negative result, psychometrics, behavioural AI

The research team, led by Heba Wageh Abozaed and colleagues, constructed thirty standardized clinical scenarios representing the kinds of cases a prosthodontist encounters in daily practice. Each scenario was fed to three AI systems: Qwen 3.7 Plus, developed by Alibaba; Claude Sonnet 5, from Anthropic; and Mistral Medium 3.5, the European contender from Mistral AI. Every model generated a treatment plan for every scenario, producing ninety plans in total. This standardized design matters enormously. Earlier studies of AI in dentistry often used ad hoc prompts and single-model assessments, making comparisons nearly impossible. By holding the clinical material constant and varying only the model, the researchers isolated exactly what they wanted to measure: the quality of machine-generated clinical reasoning.

Judging the plans fell to two blinded prosthodontic evaluators who worked independently, neither knowing which model had produced which plan. Each plan was scored on a seven-domain rubric using a five-point Likert scale, covering diagnostic accuracy, treatment selection, treatment sequencing, completeness, adherence to references, patient safety, and overall clinical appropriateness. That structure is worth pausing on, because treatment planning in fixed prosthodontics is not a single decision but a chain of them. A clinician must first interpret the diagnostic information correctly, then choose among crowns, fixed partial dentures, implants, or other restorative pathways, then order those interventions sensibly, all while protecting the patient from harm and grounding every choice in accepted evidence. A model that writes fluent prose but skips a critical step, or recommends an irreversible procedure without adequate justification, would lose points precisely where it matters most.

The statistical machinery behind the comparison was appropriately serious for non-normal, paired ordinal data. The Friedman test, a non-parametric equivalent of repeated-measures analysis of variance, assessed whether the three models differed overall, and it did so emphatically: a chi-square statistic of 35.05 with two degrees of freedom, yielding a p-value below 0.001, and a Kendall’s W of 0.584 indicating a substantial consistency in how the models were ordered across the thirty scenarios. Pairwise contrasts followed using Wilcoxon signed-rank tests with Holm adjustment, a correction that controls the inflated risk of false positives when multiple comparisons are run on the same data. Every pairwise difference survived that correction, meaning the gaps between the models were statistically robust rather than artifacts of chance.

On the raw numbers, Qwen 3.7 Plus took the crown. Its mean total score of 33.82, with a standard deviation of 1.74, topped Claude Sonnet 5’s 32.52 (standard deviation 2.30) and left Mistral Medium 3.5 trailing at 29.68 (standard deviation 2.66). Because the rubric spans seven domains on a five-point scale, the theoretical maximum total is 35, which makes Qwen’s average strikingly close to perfection and highlights just how compressed the top of the scale had become. Within the individual domains, the picture grew more nuanced. Qwen scored highest on completeness and on adherence to references, suggesting it was the most thorough and the most evidence-anchored of the three. Claude, however, achieved the highest patient safety scores, a domain that many clinicians would argue outweighs almost any other. Mistral lagged across all seven domains, a uniform rather than selective weakness.

Then comes the twist that transforms this from a straightforward leaderboard into a genuinely important methodological caution. The two evaluators did not agree with each other very well. The single-measure intraclass correlation coefficient came in at just 0.378, well below conventional thresholds for acceptable reliability, and even the average-measure ICC, which reflects the reliability of the mean of both raters, reached only 0.549. Quadratic-weighted Cohen’s kappa told a similarly sobering story. In plain terms, the two expert judges, both blinded and both using the same rubric, saw the same ninety plans and scored them in ways that diverged substantially.

A sensitivity analysis made the consequences explicit. When each evaluator’s scores were analyzed separately, the model ranking flipped. Evaluator 1 placed Claude marginally above Qwen, while Evaluator 2 ranked Qwen clearly ahead of Claude. The researchers traced this discrepancy to a pronounced ceiling effect in Evaluator 1’s ratings: that evaluator assigned the maximum score to the majority of Qwen and Claude cases, compressing the scale so severely that meaningful differences between the two leading models simply could not register. Ceiling effects are a well-known hazard in performance assessment, and here they were strong enough to overturn the headline finding of the primary analysis. The authors are admirably direct about the implication: these findings do not support universal superiority of any single model.

What, then, can responsibly be concluded? Mistral’s last-place finish across every domain appears to be the most stable result, consistent under both evaluators and across the sensitivity checks. Qwen’s strengths in completeness and reference adherence, and Claude’s edge in patient safety, are real observations from the primary analysis, but the fragility of the inter-rater agreement means any claim that one model is definitively better than the other should be treated as provisional. The study also stops short of testing whether AI-assisted planning actually improves outcomes for patients or learning outcomes for students; educational effectiveness was not directly assessed. What the study does establish is that the leading models can produce treatment plans that experts, on average, judge to be clinically appropriate across a demanding seven-domain rubric, and that the differences among them, while statistically detectable, are modest and measurement-dependent.

The educational implications may prove to be the most durable legacy of this work. The authors suggest that large language models could serve as adjunctive tools for case-based learning and for critical appraisal of treatment-planning decisions, and that framing fits the evidence well. A dental student who generates a plan with an AI system and then interrogates it, asking why a particular sequence was chosen, which alternatives were rejected, and whether the safety considerations were adequately addressed, is engaging in exactly the reflective reasoning that prosthodontic education tries to cultivate. The models become sparring partners rather than oracles. That use case also sidesteps the most dangerous failure mode, which would be a clinician or trainee accepting a machine-generated plan uncritically, particularly in a field where errors can mean irreversible removal of tooth structure.

For the broader conversation about AI in medicine, the study offers a template worth copying. Standardized scenarios, blinded expert evaluation, a multi-domain rubric, non-parametric statistics with multiplicity correction, and, crucially, a sensitivity analysis that exposed how much the conclusions depended on the raters themselves. The uncomfortable but valuable lesson is that evaluating AI is itself a measurement problem, and a rubric that allows one expert to hand out perfect scores to most cases will tell you more about the rubric than about the machine. As these models continue to improve at astonishing speed, the bottleneck in studies like this one may shift from the capability of the AI to the reliability of the humans judging it. Fixing that, perhaps through larger panels of evaluators, refined anchor descriptions for each score point, or harder scenarios that push plans away from the ceiling, will be essential before anyone can say with confidence which artificial intelligence plans a crown best. For now, the honest answer is that the top contenders are close, the measurement tools are shaky, and the most sensible role for AI in the prosthodontic workflow today is as a well-informed colleague whose plans are always worth questioning.

Subject of Research: Comparative evaluation of large language models for fixed prosthodontic treatment planning

Article Title: Comparative evaluation of three large language models for fixed prosthodontic treatment planning: a standardized scenario-based study

Article References: Abozaed, H. W., Alshenaiber, R., Albader, S., Algohar, A., Murayshed, M. S., & Alokla, M. (2026). Comparative evaluation of three large language models for fixed prosthodontic treatment planning: a standardized scenario-based study. BMC Medical Education. https://doi.org/10.1186/s12909-026-10560-9

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10560-9

Keywords: large language models, artificial intelligence, prosthodontics, treatment planning, dental education, Qwen, Claude, Mistral, clinical decision-making, inter-rater reliability, BMC Medical Education, AI in dentistry

The research team, led by prosthodontists Goknur Ozturk and Ozlem Kara together with computer engineering colleagues, built and internally validated an artificial intelligence pipeline that scores soap carvings criterion by criterion, mimicking the way educators actually evaluate student work. Rather than issuing a single holistic grade, the system was trained to assess ten distinct morphological features of the carved tooth, each rated on a scale from zero to ten. That criterion-specific design matters, because the pedagogical value of assessment lies not in a number but in the feedback it generates: a student needs to know whether the incisal edge is misshapen or the cervical line is misplaced, not merely that the carving earned a six.

To create the dataset, the team collected 280 soap-carved replicas of the FDI tooth 21, the maxillary left permanent central incisor, and photographed each specimen from five standardized views, yielding 1,400 images. Three educators then independently scored every carving on all ten criteria. Before any machine learning took place, the researchers quantified how consistently the three human raters judged the work. Using intraclass correlation coefficients, they found average-measure reliability ranging from 0.849 to 0.994 across the ten criteria, figures that indicate strong consensus when multiple expert judgments are pooled. The mean of the three educators’ scores for each criterion became the reference standard against which the artificial intelligence would be measured, an approach that smooths out individual idiosyncrasies and approximates the collective judgment of an examination board.

The machine learning architecture at the heart of the study is ResNet50, a convolutional neural network originally developed for image recognition and now widely repurposed across medicine. The researchers adapted it for multi-output regression, meaning a single model simultaneously predicts ten numeric scores, one per morphological criterion, from the set of five photographs of each carving. Training employed two-stage transfer learning, a technique in which the network first learns general visual features from a large external image dataset and is then fine-tuned on the soap carving images themselves. To make the most of a modest dataset and guard against the accident of a favorable or unfavorable data split, the team used five-fold cross-validation repeated across three random seeds, training ensembles of models whose predictions were aggregated for the final assessment.

The specimens were divided into a development set of 224 carvings used for training and an internal hold-out test set of 56 carvings that the models never saw during learning. Performance on that untouched test set was evaluated with a battery of statistical measures, each reported with bootstrap-derived 95 percent confidence intervals. The ensemble achieved a macro-averaged mean absolute error of 1.204 points on the ten-point scale, with a confidence interval running from 1.059 to 1.374, and a macro-averaged root mean squared error of 1.584. In practical terms, the system’s typical deviation from the averaged human reference was roughly one and a quarter points per criterion, a level of accuracy that could meaningfully flag a carving as strong or weak even if it cannot yet replicate a fine-grained examiner judgment.

Beneath those averages, however, lay substantial variation from criterion to criterion, and the authors are candid about it. The best-predicted feature, criterion 10, showed a mean absolute error of just 0.709, suggesting the network could track educator consensus closely on that aspect of tooth morphology. The worst, criterion 9, had a mean absolute error of 2.029, more than double. Spearman rank correlations between machine predictions and human reference scores ranged from 0.297 to 0.560 across criteria, while absolute-agreement intraclass correlation coefficients between the artificial intelligence and the multi-rater reference spanned only 0.309 to 0.429. The macro-averaged coefficient of determination, or R-squared, came in at 0.223, indicating that the model captured a modest fraction of the variance in human scores. Bland-Altman analysis, the standard tool for assessing agreement between two measurement methods, revealed small mean biases but limits of agreement wide enough to rule out treating the machine as a drop-in replacement for a human grader.

Why does performance vary so much across criteria? The study does not offer a definitive answer, but the pattern is consistent with what is known about both human and machine vision. Some morphological features, such as overall proportions or the presence of well-defined developmental grooves, produce strong visual signals that a convolutional network can readily detect. Others may depend on subtle contours, fine surface texture, or shading cues that are difficult to capture even in standardized photographs, and where even the three human raters may have found it harder to converge. The researchers also examined Gradient-weighted Class Activation Mapping, or Grad-CAM, on eight hold-out specimens to visualize which image regions drove the model’s predictions. The resulting heat maps showed heterogeneous activation patterns, meaning the network did not consistently focus on the same anatomical zones across specimens, a qualitative hint that its internal reasoning had not fully locked onto the features educators consider decisive.

The authors are explicit that these results do not support autonomous summative grading, the high-stakes use case in which an algorithm would assign official examination marks. Interchangeability with educator assessment, they conclude, is not demonstrated by the agreement statistics. What the findings do support is a more modest and arguably more realistic role: human-supervised formative assessment. In that scenario, the system could provide students with rapid, criterion-specific feedback during practice sessions, long before an instructor reviews the work, freeing faculty time for teaching rather than tallying and giving learners a faster feedback loop precisely when they are still developing their skills. The educator remains in the loop as the arbiter of record, with the machine serving as a screening and feedback instrument.

The study’s limitations are as instructive as its results. It was a single-center investigation using a single tooth type, the maxillary left central incisor, and a single assessment medium, standardized photographs rather than the physical specimens an examiner can turn in hand and inspect under varied light. Internal validation on a hold-out set, however rigorously conducted with cross-validation and multiple seeds, cannot substitute for external validation on data from other institutions, other raters, and other carving exercises. The authors state plainly that prospective external validation is required before any implementation in a live curriculum. That caution is a welcome corrective to the breathless tone that often surrounds educational artificial intelligence, and it reflects a maturing research culture in which feasibility studies are expected to define their own boundaries.

Still, the trajectory is notable. Soap carving is only the most traditional of preclinical assessments; the same photographic, multi-rater, criterion-specific framework could in principle extend to wax-ups, digital tooth preparations, and other visually assessed psychomotor tasks across the health professions. The Biruni study, which received no external funding and was approved by the university’s non-interventional research ethics committee, provides an honest baseline: a deep learning system can approach the averaged judgment of expert educators on some criteria, falls short on others, and is not yet fit to grade alone. For students sweating over a bar of soap at two in the morning, the near future may hold an algorithmic teaching assistant that points out a flawed mamelon or an over-carved lingual fossa within seconds, while the final verdict on their craftsmanship stays firmly in human hands.

Subject of Research: Deep learning assessment of preclinical dental soap carvings

Article Title: Development and internal validation of a deep learning system for criterion-specific assessment of preclinical dental soap carvings: a multi-rater study

Article References: Ozturk, G., Kara, O., Purmut, H. E., Birdal, M. B., & Sezer, A. (2026). Development and internal validation of a deep learning system for criterion-specific assessment of preclinical dental soap carvings: a multi-rater study. BMC Medical Education. https://doi.org/10.1186/s12909-026-10554-7

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10554-7

Keywords: artificial intelligence, deep learning, dental education, preclinical assessment, soap carving, automated scoring, ResNet50, inter-rater reliability, medical education, computer vision, formative assessment, Bland-Altman analysis

The study, published in BMC Medical Education, addresses a gap that has persisted despite the TEAM tool’s translation into multiple languages. The original instrument evaluates eleven observable behaviors during emergency team performance, covering domains such as leadership and team cooperation, along with global ratings of team performance and the observer’s overall impression. Until now, no Simplified Chinese version had been formally validated, which meant that emergency departments in mainland China lacked a linguistically and culturally equivalent instrument for structured observation and feedback during training in trauma and resuscitation scenarios.

The translation process followed established cross-cultural adaptation guidelines, a pipeline designed to ensure that a questionnaire means the same thing in a new language as it did in the original. The researchers began with independent forward translations of the scale into Simplified Chinese, then synthesized those versions into a single draft. That draft was back-translated into English by translators blinded to the original instrument, allowing the team to detect any drift in meaning. A multidisciplinary panel of authors, doctors, nurses, and English-language specialists then reviewed the versions side by side, reconciling discrepancies in wording until the Chinese text captured the intent of every item.

Crucially, the team did not stop at linguistic equivalence. They collaborated directly with the original authors of the TEAM questionnaire to maintain conceptual equivalence, and they assessed cultural relevance through Delphi rounds and cognitive interviews with domain experts. Delphi methods involve iterative rounds of structured expert consultation, in which panelists rate and comment on each item until consensus emerges. Cognitive interviews probe whether respondents understand an item the way its authors intended, surfacing ambiguities that a literal translation would never reveal. Together, these steps ensured that the Chinese wording resonated with the clinical realities and communication norms of Chinese emergency teams rather than simply mirroring English phrasing.

The psychometric evaluation centered on video-based assessment. Two trained raters independently scored video recordings of 46 emergency team scenarios involving severe trauma or resuscitation, producing 46 pairs of ratings and 92 individual ratings in total. Using video rather than live observation allowed both raters to see identical performances, isolating the agreement between raters from the noise of real-time perspective differences. The study received ethics approval from the Ethics Committee of the Second Affiliated Hospital of Zhejiang University School of Medicine, and written informed consent was obtained from participating healthcare professionals and from patients or their legally authorized representatives for the research use of recordings.

The results provide strong preliminary support for the new version. Expert assessment of content validity yielded a scale-level content validity index average of 0.98, remarkably close to the perfect score of 1.0, indicating near-unanimous expert agreement that the eleven behavioral items remain relevant and clear in Chinese. The universal agreement index, which is stricter because it requires every expert to rate an item as relevant, stood at 0.82, still comfortably above commonly accepted thresholds. These figures suggest that the adaptation preserved the substance of the original instrument while making it culturally natural for Chinese clinicians.

Internal consistency, which measures how coherently the items hang together as a single scale, was good, with a Cronbach’s alpha of 0.896. This statistic ranges from zero to one, and values above 0.9 are often considered excellent, though excessively high values can signal redundant items. Item-total correlations, which indicate how strongly each item relates to the overall score, ranged from 0.547 to 0.764, meaning every behavioral item contributed meaningfully to the measurement of the underlying construct. No item behaved as an outlier or dragged down the scale’s coherence, an encouraging sign that the translated items measure a unified dimension of team performance.

Inter-rater reliability, arguably the most important property for an observational rating tool, was also strong. The average-measure intraclass correlation coefficient for the total score across the eleven behavioral items was 0.902, with a 95 percent confidence interval of 0.836 to 0.938. An ICC of that magnitude indicates that two independent trained observers watching the same resuscitation would arrive at very similar assessments of team performance, which is exactly what a tool intended for feedback and training requires. At the level of individual items, average-measure ICCs ranged from 0.553 to 0.849, a more varied picture suggesting that some specific behaviors, such as those that are harder to observe or more context-dependent, may be somewhat more challenging to rate consistently than the scale as a whole.

The analysis of floor and ceiling effects revealed a pattern worth noting. No item showed a floor effect, meaning raters did not cluster scores at the bottom of the scale even for weaker team performances. However, six items showed ceiling effects ranging from 15.22 percent to 26.09 percent, indicating that a meaningful share of the observed teams received the maximum rating on those behaviors. Ceiling effects can compress the range of scores at the top end and reduce the instrument’s sensitivity to differences among high-performing teams. The authors suggest that future research should examine the structural validity and responsiveness of the Simplified Chinese version, as well as its performance across other clinical settings, to determine whether the ceiling pattern persists and how best to address it.

The practical implications extend well beyond psychometrics. Structured assessment tools like TEAM are the backbone of simulation-based training in emergency medicine, giving instructors an objective framework for debriefing after mock codes and trauma drills. A validated Simplified Chinese version means that Chinese emergency departments, simulation centers, and nursing schools can now deliver feedback grounded in an instrument whose measurement properties have been demonstrated in their own language and cultural context. It also enables Chinese teams to participate in international research comparing team performance across countries, since scores from the adapted version can be interpreted against the same construct as the original. The researchers, led by Yukun Zhang of the Second Affiliated Hospital of Zhejiang University School of Medicine and Universiti Malaya, together with LiYoong Tang, MeiChan Chong, Shanshan Li, and Yuwei Wang, frame the work as a foundation rather than a finish line. The evidence they report covers content validity, internal consistency, and inter-rater reliability in trauma and resuscitation scenarios; broader claims about responsiveness to training interventions and generalizability to other acute care settings await further study. For now, the study stands as a careful example of how a measurement instrument crosses a linguistic border without losing its meaning, and as a practical gift to the clinicians who train the teams that respond when every second counts.

Subject of Research: Cross-cultural adaptation and psychometric validation of a Simplified Chinese version of the Team Emergency Assessment Measure for emergency team performance

Article Title: Cross-cultural adaptation, translation, and psychometric evaluation of the simplified Chinese version of the Team Emergency Assessment Measure (TEAM)

Article References: Zhang, Y., Tang, L., Chong, M., Li, S., & Wang, Y. (2026). Cross-cultural adaptation, translation, and psychometric evaluation of the simplified Chinese version of the Team Emergency Assessment Measure (TEAM). BMC Medical Education. https://doi.org/10.1186/s12909-026-10529-8

Image Credits: AI Generated

DOI: 10.1186/s12909-026-10529-8

Keywords: TEAM questionnaire, non-technical skills, cross-cultural adaptation, psychometrics, emergency medicine, teamwork, resuscitation, trauma, inter-rater reliability, content validity, medical education, translation

The study, published in BMC Nursing, was led by Xiaobei Ma and Xueyan Fan, who contributed equally, together with colleagues including Fengli Gao, Xin Liu, Yi Duan, and corresponding author Zhifeng Gao. Rather than relying on a generic nursing checklist, the team constructed a framework tailored specifically to the role of the nurse anesthetist, a profession in which errors in airway management, drug dosing, or vigilance during the post-anesthesia care phase can have immediate and serious consequences. The result is a hierarchical instrument comprising five domains, twenty second-level indicators, and one hundred individual scored elements, each anchored to explicit behavioral criteria.

Building such an instrument required a multi-stage methodology that blends structured expert consensus with quantitative weighting. The researchers began with a literature review and semi-structured interviews with experts, then ran a two-round Delphi consultation, a technique in which a panel of specialists independently rates proposed items and receives feedback between rounds until agreement stabilizes. To determine how much weight each domain should carry in the final score, they applied the analytic hierarchy process, a mathematical method that converts pairwise expert judgments into a consistent set of weights, checking that the judgments did not contradict one another through a consistency ratio threshold.

The preliminary measurement evaluation focused on content validity and reliability. Experts rated the relevance of each item, yielding a scale-level content validity index average of 0.95, very close to the perfect score of 1.0, while item-level modified kappa values, which correct expert agreement for chance, ranged from 0.831 to 1.000. To estimate inter-rater reliability, five nurses were independently scored by two prespecified evaluators at program entry; the total-score intraclass correlation coefficient using the two-way mixed-effects absolute-agreement single-measure definition, ICC(A,1), was 0.889, with a 95 percent confidence interval of 0.388 to 0.988. The authors are candid that the narrow confidence interval’s lower bound reflects the tiny reliability subsample, making this preliminary, low-precision evidence rather than comprehensive validation.

The longitudinal component is where the study becomes genuinely distinctive. The team retrospectively analyzed a census of all eligible nurses entering the training program between August 2016 and August 2025, defining T0 as program entry after pre-employment preparation and T1 through T6 as the completion of each of the first six months. Of 77 screened records, 68, or 88.3 percent, had complete T0 through T6 data and entered the analysis; five nurses resigned during the period and four had at least two consecutive missing monthly assessments. All analyses were performed in R version 4.5.2, with repeated-measures analysis of variance serving as the primary within-evaluator test of change over time.

The trajectory of improvement was striking. Mean self-assessment scores climbed from 35.95 at entry to 96.83 at month six, while faculty ratings rose from 29.20 to 95.32. After applying the Greenhouse-Geisser correction, which guards against violations of the statistical assumption of sphericity in repeated-measures designs, the time effects were highly significant for both perspectives, with P values below 0.001 and partial eta-squared values of 0.977 for self-ratings and 0.984 for faculty ratings, meaning that time explained nearly all of the variance in scores. In practical terms, new nurse anesthetists moved from roughly a third of the maximum score to near-ceiling performance within six months under this framework.

Perhaps the most psychologically interesting finding concerns the gap between self-perception and external judgment. At every single time point, self-ratings exceeded faculty ratings, but the size of that gap changed systematically. A participant-level mixed-effects model, which accounts for repeated observations nested within individuals, estimated the model-based difference between evaluator trajectories as largest at month two, at 9.51 points, and shrinking to just 1.50 points by month six. The time-by-evaluator interaction was significant in a likelihood ratio test yielding a chi-square statistic of 110.67 with six degrees of freedom and P below 0.001. The pattern suggests that new nurses begin with a degree of overconfidence relative to their supervisors’ assessments, then calibrate their self-appraisal as real clinical experience accumulates.

The framework also showed meaningful concurrent associations with established assessment tools. At month three, self-ratings correlated with written examination scores at a Spearman coefficient of 0.815 and faculty ratings at 0.582, while the corresponding correlations with objective structured clinical examination scores, the standardized practical tests widely used in medical education, were 0.580 and 0.558. By month six these associations remained positive, at 0.752 and 0.532 for the written examination and 0.689 and 0.340 for the OSCE. Notably, at program entry the correlations with the written examination were negligible, which makes sense: before any anesthesia-specific training, general knowledge tests say little about how a nurse will perform in the operating room, so the framework’s scores capture something the written test does not.

The authors also report an exploratory analysis in a separate prespecified five-person subset examining whether every individual improved from T0 to T6. All five showed positive change, but an exact two-sided sign test produced a P value of 0.0625, just shy of the conventional 0.05 threshold, a mathematical inevitability when only five participants are tested, since the smallest achievable two-sided P value for a sign test with five observations is exactly 0.0625. The team treats this honestly as suggestive rather than confirmatory, and the supplementary materials go further, documenting acceptability questionnaires completed by all 68 respondents, sensitivity analyses, and exploratory safety surveillance, an unusually transparent level of statistical disclosure for a single-centre education study.

What the study does not claim is perhaps as important as what it does. The authors explicitly state that their results provide preliminary, low-precision measurement evidence rather than comprehensive validation or causal proof that the training program itself drove the score increases, and they call for prospective multicenter evaluation before the framework is used for any high-stakes decisions such as hiring, promotion, or certification. Still, the implications are considerable. With anesthesia teams worldwide facing persistent staffing pressures, a validated, behaviorally anchored, month-by-month competency map could help educators target support precisely where new nurses lag, and the finding that self-assessment converges with faculty judgment over time offers a measurable signal of professional maturation. If larger multicenter studies replicate these trajectories, the humble scoring rubric developed in a Beijing hospital could become a template for how surgical specialties everywhere turn the anxious first months of a new clinician’s career into a quantified, coachable journey.

Subject of Research: Development and longitudinal evaluation of a competency assessment framework for newly recruited nurse anesthetists

Article Title: Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists

Article References: Ma, X., Fan, X., Gao, F., Liu, X., Duan, Y., & Gao, Z. (2026). Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists. BMC Nursing. https://doi.org/10.1186/s12912-026-05424-y

Image Credits: AI Generated

DOI: 10.1186/s12912-026-05424-y

Keywords: nurse anesthetists, competency framework, Delphi method, longitudinal assessment, mixed-effects model, content validity, inter-rater reliability, OSCE, nursing education, self-assessment, analytic hierarchy process, BMC Nursing

The course at the heart of the study was no ordinary lecture series. Across a sixteen-week semester, twenty graduate students in a master’s-level information technology program worked with approximately thirty course-specific large language model applications, built through OpenAI’s custom GPT interface. Each application ran pre-scripted prompt sequences that cast the AI in structured instructional roles: simulator, role-player, Socratic dialogue partner, mentor, teammate, or chatbot. Students used these agents to conduct systems analysis, design user interfaces, model business cases, run multi-phase negotiation scenarios, and explore ethical considerations. Crucially, students accessed the tools through shared URLs without being able to view or modify the underlying prompt architecture, which meant the instructional design, not the students’ prompting skill, determined how the AI behaved in each activity.

At the end of the semester, all twenty students completed an anonymous, Institutional Review Board-approved survey containing nine Likert-scale items and eight open-ended questions. The quantitative results were strikingly positive. Most responses to the engagement and motivation items fell into the Strongly Agree category, and fifteen students strongly agreed that generative AI helped them understand complex concepts. Most students also agreed or strongly agreed that the tools supported independent exploration and helped them work more efficiently. The most emphatic result of all concerned the future: eighteen of nineteen respondents selected Definitely Yes when asked whether they would continue using generative AI tools in future courses or projects.

But the researchers are careful, almost insistently so, about what those numbers do not mean. The survey was developed for this exploratory course evaluation and was never psychometrically validated as a scale. It was administered once, after a highly scaffolded, technology-intensive course, with no baseline measures, no comparison condition, and no objective assessments. Self-reported learning gains, the literature reminds us, often track satisfaction and motivation more closely than actual cognitive achievement. The study therefore treats its findings as evidence about students’ perceived experiences, not as proof that the AI-supported course improved learning, critical thinking, retention, or transfer. Responses to the critical-thinking item were notably more varied than the rest, a hesitation that echoes recent meta-analytic work finding no statistically significant average effect of generative AI on metacognition even when other outcomes improve.

The genuinely novel contribution lies in the second half of the study, where the researchers staged a quiet contest between human judgment and machine judgment. Two human researchers independently read all the open-ended responses and, through iterative discussion, built a shared thematic codebook for each of the eight questions. Then two contemporary large language models, GPT-4o and Claude 3.7 Sonnet, were given the same codebook and asked to apply it to the same responses in the same binary format, with no example-coded cases and no corrective feedback. Agreement was quantified using Cohen’s kappa for each coder pair and Fleiss’ kappa across all four coders, benchmarks interpreted through the classic Landis and Koch thresholds.

The results revealed a clear hierarchy of reliability. Human-human agreement ranged from moderate to almost perfect, with kappa values between 0.52 and 0.84. Average human-LLM agreement was lower and more variable, spanning 0.36 to 0.62, and agreement between the two models themselves ranged from 0.30 to 0.70. Across all eight questions, Claude 3.7 Sonnet aligned more closely with the human coders than GPT-4o did when the two human-model kappas were averaged, though the authors warn that this is a snapshot of two proprietary models under one prompt design and a single coding run, not a general ranking of model quality. Four-rater Fleiss’ kappa ranged from 0.42 to 0.63, peaking on the most concrete prompts.

The pattern of where agreement flourished and where it collapsed is perhaps the most scientifically interesting finding. The strongest four-rater agreement, kappa of 0.63, occurred for a concrete retrospective question asking students to describe specific instances where AI helped them understand a topic or solve a problem. Responses to such questions contain observable task cues: coding, debugging, project work, interface design. The weakest agreement, kappa of 0.42, occurred on the two prospective prompts, one asking students to propose desired future features and another asking them to imagine a strategy for a difficult hypothetical assignment. These questions invite multiple reasonable levels of abstraction, from technical feature to learning process to emotional outcome, and the models fragmented broad human themes into narrower subcodes or missed human-identified themes altogether.

Some discrepancies were more than statistical noise. In one question about challenges encountered, both models treated statements reporting no challenge as a substantive theme, while the human coders recognized them as the absence of a challenge code, effectively assigning meaning to non-responses. The authors also point to codebook maturity as a confounding factor: the codebook was developed from the same small set of twenty students’ responses, and human-human kappa was only moderate for four of the eight questions, meaning some theme boundaries were unstable even for trained researchers. Brief open-ended responses compound the difficulty, because meaning is often implicit, compressed, and socially situated. A student’s mention of working faster might signal efficiency, reduced anxiety, or creeping dependence, and disentangling those possibilities requires exactly the contextual judgment that current models lack.

The study frames its instructional findings through the hybrid intelligent feedback framework, which defines effective practice as a pedagogically designed combination of complementary human and artificial cognition. In the course, the LLM applications supplied immediate examples, stepwise prompts, role-play, and iterative suggestions, while the instructor set disciplinary goals, ethical boundaries, and verification expectations, and students remained responsible for evaluating and using what the AI produced. The authors apply this lens retrospectively and with deliberate restraint, distinguishing the pedagogical level of hybridity, where humans and AI shared instructional responsibility, from the methodological level, where humans and models shared a coding workflow. Conflating the two, they argue, would blur claims about classroom experience with claims about research reliability.

The practical upshot is a bounded division of labor rather than a replacement scenario. The researchers conclude that large language models can assist qualitative researchers with screening, discrepancy detection, and code suggestions, but that human researchers remain necessary for contextual interpretation, validation, and adjudication. They recommend that researchers establish mature codebooks with explicit definitions, positive and negative examples, and rules for non-substantive responses before any model comparison; document the provider, model version, prompts, and number of runs for auditability; and treat LLM coding as a reproducibility and sensitivity problem requiring multiple independent runs rather than a single definitive judgment. For educators, the study suggests a sequence of bounded purpose, structured interaction, comparison against disciplinary criteria, and closing reflection on what was accepted, revised, or rejected. In an era when students overwhelmingly intend to keep using these tools, the study’s most durable message may be that the technology works best when it is designed into the course, verified by the learner, and never allowed to have the last word on what learning means.

Subject of Research: Using large language models to thematically code graduate students' reflections on generative AI in coursework

Article Title: Using large language models to analyze student reflections on generative AI in graduate coursework

Article References: Muiznieks, V. J., Anderson, B., & Price, T. (2026). Using large language models to analyze student reflections on generative AI in graduate coursework. Discover Education, 5(1), Article 1065. https://doi.org/10.1007/s44217-026-02231-0

Image Credits: AI Generated

DOI: 10.1007/s44217-026-02231-0

Keywords: generative AI, large language models, graduate education, thematic coding, inter-rater reliability, qualitative research, student reflections, hybrid intelligent feedback, GPT-4o, Claude 3.7 Sonnet, instructional design, AI literacy