The study, published in BMC Nursing, was led by Xiaobei Ma and Xueyan Fan, who contributed equally, together with colleagues including Fengli Gao, Xin Liu, Yi Duan, and corresponding author Zhifeng Gao. Rather than relying on a generic nursing checklist, the team constructed a framework tailored specifically to the role of the nurse anesthetist, a profession in which errors in airway management, drug dosing, or vigilance during the post-anesthesia care phase can have immediate and serious consequences. The result is a hierarchical instrument comprising five domains, twenty second-level indicators, and one hundred individual scored elements, each anchored to explicit behavioral criteria.

Building such an instrument required a multi-stage methodology that blends structured expert consensus with quantitative weighting. The researchers began with a literature review and semi-structured interviews with experts, then ran a two-round Delphi consultation, a technique in which a panel of specialists independently rates proposed items and receives feedback between rounds until agreement stabilizes. To determine how much weight each domain should carry in the final score, they applied the analytic hierarchy process, a mathematical method that converts pairwise expert judgments into a consistent set of weights, checking that the judgments did not contradict one another through a consistency ratio threshold.

The preliminary measurement evaluation focused on content validity and reliability. Experts rated the relevance of each item, yielding a scale-level content validity index average of 0.95, very close to the perfect score of 1.0, while item-level modified kappa values, which correct expert agreement for chance, ranged from 0.831 to 1.000. To estimate inter-rater reliability, five nurses were independently scored by two prespecified evaluators at program entry; the total-score intraclass correlation coefficient using the two-way mixed-effects absolute-agreement single-measure definition, ICC(A,1), was 0.889, with a 95 percent confidence interval of 0.388 to 0.988. The authors are candid that the narrow confidence interval’s lower bound reflects the tiny reliability subsample, making this preliminary, low-precision evidence rather than comprehensive validation.

The longitudinal component is where the study becomes genuinely distinctive. The team retrospectively analyzed a census of all eligible nurses entering the training program between August 2016 and August 2025, defining T0 as program entry after pre-employment preparation and T1 through T6 as the completion of each of the first six months. Of 77 screened records, 68, or 88.3 percent, had complete T0 through T6 data and entered the analysis; five nurses resigned during the period and four had at least two consecutive missing monthly assessments. All analyses were performed in R version 4.5.2, with repeated-measures analysis of variance serving as the primary within-evaluator test of change over time.

The trajectory of improvement was striking. Mean self-assessment scores climbed from 35.95 at entry to 96.83 at month six, while faculty ratings rose from 29.20 to 95.32. After applying the Greenhouse-Geisser correction, which guards against violations of the statistical assumption of sphericity in repeated-measures designs, the time effects were highly significant for both perspectives, with P values below 0.001 and partial eta-squared values of 0.977 for self-ratings and 0.984 for faculty ratings, meaning that time explained nearly all of the variance in scores. In practical terms, new nurse anesthetists moved from roughly a third of the maximum score to near-ceiling performance within six months under this framework.

Perhaps the most psychologically interesting finding concerns the gap between self-perception and external judgment. At every single time point, self-ratings exceeded faculty ratings, but the size of that gap changed systematically. A participant-level mixed-effects model, which accounts for repeated observations nested within individuals, estimated the model-based difference between evaluator trajectories as largest at month two, at 9.51 points, and shrinking to just 1.50 points by month six. The time-by-evaluator interaction was significant in a likelihood ratio test yielding a chi-square statistic of 110.67 with six degrees of freedom and P below 0.001. The pattern suggests that new nurses begin with a degree of overconfidence relative to their supervisors’ assessments, then calibrate their self-appraisal as real clinical experience accumulates.

The framework also showed meaningful concurrent associations with established assessment tools. At month three, self-ratings correlated with written examination scores at a Spearman coefficient of 0.815 and faculty ratings at 0.582, while the corresponding correlations with objective structured clinical examination scores, the standardized practical tests widely used in medical education, were 0.580 and 0.558. By month six these associations remained positive, at 0.752 and 0.532 for the written examination and 0.689 and 0.340 for the OSCE. Notably, at program entry the correlations with the written examination were negligible, which makes sense: before any anesthesia-specific training, general knowledge tests say little about how a nurse will perform in the operating room, so the framework’s scores capture something the written test does not.

The authors also report an exploratory analysis in a separate prespecified five-person subset examining whether every individual improved from T0 to T6. All five showed positive change, but an exact two-sided sign test produced a P value of 0.0625, just shy of the conventional 0.05 threshold, a mathematical inevitability when only five participants are tested, since the smallest achievable two-sided P value for a sign test with five observations is exactly 0.0625. The team treats this honestly as suggestive rather than confirmatory, and the supplementary materials go further, documenting acceptability questionnaires completed by all 68 respondents, sensitivity analyses, and exploratory safety surveillance, an unusually transparent level of statistical disclosure for a single-centre education study.

What the study does not claim is perhaps as important as what it does. The authors explicitly state that their results provide preliminary, low-precision measurement evidence rather than comprehensive validation or causal proof that the training program itself drove the score increases, and they call for prospective multicenter evaluation before the framework is used for any high-stakes decisions such as hiring, promotion, or certification. Still, the implications are considerable. With anesthesia teams worldwide facing persistent staffing pressures, a validated, behaviorally anchored, month-by-month competency map could help educators target support precisely where new nurses lag, and the finding that self-assessment converges with faculty judgment over time offers a measurable signal of professional maturation. If larger multicenter studies replicate these trajectories, the humble scoring rubric developed in a Beijing hospital could become a template for how surgical specialties everywhere turn the anxious first months of a new clinician’s career into a quantified, coachable journey.

Subject of Research: Development and longitudinal evaluation of a competency assessment framework for newly recruited nurse anesthetists

Article Title: Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists

Article References: Ma, X., Fan, X., Gao, F., Liu, X., Duan, Y., & Gao, Z. (2026). Development, preliminary measurement evaluation, and longitudinal application of a competency assessment framework for newly recruited nurse anesthetists. BMC Nursing. https://doi.org/10.1186/s12912-026-05424-y

Image Credits: AI Generated

DOI: 10.1186/s12912-026-05424-y

Keywords: nurse anesthetists, competency framework, Delphi method, longitudinal assessment, mixed-effects model, content validity, inter-rater reliability, OSCE, nursing education, self-assessment, analytic hierarchy process, BMC Nursing

The research team, led by Hong Wu, Quzhi Liu, and colleagues at Hohai University in Nanjing, designed a cross-sectional online survey around two parallel 11-item experience scales. Each item probed a distinct dimension of perceived support quality: acceptance, understanding, goal alignment, collaboration, professional competence, response efficiency, safety and privacy, trust, reduced concern about being judged, willingness for deep self-disclosure, and disclosure of negative emotions. Because every participant rated both modalities on identical items, the design enabled paired statistical comparisons that control for the stable personality traits and life circumstances each individual brings to any evaluation. This within-person approach matters enormously in this field, because people who choose to use chatbots may differ systematically from people who seek therapy, and those pre-existing differences can masquerade as modality effects in cruder between-group studies.

The statistical machinery behind the study was deliberately layered. The primary analysis used a paired-samples t test on a 10-item composite score, constructed after the researchers discovered that the response-efficiency item was not strictly measurement-equivalent across the two modalities, meaning that a chatbot’s instant reply and a therapist’s considered response may simply not measure the same underlying quality. To guard against distortion from this problematic item, the team removed it from both modality scores and re-ran the comparison. They also applied paired Wilcoxon signed-rank tests with Monte Carlo two-tailed p values and Holm correction to the item-level data, a procedure that controls the inflated risk of false positives when many comparisons are tested simultaneously. Finally, a repeated-measures linear mixed-effects model served as a sensitivity analysis, explicitly modeling the within-person paired structure and testing whether the size of the modality difference varied according to individual characteristics such as attachment style.

The headline finding was a small but statistically robust overall advantage for AI-based support. On the 10-item composite, AI support scored a mean of 57.77 with a standard deviation of 6.55, while human counseling scored 56.80 with a standard deviation of 6.75. The mean difference, calculated as human minus AI, was −0.97 points, with a 95 percent confidence interval spanning −1.73 to −0.21, a paired t value of −2.53 on 232 degrees of freedom, and a p value of .012. The standardized effect size was modest, dz = 0.17, which the researchers emphasize is a small difference. Yet its persistence after the potentially non-equivalent response-efficiency item was excluded suggests the pattern is not merely an artifact of comparing a chatbot’s instant replies with a therapist’s slower turnaround. At the item level, AI support received higher ratings on five exploratory indicators, while human counseling won on exactly two: mutual understanding and professional competence.

That two-item advantage for human counselors is arguably the most psychologically meaningful part of the results. Mutual understanding and professional competence are precisely the qualities that define the therapeutic relationship in classical accounts of counseling, and the study suggests they remain the territory where human practitioners are perceived to excel, even among people who also use and generally rate AI support favorably. Conversely, the domains where AI scored higher map onto what the researchers describe as low-threshold and low-social-threat experiences. Talking to a chatbot appears to reduce the fear of being judged, makes deep self-disclosure feel safer, and lowers the barrier to seeking support in the first place. For people who find the prospect of sitting across from a professional intimidating, a machine that never sighs, never frowns, and never files a clinical note about your worst thoughts may feel like a genuinely safer opening move.

Attachment theory provided the study’s central individual-difference framework, and the regression results revealed a coherent pattern. Attachment anxiety, characterized by a heightened need for reassurance and fear of rejection, positively predicted perceived experience quality in both modalities, meaning anxiously attached participants rated both AI support and human counseling more favorably. Attachment avoidance, characterized by discomfort with closeness and interdependence, negatively predicted both outcomes, so avoidantly attached participants rated both forms of support less favorably. In other words, the same relational dispositions color how people experience support whether it comes from a person or a machine, suggesting that chatbots do not escape the attachment dynamics that shape human relationships. Notably, specialized mental health and therapy chatbot use was associated with higher AI-experience scores, while associations involving shorter AI-use duration were less consistent, hinting that purpose-built tools may deliver a better experience than general-purpose assistants pressed into therapeutic duty.

The mixed-effects sensitivity analysis added a subtler layer to the attachment findings. The interaction between modality and attachment avoidance was statistically significant, indicating that the relative gap between AI-based support and human counseling was not uniform across participants but varied according to how avoidant they were in close relationships. Although the study’s published summary does not detail the direction of this interaction, its existence implies that attachment avoidance may shape which modality feels more comfortable, a possibility with direct clinical implications for matching people to support formats. Meanwhile, the hierarchical regression on the human-minus-AI difference score showed improved model fit after the attachment variables were added, but the final full model was not significant at the omnibus level, so the researchers treat the difference-score analysis as suggestive rather than conclusive.

The authors are unusually explicit about the limits of their evidence, and these caveats deserve as much attention as the findings. The design was cross-sectional and entirely self-report, so the data capture perceived experiences rather than objectively measured therapeutic outcomes, and no causal claims about which modality actually helps more can be sustained. The distributions showed restricted upper-end variability, with medians generally high and often identical across modalities, which compresses the statistical room to detect differences. The AI tools participants reported using were heterogeneous, spanning whatever chatbots they happened to try, and the 11-item experience indicators were newly developed for this study and remain exploratory rather than validated instruments. The researchers also caution that the sensitivity analysis relied on a composite score derived from ordinal item ratings, a measurement choice that warrants care when interpreting the small mean difference that survived.

What the study ultimately delivers is a differentiated map rather than a verdict. Among dual users, perceived experiences differed by domain rather than showing a uniform preference for either machines or humans. AI-based psychological support appears to excel as a low-barrier, judgment-free channel that encourages disclosure, while human counseling retains perceived advantages in mutual understanding and professional competence, the relational core of formal therapy. For clinicians, policymakers, and the rapidly growing industry building mental health chatbots, the implication is that the two modalities may be complements rather than competitors: AI could serve as an accessible first step or an adjunct between sessions, while human professionals remain the reference point for depth and expertise. As conversational AI becomes more capable and more widely adopted, studies that compare experiences within the same individuals, as this one does, will be essential for understanding not just whether people use these tools, but how the tools make them feel, and for whom they work best.

Subject of Research: Perceived experiences of AI-based psychological support compared with human counseling among adults who have used both modalities

Article Title: Differences in perceived experiences of AI-based versus human psychological support among dual users: paired comparisons, hierarchical regression, and repeated-measures sensitivity analysis

Article References: Wu, H., Liu, Q., Shi, Y., Yang, M., Zhang, J., & Zhang, Y. (2026). Differences in perceived experiences of AI-based versus human psychological support among dual users: paired comparisons, hierarchical regression, and repeated-measures sensitivity analysis. BMC Psychology. https://doi.org/10.1186/s40359-026-05676-y

Image Credits: AI Generated

DOI: 10.1186/s40359-026-05676-y

Keywords: conversational AI, mental health chatbots, human counseling, attachment theory, self-disclosure, user experience, paired comparison, hierarchical regression, mixed-effects model, psychological support, trust, BMC Psychology