High Satisfaction Scores Do Not Prove a Physician Retraining Program Works

A letter published in Academic Psychiatry argues that a high satisfaction score should not be mistaken for evidence that a training program actually works. The correspondence, written by Chujie Chen, addresses a survey of 198 physicians who completed a psychiatric job-transfer training program in Guangdong Province, China, a program designed to retrain physicians to help address a shortage of psychiatrists. The original report concluded from a mean satisfaction score of 4.87, with a standard deviation of 0.29 on a 5-point scale, that the program was effective and well received.

The letter does not dispute the rating itself. The physicians who completed the program did rate it highly, and the questionnaire was thoughtfully constructed. The concern is narrower: the data establish only that completers were satisfied, and three features of the study limit what can be concluded beyond that single number.

The first limitation is a near-ceiling pattern. A mean of 4.87 with a standard deviation of 0.29 is consistent with most respondents choosing the highest score on the scale. If that is what happened, the scale left little room to distinguish more satisfied physicians from less satisfied ones, or to detect differences between subgroups. The letter points to Table 4 of the original paper, where six of seven demographic comparisons showed no difference and every subgroup mean except that of senior physicians was 4.8 or higher. Because the paper does not report the percentage of respondents choosing each score, readers cannot judge for themselves how much discriminating power the scale retained. The reported internal consistency, with a Cronbach’s alpha of 0.96, shows that items were answered consistently, not that the scale could tell respondents apart.

The second limitation concerns who was surveyed. Eligibility required completing the full training, so both the ratings and the reported 100% response rate come only from those who finished. The reaction data therefore cannot speak for anyone who entered the program but disengaged, and any program-wide conclusion needs that qualification.

The third limitation involves data cleaning that cannot be evaluated. Eight of the 198 responses were removed as abnormal data points identified visually from box plots. The paper does not report which variable was screened, the exclusion rule applied, the values removed, or any sensitivity analysis retaining them. The letter notes that a statistically extreme response is not necessarily an invalid one. It also flags an internal inconsistency in the participant flow: removing eight responses from 198 leaves 190, not the 191 reported.

The original study’s own results caution against taking the mean at face value. Only 33.5% of respondents endorsed the statement that no improvements were needed, while 53.4% asked for the rotation arrangements to be optimized. High satisfaction and a majority requesting change can coexist, but together they show that a single mean is an incomplete summary of how the training was received.

The letter does accept the study’s one significant demographic difference: senior physicians reported the lowest overall satisfaction, at 4.65 with the widest spread in the overall-satisfaction column, and were the only subgroup whose mean fell away from the maximum. The concern is not that contrast but the near-uniform overall ratings and the effectiveness claim built upon them.

Under the Kirkpatrick evaluation framework, even a well-functioning Level-1 measure captures participants’ reactions, not learning, behavioral change, or program effectiveness. The letter recommends that future reports present the full distribution of responses or the proportion of respondents at the highest score rather than means alone, include a sensitivity analysis retaining excluded responses, and report how many physicians entered and completed the program. Such steps, it argues, would better align the conclusions with the evidence for a program whose aim is to expand the psychiatric workforce.

Subject of Research: A critique of using participant satisfaction scores to claim the effectiveness of a psychiatric job-transfer training program for physicians in Guangdong Province, China.

Article Title: High Satisfaction is Not Program Effectiveness: Comment on “Satisfaction with Job Transfer Training Among Physicians Transitioning to Psychiatric Practice in Guangdong Province, China”

Article References: Chen, C. (2026). High Satisfaction is Not Program Effectiveness: Comment on “Satisfaction with Job Transfer Training Among Physicians Transitioning to Psychiatric Practice in Guangdong Province, China”. Academic Psychiatry. https://doi.org/10.1007/s40596-026-02469-8

Image Credits: AI Generated

DOI: 10.1007/s40596-026-02469-8

Keywords: physician retraining, psychiatry workforce, job-transfer training, participant satisfaction, Kirkpatrick Level-1, survey methodology, Guangdong Province, Academic Psychiatry, program evaluation, data cleaning, response bias, psychiatrist shortage