The answer, published by Volkan Duran, Abdulhakim Tuğluk and Murat Ali Karavelioğlu, is sobering and instructive in equal measure. Of the five content-analytic dimensions the team tracked across the novel’s 44 chapters, only two met a pre-specified reliability criterion. Emotional valence—the dominant positive or negative tone of each chapter—reached an intraclass correlation coefficient of 0.79, with a 95% confidence interval from 0.64 to 0.88. Empirical reasoning, the concrete sensory and bodily mode of thinking, came in at 0.70, with a wider but still acceptable interval of 0.48 to 0.83. The remaining three dimensions—metaphorical reasoning at 0.50, hypothetical reasoning at 0.42 and analogical reasoning at 0.31—fell below the 0.60 threshold and were formally withdrawn from interpretation.

That act of withdrawal is the study’s most distinctive feature. In much AI-assisted content analysis, dimensions that score poorly on reliability are quietly reported anyway, their numbers presented as if they described the text rather than the quirks of a particular scoring procedure. Here, the authors did the opposite. The values for the three failed dimensions are reported only as properties of the adjudicated dataset—the specific set of scores produced by the AI model and then reviewed and finalized by two human experts. They are explicitly not treated as findings about the novel itself, because no independent reader could reproduce them.

The theoretical motivation for the analysis comes from psychological accounts of how affective state couples to mode of reasoning. Pain, according to this framework, biases processing toward the immediate sensory present: the body’s signals crowd out abstraction, and construal becomes concrete, empirical, anchored in what can be felt. Distance from threat, by contrast, supports abstract, comparative and possibility-oriented thinking—the mind is free to imagine alternatives, construct analogies and entertain hypotheticals. An extended first-person illness narrative, the researchers reasoned, offers a complementary source of description for this coupling: if the theory holds, chapters dominated by negative affect should also be chapters dominated by concrete, sensory and bodily representation.

The methodological design was built to test that idea without contaminating it. Each of the 44 chapters was characterized for dominant emotional valence and for four reasoning modes—metaphorical, analogical, hypothetical and empirical—by an AI-assisted first-pass scoring procedure using a GPT-4-class model. All AI-generated candidate scores were then reviewed, adjudicated and finalized by two human experts, so the final dataset reflected human consensus judgments. But because that adjudicating procedure exposes the human readers to the model’s proposals, it risks a subtle form of anchoring: the experts may unconsciously gravitate toward the machine’s candidates, a phenomenon psychologists call automation bias. To guard against it, the team recruited two further readers who rescored all 44 chapters working independently and blind to both the model’s candidates and the adjudicated values.

The results of that blind replication are the study’s core contribution. The association between the two surviving dimensions replicated: emotional valence and empirical, sensory-bodily content co-varied inversely. In the blind scoring, the Spearman correlation was −0.54, with a Pearson coefficient of −0.49 and a value of −0.57 once corrected for the unreliability of the measures. In the adjudicated dataset, the same association was stronger, at −0.78. The direction and rough magnitude held across both scoring procedures; only the strength differed, which is exactly the pattern one expects when the blind scores are noisier but unbiased. The negative affective baseline of the novel was likewise reproduced by every score set: 32, 39 and 37 of the 44 chapters were scored negative across the three procedures, a striking consensus about a text whose narrator lives under the shadow of amputation.

The fate of the three withdrawn dimensions tells the complementary story. Associations involving metaphorical, hypothetical and analogical reasoning differed substantially across scoring procedures: one changed sign, one increased in magnitude, and one approached zero. In other words, whatever relationship the adjudicated dataset suggested for these dimensions was an artifact of the scoring process, not a stable property of the text. Had the team reported only the adjudicated results, readers would have been presented with findings that independent, blind raters could not reproduce—an outcome that mirrors, in miniature, the replication crisis that has reshaped experimental psychology over the past decade.

The authors are careful about what their one secure finding does and does not mean. The inverse coupling of negative affect and concrete sensory-bodily representation in this text is a replicated descriptive pattern, not a measured psychological association, and no causal or generalizing claim follows from it. A single novel, however canonical, cannot establish how pain shapes cognition in general; it can only show that in this particular narrative, written by an author who himself experienced chronic illness, the two features co-occur chapter by chapter in a way that independent readers agree on. The finding is idiographic—about one text, one narrator, one illness trajectory—and the study’s honesty about that scope is part of its methodological message.

The transparency surrounding the use of AI is equally noteworthy. The authors provide a detailed generative AI disclosure: a GPT-4-class model served strictly as a structured first-pass scoring instrument, Claude was used for language editing of author-drafted English text, for organizational assistance in reordering arguments and improving transitions, and for brainstorming and stress-testing the framing of the discussion. No generative AI tool contributed an original scientific idea, hypothesis, theoretical construct or interpretive conclusion, the authors affirm, and all decisions about content and interpretation rest with them. The blind readers scored the chapters without AI assistance of any kind, and their score files are released unmodified as supplementary material, alongside the analysis scripts and the adjudicated dataset. Pre-editing author drafts are retained and available to the editorial office on request, so the scope of AI editing can be inspected directly against the submitted text.

For a field increasingly tempted to let language models do the reading, the study offers a template rather than a warning shot. The wider contribution, the authors write, is methodological: AI-assisted, human-adjudicated annotation produced a coherent coefficient structure of which only one association survived blind rescoring. Reliability, they argue, should be established before substantive interpretation, not after it. Applied to the growing literature in which machines score texts for sentiment, style, metaphor or argument, the lesson is blunt: a coefficient structure that looks impressive in an adjudicated dataset can dissolve under blind replication, and only the findings that survive both stages deserve the name of result. In this case, what survived is a quiet, humanly verifiable pattern in a century-old Turkish novel—that where the pain is sharpest, the language turns toward the body.

Subject of Research: AI-assisted content analysis and blind replication of emotional valence and reasoning styles in a Turkish illness narrative

Article Title: AI-assisted content analysis of a literary illness narrative: a case study in reliability and replication in Peyami Safa’s Dokuzuncu Hariciye Koğuşu (The Ninth External Ward)

Article References: Duran, V., Tuğluk, A., & Karavelioğlu, M. A. (2026). AI-assisted content analysis of a literary illness narrative: a case study in reliability and replication in Peyami Safa’s Dokuzuncu Hariciye Koğuşu (The Ninth External Ward). BMC Psychology. https://doi.org/10.1186/s40359-026-05773-y

Image Credits: AI Generated

DOI: 10.1186/s40359-026-05773-y

Keywords: AI-assisted content analysis, inter-rater reliability, blind replication, emotional valence, reasoning styles, illness narrative, Peyami Safa, Turkish literature, medical humanities, automation bias, intraclass correlation, BMC Psychology