AI ‘Surprise’ Scores Reveal a Stable Signature of How Each Person Writes

Every time a person sits down to write—an essay, an answer on a questionnaire, a story about their life—they make thousands of tiny choices about which words to use and in what order. A new study published in Behavior Research Methods suggests that those choices leave behind a measurable, surprisingly stable fingerprint. Researchers led by José Ángel Martínez-Huertas of the National Distance Education University in Madrid report that a quantity borrowed from information theory and computed with large language models, known as surprisal, behaves like a genuine psychological trait: it stays consistent across different writing tasks for the same individual, regardless of what the text is actually about.

Surprisal quantifies how unexpected a word is given the context that precedes it, as judged by a probabilistic model. In psycholinguistics, the concept has a long pedigree: work by John Hale in 2001 and Roger Levy in 2008 formalized the idea that the surprisal of a word predicts how hard it is to process, and subsequent studies linked word-level surprisal to reading times and to neural signatures of comprehension such as the N400 brain response. Words with high surprisal are less predictable and typically demand more cognitive effort; words with low surprisal glide by. Modern large language models, trained to predict the next word in a sequence, provide a convenient and powerful engine for estimating these probabilities at scale.

What makes the new work distinctive is its shift in perspective. Instead of using surprisal to probe how readers process a fixed text, the Spanish team asked whether surprisal could characterize the producer of a text. When someone generates language, the average surprisal of their words reflects how predictable—or unconventional—their discourse tends to be. That raises an obvious psychometric question: is mean surprisal a stable property of the person, or merely a quirk of one particular answer that will not generalize to anything else that person writes? If the former, surprisal joins the ranks of individual differences like vocabulary size or processing speed; if the latter, it is little more than a text-level statistic.

To find out, the researchers conducted a series of four studies using datasets from previously published work, reanalyzing language-based responses produced by the same individuals across multiple different items and tasks. Crucially, they treated the problem with the standard toolkit of psychological measurement: reliability analysis, inter-item correlations, and corrections for measurement error. If surprisal were merely a property of individual texts, scores from different writing prompts completed by the same person should be essentially unrelated. If it were a trait, those scores should correlate, just as different items on a well-constructed questionnaire do.

The results were striking. When surprisal scores were averaged across the different language-based items within a task, reliability was excellent—the measure behaved like a coherent psychological scale. Within individual items, reliability varied, but across items the scores converged. In other words, a person who produced highly predictable language on one prompt tended to produce highly predictable language on another, even when the content differed completely. The team also found, however, that surprisal was significantly influenced by the length of the text answer, a methodological caveat that future studies will need to control for when comparing texts of different sizes.

The researchers then examined how mean surprisal scores related to other variables, drawing on datasets that included measures of language skills, domain-general cognitive abilities, and personality traits. The pattern of correlations suggested that surprisal is not an isolated curiosity. It related to certain cognitive functions, consistent with the idea that producing predictable versus unpredictable language taps into underlying processing efficiency. The authors also propose that surprisal may capture alternative styles of speech or communication—broad tendencies in how people assemble their discourse that go beyond what any single answer reveals.

Technically, the measure is closely related to perplexity, a familiar quantity in computational linguistics; the authors note that surprisal scores can be seen as a linear transformation of perplexity but argue that surprisal is more interpretable from a psychological standpoint. The models used were left-to-right next-word predictors, meaning that each word’s surprisal was computed from the words preceding it, mirroring the incremental nature of human language production. Averaging word-level surprisal across a response yields a single predictability score per text, and averaging across texts yields the individual-difference measure at the heart of the study.

The implications reach well beyond psycholinguistics. Automated scoring of essays, the assessment of constructed responses, and the growing field of speech-based biomarkers for neurological conditions all depend on extracting meaningful, stable signals from language samples. If average surprisal tracks a durable characteristic of the speaker or writer, it could complement existing lexical and semantic features in applications ranging from educational assessment to the detection of cognitive decline, where changes in the structure and predictability of spontaneous speech have been linked to conditions such as Alzheimer’s disease and frontotemporal dementia. The new findings provide the psychometric foundation such applications require: evidence that the signal is reliable, not noise.

The study also carries a cautionary message for the rapidly expanding literature that uses large language models to score human language. Any single text-based score is contaminated by measurement error and by item-specific effects, and correlations between such scores and other variables will be attenuated accordingly. The authors show, for example, that correcting for attenuation can substantially raise observed inter-item correlations, meaning that raw correlations should be read as lower bounds. Transparency was a priority for the team: all surprisal measures, code, and supplementary materials are publicly available through the Open Science Framework, allowing other researchers to scrutinize and extend the approach. As artificial intelligence increasingly mediates how human language is measured, this work demonstrates that concepts born in information theory can be tamed into rigorous psychological instruments—provided researchers ask the same questions of them that psychometricians have asked of tests for a century: Is it reliable? Does it generalize? And what, exactly, does it measure about the person behind the words?

Subject of Research: Using large language model surprisal as a stable individual-difference measure of language production predictability

Article Title: Surprisal from Large Language Models as an individual difference: Capturing the individual regardless of the text answer

Article References: Martínez-Huertas, J. Á., Martínez-Mingo, A., López Rosell, A., Kreitchmann, R. S., & Jorge-Botana, G. (2026). Surprisal from Large Language Models as an individual difference: Capturing the individual regardless of the text answer. Behavior Research Methods, 58(10), Article 288. https://doi.org/10.3758/s13428-026-03117-7

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03117-7

Keywords: surprisal, large language models, individual differences, psycholinguistics, language production, reliability, predictability, Behavior Research Methods, psychometrics, information theory, cognitive skills, automated text scoring