Your Language History Shapes What You Expect Sentences to Say Next

Every time you read a sentence like “The boy went outside to play with a…”, your brain quietly places a bet on what comes next. For decades, psycholinguists have measured those bets with a deceptively simple tool called the cloze task, in which people complete sentence fragments with the first word that springs to mind. The assumption has always been that the resulting scores, known as sentence constraint norms, reflect the sentences themselves. A new study published in Behavior Research Methods by Ana Bautista and Clara D. Martin of the Basque Center on Cognition, Brain and Language shows that this assumption is far shakier than the field has believed. Who you are, which languages you speak, and which of those languages dominates your daily life all leave measurable fingerprints on the words you predict, with consequences that ripple through everything from laboratory experiment design to the evaluation of artificial intelligence language models.

The cloze task, introduced by Wilson Taylor in 1953, has become the workhorse of language prediction research. Researchers present participants with incomplete sentences and record the distribution of their completions. If most people supply the same word, the sentence is considered highly constraining; if responses scatter widely, it is not. Traditionally, this is quantified as cloze probability, the percentage of participants who produce the most frequent continuation. Some studies treat sentences below 37 percent as low-constraining and those above 67 percent as highly constraining. But cloze probability has a blind spot: a distribution of 60-10-10-10-10 and one of 60-40 yield the same score despite radically different uncertainty. To capture that hidden variability, the researchers also computed response entropy, a measure borrowed from information theory that sums the probabilities of every unique answer, weighted logarithmically. Entropy bottoms out at zero when everyone agrees and climbs toward its maximum when every response is unique. Previous work has suggested entropy may even predict reading behavior better than cloze probability alone.

The critical question Bautista and Martin posed was deceptively simple: does the population used to establish these norms actually matter? The validity of any cloze pre-test depends on how closely the norming sample matches the people who will later be tested in the main experiment. Studies of aging had already shown that cloze responses are stable across adulthood, which allowed researchers to norm materials with young students and use them with older adults. But for language, the intuition that proficiency shapes cloze answers had never been empirically tested. This gap has grown urgent as bilingualism research moves away from comparing native speakers against everyone else and toward more sophisticated designs, such as mirrored language groups or multiple language pairings, that control for confounds the old paradigms ignored.

To fill that gap, the team recruited 240 participants through the online platform Prolific, forming four groups of sixty. Two groups were dominant speakers: people born in France who reported French as their first language, and people born in Spain who reported Spanish as their first language. Two further groups were fluent but non-dominant speakers of French: Spanish-French bilinguals born in Spain and English-French bilinguals born in the United Kingdom. Proficiency was verified with LexTALE vocabulary tests, which confirmed that the two bilingual groups did not differ significantly in their French proficiency, a crucial control. Each participant completed a cloze task on 110 sentence contexts drawn from a shared pool of 330 sentences, each designed to bias toward a final noun phrase such as “a cow” after “On farms, to get milk, it is necessary to milk…”. Every sentence had a French and a Spanish translation equivalent, with matched target nouns in length and lexical frequency.

The first analysis asked whether language itself shapes sentence constraint. French-dominant and Spanish-dominant speakers completed the same sentences in their respective languages, and their average response entropy turned out to be strikingly similar: 1.05 in French versus 0.93 in Spanish, a difference that fell short of statistical significance. On the surface, translation seemed to preserve constraint. But when the researchers examined the item-by-item alignment between the two groups using Spearman correlations, benchmarked against bootstrapped estimates of chance-level and maximum possible alignment, the picture changed. The observed correlation of 0.55 was far above chance yet clearly below the internal consistency of each group, which sat at 0.67. In other words, individual sentences did not reliably elicit the same degree of constraint across the two languages, even though the languages are closely related Romance cousins sharing substantial vocabulary and culture.

One striking example reveals why. The sentence “The elderly woman bought potatoes and eggs because she wanted to prepare…” drove 90 percent of Spanish-dominant speakers to answer “tortilla”, the iconic Spanish omelette, but only 40 percent of French-dominant speakers gave the equivalent response. The constraint difference here reflects culturally specific knowledge rather than anything linguistic. The authors argue that such cultural iconicity can shape cloze answers beyond language per se, and they speculate that even French speakers from France, Quebec, and Senegal might diverge on identical sentences. The methodological implication is blunt: translating cloze materials from one language to another introduces response variability even under strict translation, so the most reliable practice remains pre-testing sentences with the target language community itself.

The second analysis tackled language dominance, comparing the three groups who completed sentences in French. Here the average differences were unmistakable. Response entropy averaged 1.05 for French-dominant speakers but jumped to 1.50 for Spanish-French bilinguals and 1.47 for English-French bilinguals, a highly significant difference. Sentences were simply less constraining for people reading in their non-dominant language, confirming empirically what researchers had long assumed but never demonstrated. Alignment between dominant and non-dominant groups was again only moderate, with correlations of 0.52 and 0.60 falling well short of the maximum-alignment benchmarks. Practically, this means that using cloze norms collected from dominant speakers to design experiments with non-dominant populations will inevitably introduce error in estimates of lexical constraint.

The most surprising result emerged when the two non-dominant groups were compared with each other. Spanish-French and English-French bilinguals, despite having entirely different dominant languages, showed nearly identical average entropy and an item-by-item correlation of 0.70, the highest in the study and the closest to what would be expected if the groups came from a single population. This suggests that what matters most for cloze responses is the speaker’s relationship to the language being tested, not the identity of their other language. Norms gathered from one non-dominant group may transfer to another non-dominant group of similar proficiency, potentially saving researchers an entire norming study. The authors caution, however, that both bilingual groups shared European cultural backgrounds, and the alignment might not extend to more distant pairings such as Mandarin Chinese-French bilinguals.

Both measures of constraint, the traditional cloze probability and the newer response entropy, led to the same conclusions and correlated above 0.96 within every group, suggesting neither needs to be abandoned. Yet the study also exposed limitations in the entropy formula, which can produce counterintuitive values, rating a 50-50 split as less uncertain than a 70-10-10-10 distribution. The authors propose that a normalized version might better capture predictive uncertainty. They also note that even within a single population, split-half consistency never exceeded 0.70, meaning cloze norms are inherently probabilistic estimates rather than fixed standards, and exact scores from a pre-test will never perfectly match those of the main experiment.

The findings reach beyond the laboratory into the age of large language models. Surprisal estimates from these models increasingly predict well-established neural markers of language prediction, such as N400 brain responses and eye movements during reading, but most of that evidence comes from English and other Germanic languages that dominate training data, and the models carry documented biases toward Western cultural norms. If human sentence expectations are shaped by language and dominance, models are unlikely to capture the learning trajectories and sociolinguistic profiles of non-English-dominant speakers. The authors recommend that researchers studying bilingual populations consider bilingual language models trained on comparable language pairs, and those studying monolingual non-English populations use monolingual models for that language. Systematic comparisons between model-derived surprisal and human cloze norms across diverse groups are now needed. The team has released its full database of cloze responses from all four populations to 330 translated sentences through the Open Science Framework, giving the field both the evidence and the resources to stop treating prediction as language-independent.

Subject of Research: How language and language dominance influence lexical sentence constraint measured by cloze tests

Article Title: Not that cloze: Sentence constraint is influenced by language and language dominance

Article References: Not that cloze: Sentence constraint is influenced by language and language dominance. (n.d.). https://doi.org/10.3758/s13428-026-03140-8

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03140-8

Keywords: cloze test, sentence constraint, language prediction, bilingualism, language dominance, response entropy, psycholinguistics, cross-linguistic differences, large language models, surprisal, Behavior Research Methods, norming