How Hard Is That Text? Eye Movements Reveal the Gap Between Measured and Felt Reading Difficulty

How difficult is a given English text? For decades, the answer to that question has come from formulas: counts of long words, averages of sentence length, and other measurable features of language that can be crunched into a single readability score. A new study from researchers in Japan suggests that while these objective measures do a remarkably good job of predicting how hard readers will find a passage, they tell only part of the story. The way an individual reader experiences a text—and the way their eyes move across the page while processing it—contains information that no text-level formula can fully capture.

The research, conducted by Assistant Professor Shingo Nahatame of the Institute of Human Sciences at the University of Tsukuba and Associate Professor Tomoko Ogiso of the Faculty of Education at the University of Toyama, was published in the journal Reading & Writing. The study set out to address a central concern in English language education: how to evaluate the difficulty of the texts used in educational materials and language assessments. Traditionally, researchers have relied on objective measures of text difficulty grounded in linguistic features, including characteristics of words and sentences. Yet learners can differ substantially in how difficult they perceive the very same text to be, and understanding that gap matters for anyone who selects, writes, or assesses reading material.

To examine the relationship between objective and subjective text difficulty, the researchers designed a study with two complementary strands. First, they evaluated the objective difficulty of 30 English reading passages spanning a broad range of difficulty levels. Objective difficulty was assessed in two ways: using established readability measures based on linguistic characteristics, and using an evaluation method based on a large language model, reflecting the growing role of artificial intelligence tools in text analysis. This dual approach allowed the team to compare how well both traditional formula-based indices and modern computational methods captured the demands each passage placed on readers.

The second strand of the study turned to the readers themselves. Forty-one undergraduate and graduate students whose first language was Japanese read the same 30 passages and rated the difficulty of each one on a nine-point scale. These ratings provided a direct window into subjective difficulty—the felt experience of struggling with, or sailing through, a text. Because all participants read all passages, the design made it possible to compare, passage by passage and reader by reader, how objective estimates lined up with personal perceptions.

The analyses revealed a strong relationship between objective and subjective difficulty. Passages rated as more difficult by objective measures were also perceived as more difficult by readers. In other words, the traditional intuition behind readability formulas holds up: texts with longer sentences, rarer words, and more complex linguistic characteristics genuinely do feel harder to read, at least on average. The large language model–based evaluation method, likewise, produced difficulty estimates that corresponded with readers’ perceptions, suggesting that computational approaches can serve as viable complements to conventional readability indices.

But the study did not stop at ratings. The researchers also recorded participants’ eye movements as they read, adding a process-level measure of reading behavior to the product-level measures of text features and subjective judgments. Eye-tracking has long been used in reading research because the eyes provide a moment-by-moment record of cognitive effort: fixations lengthen when processing is demanding, and readers skip words that are easy to identify. In this study, objective and subjective difficulty measures were both associated with eye-movement patterns during reading. More difficult texts tended to be read more slowly, and readers skipped fewer words when reading them. The convergence is striking: three independent sources of evidence—linguistic formulas, learner judgments, and real-time gaze behavior—all pointed in the same direction.

The most novel finding, however, concerned the discrepancy between the two kinds of difficulty. The researchers found that the extent to which an individual reader’s rating differed from the group’s average rating for the same passage provided additional information for explaining eye-movement behavior beyond what could be accounted for by text-level difficulty measures alone. In practical terms, if a particular student found a passage harder than their peers did, that deviation was reflected in the way their eyes moved through the text, even after the general difficulty of the passage had been taken into account. Subjective difficulty, in other words, is not simply a noisy version of objective difficulty—it carries unique, reader-specific signal.

These findings indicate that objective measures of text difficulty and subjective perceptions of learners generally correspond closely but are not identical. Objective measures are useful for characterizing the general difficulty of a text, and they remain an efficient tool for grading passages, leveling curricula, and calibrating assessments across large populations. But subjective evaluations provide additional insight into how individual learners experience and process the same text. A readability score describes the passage; a learner’s rating describes the encounter between the passage and a particular mind. The study suggests that both perspectives are needed to build a complete picture of reading difficulty.

The implications reach into several corners of language education. For teachers and materials developers, the results support a two-step approach: objective measures can be used to shortlist texts at an appropriate general level, while learner feedback can then be used to fine-tune selections for specific students or classes. For researchers, the demonstration that rating deviations predict eye-movement behavior opens the door to more personalized models of second-language reading, in which individual differences in perception are treated as meaningful data rather than measurement error. The work also speaks to the growing interest in using large language models for text evaluation, showing that such tools can align with human perceptions while still leaving room for individual variability.

More broadly, the study is a reminder that difficulty is not a property of a text alone, nor of a reader alone, but of the interaction between them. The 30 passages, 41 readers, nine-point ratings, and recorded eye movements together paint a technically grounded picture of that interaction: strong average correspondence between formulas and feelings, reliable links between both and the microstructure of gaze, and a persistent, informative residue of individual difference. As English language education moves toward more personalized approaches to reading support, the message from Tsukuba and Toyama is clear—measure the text, but also listen to the reader, because the two are related, yet not the same.

Subject of Research: The relationship between objective and subjective text difficulty in second-language English reading and its link to eye-movement behavior

Article Title: Subjective and objective difficulty of English texts are related—but not the same

Article References: Subjective and objective difficulty of English texts are related—but not the same. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: text difficulty, readability, second-language reading, eye-tracking, English language education, large language models, reading assessment, subjective perception, University of Tsukuba, Reading & Writing, L2 learners, reading materials