ChatGPT-4o Nearly Matches Human Examiners in Grading Finland’s Toughest Exam

An artificial intelligence system has come strikingly close to matching the judgment of human examiners in one of the most demanding settings imaginable: a national, high-stakes examination that helps determine students’ futures. A new study from the University of Turku in Finland reports that ChatGPT-4o, when equipped with a carefully engineered grading framework, produced scores on Finland’s matriculation examination that largely tracked the official grades awarded by trained human evaluators. The findings, published in the journal Frontiers of Digital Education, offer one of the most rigorous large-scale benchmarks to date of large language models as graders of open-ended written answers, and they arrive at a moment when education systems worldwide are wrestling with whether generative AI can be trusted with consequential decisions.

The research team, led by Jussi S. Jauhiainen together with Agustín Garagorry Guerra, did not test the AI on toy examples or classroom quizzes. Instead, they turned to the Finnish matriculation examination, the national test that concludes upper secondary education and is enshrined in Finnish law. The dataset covered the open-ended written responses of 1,016 students, graded on the examination’s official 0-to-15 scale. This scale is narrow enough that even a two-point deviation can matter, and it demands nuanced judgment about argumentation, subject knowledge, and contextual use of concepts—precisely the qualities that have historically made automated essay scoring so difficult.

Technically, the researchers did not simply paste student answers into a chatbot and hope for the best. They deployed ChatGPT-4o within a reranked retrieval-augmented generation, or RAG, framework. In a RAG pipeline, the model first retrieves relevant reference material—in this case, grading criteria and model answers associated with each examination question—then reranks the retrieved content so the most pertinent information sits closest to the prompt, and finally generates its assessment grounded in that curated context. This architecture matters because it anchors the model’s judgment in the official grading instructions rather than relying purely on the statistical patterns encoded in its training data. The approach builds on the authors’ earlier work applying RAG-based grading to models such as GPT-3.5, GPT-4, Claude-3, and Mistral-Large, but the new study scales the method up and subjects it to far stricter comparison against human ground truth.

The headline result is arresting. Measured with descriptive statistics and correlation analyses, GPT-4o’s scores landed within two points of the official human grades in 75.00 percent of cases. Severe outliers—grades that diverged so far as to raise questions about fundamental misunderstanding—accounted for only 3.00 percent of responses. For a machine assigning integer scores on a 16-point scale to free-form, argumentative prose in a national exam, that level of agreement represents a genuine milestone. Earlier automated scoring systems, from latent semantic analysis in the late 1990s to modern BERT-based essay scorers, typically required extensive task-specific training on thousands of pre-graded examples. Here, a general-purpose language model, guided by retrieved grading criteria, approached human-level agreement largely out of the box.

Yet the study’s second experiment may prove the most consequential for the billions of people who do not speak English natively. Finnish is a low-resource language in the world of large language models: it appears far less frequently in training corpora than English, and its rich morphology and flexible word order pose additional challenges for statistical language processing. When the researchers translated the students’ Finnish responses into English and had the model grade the translations, alignment with human examiners jumped from 75.00 percent to 85.00 percent. The implication is sobering and empowering at once. The same AI system performs meaningfully better in a high-resource language than in a low-resource one, but a relatively simple translation step can recover much of the lost accuracy. For education systems in smaller language communities, that could determine whether AI-assisted grading is a dangerous shortcut or a viable supplement.

The study also probed a subtler question: not just whether the machine’s final scores matched human scores, but whether the machine was grading for the right reasons. The researchers examined whether GPT-4o recognized grading-relevant keywords—the technical terms and concepts that official criteria require examiners to spot. Broadly, it did. The model identified relevant keywords in the great majority of responses, which helps explain its strong score alignment. But the failures were revealing. In a minority of cases, the model misinterpreted the contextual usage of a term, registering a keyword as present when a student had used it incorrectly or in a way that did not satisfy the criterion, or occasionally overlooking legitimate conceptual content expressed in unusual phrasing. Those misreadings translated directly into grading errors, underscoring that surface-level keyword matching is not the same as genuine comprehension.

This distinction between matching scores and matching understanding sits at the heart of the paper’s conclusions. The authors are explicit that the findings demonstrate both the promise and the current limitations of LLM-based assessment. A model that agrees with human graders three-quarters or more of the time is a powerful tool, but the residual disagreements are not random noise; they cluster exactly where human interpretive depth matters most—irony, implicit reasoning, unconventional but valid arguments, and nuanced use of domain concepts. High-stakes examinations leave no room for such errors to be dismissed as acceptable variance when a student’s university admission or graduation certificate hangs in the balance.

The practical prescription that emerges is not replacement but supervision. The researchers argue that large language models hold significant potential as supplementary grading tools, particularly in high-resource languages, while cautioning that they do not yet match the consistency or interpretative depth of trained human evaluators. Deploying them responsibly, the study suggests, requires human oversight of every consequential grade, rigorous validation against official outcomes before any deployment, and careful attention to language effects—especially the systematic penalty that low-resource languages appear to suffer. The authors also frame the technology within the broader debate on trust and transparency in AI: teachers’ willingness to accept AI-powered tools has been shown to depend on demonstrated reliability, and the so-called black-box problem remains a psychological and institutional barrier even when statistical performance is strong.

The context of the study amplifies its significance. Finland’s matriculation examination is a legally regulated national institution, with data access restricted by the Finnish National Agency for Education, which makes any benchmark against its official grades unusually authoritative. Open-ended questions on such exams have long been the last bastion of exclusively human assessment; multiple-choice testing was automated decades ago, but essays, structured arguments, and analytical responses resisted automation because grading them is an act of interpretation. If a general-purpose model with a well-designed retrieval framework can now approach human agreement on that terrain—and if translation can substantially close the gap for smaller languages—then the question facing ministries of education is no longer whether AI can grade, but under what safeguards it should be allowed to try.

What comes next, the researchers suggest, is continued benchmarking across subjects, question types, and languages, alongside the development of hybrid workflows in which the machine proposes and the human disposes. The vision is not an examination hall emptied of teachers, but one in which AI handles the first pass at enormous scale, flags uncertain cases, and frees human experts to focus their judgment where it counts. The Finnish results show that such workflows are technically plausible today, with measured agreement rates that would have seemed fanciful only a few years ago. They also show, with equal clarity, that the last few percentage points of agreement—the difference between a machine that assists grading and a machine that can be trusted to grade alone—remain stubbornly human territory. For now, the examiners of Finland, and everywhere else, can breathe a measured sigh of relief: the machine is close, but not close enough, and the study that proved it may shape how the world’s schools adopt AI for years to come.

Beyond the headline agreement rates, the study’s methodology reflects a broader shift in how automated assessment is evaluated. Rather than relying on a single aggregate statistic, the researchers combined descriptive statistics with correlation analyses and keyword-recognition checks, giving a multidimensional picture of where the model succeeded and failed. This kind of layered evaluation is increasingly seen as essential, because two grading systems can produce identical score distributions while reasoning about student answers in fundamentally different ways.

The translation experiment also carries implications for how multilingual capability is measured in language models. Performance gaps between high- and low-resource languages are well documented in the machine learning literature, but this study quantifies the gap in a concrete, consequential setting: a legally regulated national examination. The finding that translating responses into English improved alignment by ten percentage points suggests that language itself, not the difficulty of the exam content, was a major source of grading error.

For institutions considering similar deployments, the Finnish data restrictions highlight a practical constraint. Because the examination data could not be shared publicly, independent replication remains limited, reinforcing the authors’ call for rigorous local validation before any AI-assisted grading system is adopted in high-stakes contexts.

Subject of Research: Benchmarking ChatGPT-4o against human expert grading of open-ended responses in Finland's national high-stakes matriculation examination across high- and low-resource languages

Article Title: Evaluating Open-Ended High-Stakes Examinations with LLMs: Alignment Between ChatGPT-4o and Human Grading in High- and Low-Resource Languages

Article References: Jauhiainen, J. S., & Guerra, A. G. (2026). Evaluating Open-Ended High-Stakes Examinations with LLMs: Alignment Between ChatGPT-4o and Human Grading in High- and Low-Resource Languages. Frontiers of Digital Education, 3(2), Article 17. https://doi.org/10.1007/s44366-026-0091-1

Image Credits: AI Generated

DOI: 10.1007/s44366-026-0091-1

Keywords: ChatGPT-4o, large language models, automated grading, high-stakes examination, Finnish matriculation examination, retrieval-augmented generation, open-ended assessment, low-resource languages, educational assessment, human-AI agreement, AI in education, automated essay scoring