The benchmark, developed by Tianqiao Liu, Zui Chen, Zhensheng Fang, Weiqi Luo, Mi Tian, and Zitao Liu of the Guangdong Institute of Smart Education at Jinan University and TAL Education Group, consolidates 22 distinct datasets into one evaluation framework. The collection spans an enormous range of mathematical territory: elementary arithmetic and word problems, competition-level mathematics, and higher mathematics at university and beyond. It covers problems written in both English and Chinese, and it deliberately includes material at every difficulty level from primary school exercises to problems that challenge even strong human mathematicians. By drawing on well-known resources such as GSM8K, MATH, MathQA, MAWPS, Ape210K, OlympiadBench, and AGIEval, the benchmark situates itself within the existing landscape of evaluation research while unifying those scattered efforts under one roof.

The motivation for such a unification is straightforward. Previous assessments of language models on mathematics have been, in the words of the research team, inconsistent and incomplete. One study might report that a model solves 80 percent of grade-school word problems, while another finds the same model failing on similar material because the answer format differed, the prompt was phrased differently, or the grading script treated equivalent expressions as mismatched. Because mathematical outputs can take many valid forms — a fraction versus a decimal, a simplified radical versus an approximation, a solution set written in different notation — automatically comparing a model’s answer to a reference answer is far harder than it might appear. Small differences in extraction and comparison logic can shift reported accuracy by several percentage points, enough to change the ranking of competing models.

MathEval tackles this grading problem with an unusual and technically interesting solution: it uses GPT-4 itself as an automated pipeline for answer extraction and comparison. Rather than relying on brittle regular expressions or rigid string matching, the benchmark prompts GPT-4 to read a model’s full response, identify the final answer, and judge whether it is mathematically equivalent to the reference solution. This approach adapts to diverse models, diverse output styles, and diverse prompt formats, which is essential when evaluating dozens of systems that each express their conclusions differently. The team validated the reliability of this automated judging process, drawing on established statistical measures of agreement to confirm that the pipeline’s judgments are consistent and trustworthy.

There is, however, an obvious practical drawback to using GPT-4 as a grader: not every research group has reliable access to it, and running every comparison through a commercial frontier model is expensive. To solve this, the researchers trained a publicly available model, built on the DeepSeek-LLM-7B-Base architecture, using GPT-4’s comparison results as training data. The result is a compact answer-validation model that reproduces GPT-4’s grading behavior without requiring any access to GPT-4 itself. This distillation strategy means that any laboratory, anywhere, can run the full MathEval evaluation pipeline on local hardware, democratizing access to a high-quality assessment tool and lowering the barrier to rigorous, reproducible measurement of mathematical reasoning.

Perhaps the most forward-looking feature of MathEval is its defense against data contamination. A persistent worry in language model evaluation is that test problems leak into training data: web-scraped corpora inevitably contain popular benchmark datasets, so a model may appear to solve problems it has effectively memorized rather than reasoned through. This concern inflates scores and makes genuine progress impossible to measure. MathEval addresses it by incorporating an annually refreshed set of problems drawn from the most recent Chinese National College Entrance Examination, the Gaokao, including the 2023 and 2024 editions. Because these exam questions are newly written each year and administered under strict security, they cannot have contaminated any model’s training data at evaluation time. Performance on the Gaokao subset therefore offers a cleaner signal of true problem-solving ability, and the benchmark is designed to keep pace with each year’s exam, providing a moving target that models cannot simply memorize their way past.

The choice of the Gaokao is also scientifically apt. The examination is famous for its demanding mathematics section, which requires not only computation but also careful reading, multi-step planning, and the synthesis of ideas from algebra, geometry, trigonometry, and calculus. It is bilingual in practice, since the benchmark as a whole tests both English and Chinese, and cross-lingual evaluation matters: a model that excels on English problems may falter on equivalent Chinese ones, and vice versa. By embedding these fresh exam problems within a broader multilingual, multi-domain suite, MathEval can reveal whether a model’s mathematical competence is a genuine, transferable skill or a narrow, language-specific and dataset-specific trick.

The benchmark’s design also reflects a broader trend in artificial intelligence research toward holistic evaluation. Earlier efforts such as BIG-Bench, HELM, LongBench, and CMMLU demonstrated that single-number scores on isolated tasks tell an incomplete story about what language models can and cannot do. MathEval extends that philosophy to mathematics specifically, evaluating models across multiple dimensions simultaneously: mathematical discipline, language, problem category, and difficulty level. This multidimensional structure allows researchers to localize failures with precision. A model might handle arithmetic flawlessly yet collapse on olympiad geometry, or perform well in English while degrading in Chinese, or succeed on familiar textbook formats while stumbling on novel competition problems. Each of these failure patterns suggests a different underlying weakness and points toward different remedies in training data or model design.

The stakes of getting this measurement right extend well beyond leaderboard rivalry. Mathematical reasoning underpins applications that society increasingly wants to hand to AI systems: tutoring students, verifying scientific calculations, assisting engineers, and generating reliable quantitative analyses. The research team’s own related work on math reasoning in language models, on step-level reward models, and on chain-of-thought decoding shows how much active effort is being invested in improving these capabilities. But improvement cannot be demonstrated without measurement that is fair, consistent, and resistant to gaming. A benchmark that changes its test set annually, grades answers with validated automated judges, and spans the full spectrum of mathematical difficulty provides exactly the kind of stable yardstick that the field has lacked.

MathEval is already gaining traction in the research community, with the published paper accumulating citations and thousands of accesses since its release in May 2025. Its open availability, combined with the distilled answer-checking model that removes the GPT-4 dependency, makes it a practical standard that other groups can adopt immediately. As each new generation of language models arrives claiming better reasoning, the benchmark’s refreshed Gaokao problems will be waiting — unseen, unmemorized, and unforgiving. In a field where the gap between claimed and actual capability can be obscured by contaminated test sets and inconsistent grading, that kind of honest examination may prove to be the most valuable result of all.

Subject of Research: Benchmarking the mathematical reasoning capabilities of large language models

Article Title: MathEval: A Comprehensive Benchmark for Evaluating Large Language Models on Mathematical Reasoning Capabilities

Article References: Liu, T., Chen, Z., Fang, Z., Luo, W., Tian, M., & Liu, Z. (2025). MathEval: A Comprehensive Benchmark for Evaluating Large Language Models on Mathematical Reasoning Capabilities. Frontiers of Digital Education, 2(2), Article 16. https://doi.org/10.1007/s44366-025-0053-z

Image Credits: AI Generated

DOI: 10.1007/s44366-025-0053-z

Keywords: MathEval, large language models, mathematical reasoning, benchmark, GPT-4, DeepSeek, Gaokao, answer grading, data contamination, evaluation metrics, artificial intelligence, education

The work, led by Haotian Zhang of China Mobile Group Henan and Beijing Normal University, together with Qi Jia of Yuntu Zhixing and colleagues at IEIT Systems, Southeast University, and Beijing Normal University, addresses three long-standing weaknesses in existing geometry problem solvers. According to the authors, current methods struggle to capture the structural information embedded in diagrams, to align textual symbols with the visual primitives they refer to, and to cope with the sparse distribution of geometric primitives in typical figures. Each of these gaps can cause a model to miss the very relationships a problem depends on.

The core idea of the new system is a graph construction method that converts every diagram into a heterogeneous graph. In such a graph, nodes represent different kinds of geometric entities, for example points, line segments, circles, and angles, while edges encode the diverse relations that connect them, such as incidence, parallelism, or containment. Because the node and edge types differ, the representation is called heterogeneous, and it preserves far more of the diagram’s logical structure than a flat grid of pixels. The model can then reason over explicit geometric relationships rather than having to infer them implicitly from raw visual features.

Building on this construction method, the team created the Geometric Relation Heterogeneous Graph dataset, abbreviated Geo-RHG. The dataset pairs geometric diagrams with their corresponding textual descriptions and, crucially, with heterogeneous graph representations of each diagram. This triple structure gives researchers a resource for training and evaluating models that must understand both what a diagram shows and what the accompanying words say. The authors state that the dataset and code will be released at the project’s GitHub repository upon publication, making the resource available to the wider research community.

A second pillar of the approach is a cross-modal pre-training strategy designed to bridge the gap between language and vision. Text in a geometry problem refers to visual objects: a phrase such as ‘AB is parallel to CD’ names specific segments in the figure. The pre-training procedure explicitly aligns textual symbols with their corresponding visual primitives, teaching the model which words point to which parts of the graph. This grounding step is intended to eliminate a common failure mode in which a model understands the sentence and sees the diagram but never connects the two.

The third component is a hierarchical fusion mechanism that injects structural cues into visual features during the modality fusion stage. Rather than simply concatenating text embeddings and image embeddings, the system lets the graph-derived structural information refine the visual representation before and during fusion. The authors report that this integration enhances the model’s capacity to reason over geometric content and supports more accurate problem solving in multimodal contexts, because the fused features carry both appearance and relational information.

Extensive experiments described in the paper show that the proposed approach achieves superior performance on standard geometry problem solving benchmarks, outperforming existing baselines in both accuracy and generalization. Generalization matters particularly in this field: a solver that merely memorizes the problem templates of a training set is of little use in classrooms or examination settings, where problems vary widely in wording and figure style. The combination of explicit graph structure and cross-modal alignment appears to give the model a more transferable understanding of geometric content.

The new study sits within a research lineage that stretches back decades. Early efforts at automated geometry theorem proving date to the 1960s, when researchers such as Gelernter and colleagues built the first geometry theorem machines. Modern data-driven approaches began with systems like GeoSolver, which combined text and diagram interpretation, and later work harvested axiomatic knowledge from textbooks. More recently, benchmarks such as GeoQA, UniGeo, and augmented geometric question answering datasets have fueled progress, while frameworks like Inter-GPS introduced formal languages and symbolic reasoning, and GeoDRL applied reinforcement learning to deductive reasoning. Explainable solvers and layout-aware neural models have pushed the field further, yet the authors argue that none of these fully resolves the structural modeling and alignment problems their graph-based approach targets.

Technically, the system draws on a broad set of tools from modern deep learning. Vision encoders of the kind introduced in residual networks and vision transformers provide the raw visual features, while object detection architectures in the Faster R-CNN tradition help localize geometric primitives. Language understanding builds on pretrained models in the BERT and RoBERTa family. Graph convolutional networks, which were developed for modeling relational data, supply the machinery for propagating information across the heterogeneous graph, and the transformer attention mechanism underlies the cross-modal fusion. The authors also cite visual question answering research, including graph-structured representations for VQA and conditioned graph structures, as conceptual ancestors of their approach, alongside classical sequence models such as long short-term memory networks that once dominated multimodal tasks.

Diagram parsing itself has advanced rapidly in recent years, with work on plane geometry diagram parsing and alignment-based relation learning for diagram parsing showing that the geometric content of figures can be extracted with increasing fidelity. The new system builds on this progress but goes a step further by making the relational structure the central object of representation rather than a byproduct of parsing. The authors suggest that this shift is what allows their model to handle the sparse, symbol-heavy nature of geometric figures, where a handful of lines and labels carry all the meaning and background pixels carry almost none.

The implications extend beyond geometry homework. Multimodal reasoning systems that must combine images and language appear in visual question answering, document understanding, scientific figure analysis, and educational technology. A method that explicitly structures visual content as a graph and aligns it with text could transfer to any domain where the image is not a photograph but a structured drawing: circuit diagrams, flowcharts, chemical structures, or architectural plans. The hierarchical fusion idea, in which structural cues refine visual features before fusion with language, may prove useful wherever relational structure is the key to understanding an image.

For the field of automatic problem solving, the study offers a concrete recipe: represent the diagram as a heterogeneous graph, pre-train the model to align symbols with primitives, and fuse modalities hierarchically with structure in mind. The strong benchmark results reported by the team suggest that explicit structure, long championed in symbolic AI, and large-scale neural learning are not rivals but complements. As the Geo-RHG dataset and code become publicly available, other researchers will be able to test that proposition, refine the graph construction methods, and push multimodal mathematical reasoning closer to the fluency of a well-trained human geometer.

The article, ‘A geometry problem solver based on heterogeneous graph,’ appears in Applied Intelligence, volume 56, article number 439, published on 22 September 2026. The work was received on 12 November 2025 and accepted on 7 September 2026, and its authors span industry and academia in China, reflecting the growing collaboration between technology companies and universities on foundational problems in artificial intelligence. With benchmarks falling and resources opening up, geometry problem solving is shaping up as one of the clearest windows into how machines can learn to see, read, and reason as one.

Subject of Research: A multimodal AI system that solves geometry problems by representing diagrams as heterogeneous graphs and aligning them with text.

Article Title: A geometry problem solver based on heterogeneous graph

Article References: Zhang, H., Jia, Q., Liu, L., Xu, C., Jin, L., Pan, Y., Chen, F., Wang, Y., & Liu, Y. (2026). A geometry problem solver based on heterogeneous graph. Applied Intelligence, 56(15), Article 439. https://doi.org/10.1007/s10489-026-07471-z

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07471-z

Keywords: geometry problem solving, heterogeneous graph, multimodal learning, cross-modal pre-training, Geo-RHG dataset, graph neural network, visual reasoning, mathematical reasoning, AI benchmark, feature representation, automatic problem solving, Applied Intelligence