AI consensus covered 65.6% of undergraduate rubric decisions in retrospective study

Three multimodal language models grading the same handwritten mathematics and computing submissions directly from their images agreed with instructors on 91.9% of the rubric decisions they jointly accepted, according to a retrospective study of authentic undergraduate assessments. The analysis comprised 3,346 rubric-level judgments covering 1,875 submission-question responses in three courses at Chiang Mai University. Under the selective-review rule, GPT-4.1, Claude Sonnet 4.6 and Gemini 2.5 Pro produced identical, non-null binary decisions on 65.6% of cases, while disagreements or incomplete outputs were routed to an instructor. The selected subset was more reliable than the full-coverage three-model majority rule, which matched instructors on 83.7% of decisions, but the study showed that unanimous agreement is not proof of a correct grade.

The research did not use a separate optical-character-recognition system. Each model received an anonymized image containing a student response together with an instructor-approved rubric, then returned every rubric decision in a single structured JSON response. The instruction required a binary award or withholding of credit for each criterion, or null when the model could not determine the answer, with no free-form explanation. This direct image-to-decision design removes an explicit transcript stage and allows a model to use handwriting, mathematical expressions, diagrams or code alongside the grading rules. It also makes errors harder to diagnose: an incorrect result may arise from misreading a symbol, misunderstanding the mathematical reasoning, misapplying a criterion or diverging from the instructor’s judgment. Because transcription accuracy was not independently annotated, the study could measure final image-to-decision performance but could not separate visual recognition errors from reasoning or rubric-application errors.

The study had two stages. In the rubric-generation experiment, Gemini 2.5 Flash produced 922 candidate analytic criteria from 143 problems, including 44 procedural–deterministic, 48 conceptual–deterministic, 27 constructive–structured, 22 reasoning–justification and only two design–open problems. Three faculty members from mathematics, computer science and computer engineering assessed each criterion for immediate usability and each problem-level rubric for completeness. The mean criterion-level satisfactory score was 0.782, while mean problem-level completeness was 0.708. These measures had different anchors and units of analysis, so they were not directly compared. Examples of instructor refinement showed that generated criteria could contain incorrect expected answers, omit acceptable methods or become unnecessarily restrictive. The results therefore treated rubric review as a substantive stage in which instructors retained, revised, expanded, removed or added criteria, rather than allowing generated rubrics to be applied without modification.

For the grading experiment, instructors used the original questions and human-authored rubrics as references, then created final rubrics containing 57 binary items across 31 question units in Calculus, Data Structures and Discrete Mathematics. The criteria were classified retrospectively as structural, correctness, constraint, method or process, and gatekeeping checks; the questions were similarly classified from procedural–deterministic through design–open. One researcher performed these classifications, and no inter-coder reliability estimate was available. They describe the sampled material rather than established general effects of question or rubric type. The experiment also lacked a matched condition using an undecomposed or holistic rubric prompt. The study could therefore evaluate reliability under rubric decomposition and constrained prompting, but could not determine whether either feature caused an improvement over ordinary free-form scoring.

Across all 3,346 rubric decisions, individual agreement with instructors ranged from 76.5% for GPT-4.1 to 87.6% for Gemini 2.5 Pro, with Claude Sonnet 4.6 at 80.5%. A full-coverage majority rule reached 83.7%, below Gemini’s result. Unanswered outputs were treated as zero for these full-coverage comparisons, with complete-case and other null-handling analyses used as sensitivity checks. Cluster-bootstrap intervals were also reported for the nested design, but the 3,346 decisions came from only 31 question units and 11 assessment sets. The figures therefore describe this educational sample rather than providing population-level estimates from 3,346 independent observations. Question-cluster bootstrap intervals for GPT-4.1, Gemini, Claude and majority agreement were 72.2%–80.9%, 84.3%–90.7%, 75.6%–84.9% and 79.2%–87.7%, respectively.

Selective routing separated agreement levels, although the accepted and referred figures use different denominators and should not be read as evidence that consensus outperformed every full-coverage model. Of 2,195 decisions accepted by complete three-model unanimity, 91.9% matched the instructor, an 8.1% discrepancy rate. Among the 1,151 decisions referred for review, a retrospectively calculated majority rule matched the instructor on 68.0%. Of the referrals, 888 had complete outputs with binary disagreement, 245 contained nulls without an observed conflict, and 18 contained both. Gemini agreed with instructors on 95.9% of the 245 null-only cases, compared with 74.8% of complete-output disagreements, indicating that missing output and substantive conflict were distinct sources of referral. Consensus was thus most useful as a mechanism for selecting cases for review, not as a calibrated probability that an accepted decision was correct.

Performance varied substantially among courses. Calculus had 66.0% decision coverage and 94.9% agreement within its accepted subset, while Data Structures had 57.9% coverage and 88.5% accepted agreement. Discrete Mathematics produced no missing outputs, giving it 84.1% coverage, but its accepted decisions agreed with the instructor on only 85.9%. The errors in that accepted subset were directional: models awarded credit in 14.1% of cases where the instructor withheld it, with no corresponding withhold discrepancy. A particularly revealing criterion required a valid symmetry justification in one six-part group assignment. Claude and GPT awarded it in all 31 responses, although the instructor awarded it in eight; Gemini matched the instructor in 13 of 31. Unanimity retained 24 responses, including 17 awards that contradicted the instructor, producing just 29.2% agreement on that criterion. The binary outputs could not establish whether these discrepancies arose from image reading, interpretation of the justification, rubric application or instructor judgment.

At the response level, all required criteria were acceptance-eligible in 60.9% of the 1,875 submission-question records: 63.7% in Calculus, 43.2% in Data Structures and 74.2% in Discrete Mathematics. The unit of review mattered: only nine of 31 complete six-part assignments in Discrete Mathematics passed without any referred criterion, even though 74.2% of their individual parts did. These proportions indicate where a workflow could direct instructor attention, but the study did not measure actual time savings, student outcomes, feedback uptake, appeals or operational safety. Every AI output remained subject to instructor approval, and the system was not deployed in time to influence grades or feedback. “Acceptance-eligible” therefore described a retrospective selection rule, not observed instructor acceptance or automatic release of a grade.

Agreement also varied when measured at the level of weighted problem scores. The mean question-level intraclass correlation coefficient, ICC(2,1), was 0.700 for full-coverage Gemini, compared with 0.638 for the primary unanimity subset. These values describe different selected samples and do not establish a superiority of consensus. Reasoning–justification items had a mean unanimity-subset ICC of just 0.052, compared with 0.415 for Gemini over the full samples, because accepted model scores were often constant while instructor scores varied. A mapped proof-oriented course outcome had 83.1% accepted agreement and a 16.9% award-discrepancy share, whereas two Calculus outcomes reached accepted agreements of 95.1% and 93.8%. Course, task, rubric and instructor characteristics were confounded, so these were descriptive contrasts rather than causal effects of course learning outcomes.

The study supports a supervised decision aid in which cross-model disagreement and incomplete output trigger human review, while also exposing the limits of consensus. Rubric decomposition made each decision inspectable, but it could not compensate for missing criteria or rigidly under-credit unconventional reasoning. The evidence also raises fairness concerns because rigid criteria may under-credit valid approaches that differ from dominant forms, while shared model heuristics may over-credit responses resembling familiar disciplinary patterns; however, the study did not test subgroup fairness. Local validation remains necessary because the research covered three courses at one institution, used Thai rubric text with English instruction wrappers, did not independently annotate transcription accuracy and did not include a matched holistic-rubric comparison. Exact model access dates, decoding settings and some governance details were also unrecorded, limiting replication and independent verification of data-handling safeguards. The practical conclusion is neither autonomous grading nor wholesale rejection of AI assistance: unanimity can focus review, provided accepted decisions are audited, error direction is monitored and instructors retain final authority over every grade.

Subject of Research: Retrospective evaluation of multimodal AI grading and selective instructor review for undergraduate mathematics and computing assessments

Article Title: Direct multimodal grading of image-based mathematics and computing responses: rubric-constrained decisions and cross-model selective review

Article References: Rukkanchanunt, T., Chaidee, S., Isradisaikul, C., & Buchatip, K. (2026). Direct multimodal grading of image-based mathematics and computing responses: rubric-constrained decisions and cross-model selective review. Frontiers in Education, 11, Article 1948818. https://doi.org/10.3389/feduc.2026.1948818

Image Credits: AI Generated

DOI: 10.3389/feduc.2026.1948818

Keywords: multimodal AI, automated grading, handwritten responses, selective review, rubric decomposition, cross-model agreement, instructor oversight, higher education, assessment validity, Direct, multimodal, grading