Observational study finds context-dependent AI grading discrepancies in professional accounting exams

An observational study of professional accounting examinations found that a task-based machine-learning grading system produced grades that differed systematically from human assessors across performance levels and subject areas. The analysis found no significant gender-based grading bias, but higher AI confidence was associated with fewer substantial mismatches, with that relationship varying by subject. The study used archival records from South Africa’s Assessment of Professional Competence, or APC, an examination pathway for professional accounting certification.

The dataset included 15,507 complete grading records used in regression analyses and covered eight accounting-related subjects: accounting, taxation, financial management, audit, strategy, decision-making, technology and ethics. Human-assessed long-essay responses were compared with grades and confidence scores generated by a supervised machine-learning pipeline developed for the APC assessment. The pipeline was not a generative large language model producing grades directly.

It converted preprocessed responses into semantic vector representations with the nomic-embed-text-v1 sentence-embedding model, then used logistic regression classifiers trained on previously human-assessed scripts. Training and testing subsets were divided in a 75/25 ratio. The system generally assigned grades from 2 to 4 and produced a confidence score from 0 to 100 for each prediction.

Because human assessors rarely assigned the extreme grades 0, 1 and 5, cases in those categories were flagged for human review, as were short responses judged to contain insufficient semantic information for reliable automated grading. The researchers defined a significant mismatch as an AI grade differing from the human grade by more than one point. Analyses included descriptive statistics, Welch’s independent-samples t-test for the male-female comparison, one-way analysis of variance with Tukey post-hoc comparisons for performance level, proficiency category and subject area, and binary logistic regression for the probability of mismatch.

A final interaction model tested whether the association between AI confidence and mismatch varied across subjects. Performance level showed the strongest discrepancy gradient. In the low-performance quartile, the mean AI-minus-human grade difference was 1.14 points, with a standard deviation of 0.966, indicating higher AI grades.

In the high-performance quartile, the mean difference was −0.638 points, with a standard deviation of 0.624, indicating higher human grades. The difference across performance groups was statistically significant, F(3, 15503) = 6806, p < 0.001, and Tukey tests found significant pairwise contrasts. The authors cautioned that the AI system’s restricted grade range of 2 to 4, compared with the human range of 1 to 5, may have amplified this gradient.

Discrepancy also differed significantly across the eight subject areas, F(7, 15499) = 36.75, p < 0.001. The results indicate that grading consistency was not invariant across disciplines. The study considered possible influences such as subject-specific nuances, response structure and professional reasoning, but did not establish which mechanism produced the observed differences.

Gender was not associated with grading discrepancy. The male-female comparison was not statistically significant, with Welch’s t = −0.81, p = 0.42 and Cohen’s d = −0.013. Gender and proficiency category also did not significantly predict mismatch in the logistic models.

Performance level, subject area, human grade and AI confidence were associated with the probability of disagreement. In the baseline model, higher AI confidence was associated with a lower probability of mismatch, with a coefficient of −3.721 and p < 0.001. The retained subject-by-confidence interaction model improved the Akaike information criterion to 5575.53, increased the area under the curve from 0.897 to 0.901 and raised McFadden’s R-squared from 0.406 to 0.410.

Average marginal effects for confidence ranged from −1.547 for subject E to statistically nonsignificant values for subjects A and G. These model estimates indicate that identical confidence values did not have the same predictive relationship with grading agreement across all subjects. The study has several bounded limitations.

It examined one machine-learning grading system in one South African professional accounting setting, so it cannot establish whether the inconsistencies are model-specific or common to other AI graders. Including human grade as a predictor may have introduced statistical dependence between predictors and outcome, while the narrower AI grading scale means part of the performance-level pattern may reflect scale constraints rather than calibration alone. The binary mismatch measure also does not fully capture the magnitude of grading differences, and the relatively modest pseudo-R-squared suggests that unobserved factors may remain important.

Subject of Research: Context-dependent AI grading and confidence calibration in professional accounting assessment

Article Title: Understanding discrepancy in AI-human assessment systems due to hidden calibration inconsistency

Article References: Gold, N. O. (2026). Understanding discrepancy in AI-human assessment systems due to hidden calibration inconsistency. Discover Artificial Intelligence, 6(1), Article 1272. https://doi.org/10.1007/s44163-026-02226-8

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02226-8

Keywords: AI grading, assessment, professional accounting, calibration, grading discrepancy, confidence estimates, conditional fairness, machine learning, observational study, Understanding, discrepancy, AI-human