The core insight behind MSIPM is that human graders do not apply scoring criteria mechanically. When a student produces an unexpected but valid answer, a human evaluator can step back, reconsider the rubric, and adjust it before assigning a final mark. The authors illustrate this with a scenario in which an examiner asks students to design a scalable architecture for an online food delivery application.

If the initial rubric demands load balancing, caching, database replication and microservices, a candidate who proposes an event-driven architecture built on asynchronous queues and serverless components could be unfairly marked wrong, even though the design fully satisfies the intended goals of scalability, reliability and fault tolerance. A human would notice the mismatch and refine the criteria; most automated systems would not. MSIPM addresses this by splitting the grading workflow into two phases: a slow, offline rubric-refinement phase and a fast, online scoring phase.

In the refinement phase, three large language models — Mistral-7B-v0.1, Llama 2 7B and BAAI's WuDao 2.0 7B — are given the assessment question, an initial rubric, and a stratified sample of twenty student essays per prompt, drawn from low, medium and high score bands to avoid redundancy and computational waste. Each model independently critiques the rubric, suggesting improvements. A cross-model comparison step then aggregates the feedback, identifies ambiguities, missing criteria and inconsistencies against the human-assigned reference scores, and produces an updated rubric.

This loop repeats until convergence or a maximum of three iterations, yielding a final refined rubric that is then frozen for scoring. The performance gains from this iterative process are striking. On essay set one of the ASAP dataset, the initial rubric achieved a QWK of 0.58.

After the first refinement iteration, agreement with human raters rose to 0.65; the second and third iterations pushed it to 0.71 and 0.76 respectively. A final consolidation and validation step produced the optimized rubric, which reached 0.86 — an overall improvement of 48.28 percent from the starting point. The authors report that convergence was observed after three cycles on that essay set, with no further significant gains, although the number of useful iterations varied across other essay sets.

The ablation study reveals which components of the framework carry the weight. Starting from a single-LLM baseline with the initial rubric, the average QWK across all eight essay sets was just 0.43. Adding multi-LLM consensus lifted the average to 0.56, a gain of 0.13 points, suggesting that aggregating judgments from several models reduces model-specific quirks in how scoring criteria are interpreted.

Iterative rubric refinement delivered the largest single improvement, raising the average from 0.56 to 0.67. Self-reflection prompting, in which the model explicitly reconsiders its own score, added another 0.03 points, and the final verification stage contributed a further 0.01, bringing the complete pipeline to 0.71 — an absolute gain of 0.28 QWK points over the baseline. On essay set one, the full configuration scored 0.86 compared with 0.41 for the single-model setup.

The verification mechanism deserves particular attention because it tackles one of the most persistent criticisms of LLM-based assessment: hallucination and anomalous scoring. When the three models' predicted scores for an essay differ by more than one point, the framework flags the response as anomalous and triggers a re-evaluation procedure. Because the models were run with temperature set to 0.0 and no persistent conversational memory across essays, each evaluation was treated as a controlled inference step, with only explicitly supplied refinement context available to the models.

Invalid outputs were regenerated once; if that failed, the essay was marked as a scoring failure rather than assigned a fabricated number. Prompt design proved to matter enormously. The researchers tested four strategies — one-shot, few-shot, self-reflection and batching — each combined with different role-oriented personas such as teacher, educational researcher and creative writing mentor.

One-shot prompting produced stable outputs across all personas but failed to capture subtle rubric distinctions. Few-shot prompting showed more variation but aligned poorly with manual annotations under complex personas. Self-reflection prompting achieved the best alignment with human scoring, with the iterative self-correction mechanism improving consistency across the four scoring dimensions of elaboration, organization, fluency and audience awareness.

Batching, in which multiple student answers are graded jointly to check rubric fairness, performed reasonably but suffered occasional inconsistencies attributed to conflicting interpretations of the scoring rules. The comparisons with established baselines are instructive. Fine-tuned BERT models delivered moderate, stable scores across the essay sets, while GPT-4, evaluated without structured guidance, showed inconsistent and often low performance, failing to outperform the transformer baselines on several sets.

LLaMA 7B performed better than GPT-4 but varied sharply across prompts, and Llama 2 7B and WuDao 2.0 7B alone showed comparatively weak and unstable results. MSIPM achieved the highest or near-highest QWK on most essay sets, outperforming prior methods such as Joint Essay Scoring and Protact on sets 1, 3, 6 and 7. The authors are careful to note that prior work using GPT-4 with extensive prompting reached a higher peak of 0.92 QWK, but they frame their contribution differently: their framework uses smaller, free, non-commercial models that can be deployed locally by schools, avoiding ongoing API costs and student data privacy concerns.

Not everything scaled uniformly. Essay sets 1 and 3 hit the 0.86 peak, while sets 4 and 8 lagged at 0.56 and 0.57. The authors attribute this variation to prompt-specific characteristics rather than architectural instability: the ASAP sets differ in genre, from source-dependent textual analysis to narrative and argumentative tasks, in average essay length, and in grading scales that range from 0–3 points up to 0–36 points.

Because QWK is sensitive to the number of ordinal score classes and their distribution variance, non-uniform performance across sets is a well-documented feature of AES benchmarks. The team also took explicit precautions against overfitting: rubric refinement used 75 percent of the data, the held-out 25 percent was used only for final evaluation, and no test-set samples or labels informed prompt construction or rubric design. The researchers are candid about the limits of their work.

The evaluation relied on a single dataset, so generalization to other domains, educational levels and essay types remains untested. Running multiple LLMs through iterative refinement increases computational and inference demands compared with single-model approaches. The framework remains sensitive to prompt wording in some cases, and the authors call for more research on the interpretability, transparency and reproducibility of individual scoring decisions.

Their conclusion is measured rather than triumphant: LLM-based essay scoring should be treated as an assistive tool rather than a replacement for human evaluation, with human oversight remaining essential for high-stakes assessment. Future directions include multimodal assessment of diagrams and mathematical expressions, explainability mechanisms that show graders the evidence behind each score, and adaptive prompting that adjusts evaluation criteria to the characteristics of each essay prompt. What the study establishes, within its scope, is that letting several AI models debate and refine the rules before grading — rather than trusting any single model's judgment — measurably moves automated scoring closer to the way humans actually evaluate writing.

Subject of Research: A collaborative multi-LLM framework with iterative self-reflection prompting for automated essay scoring

Article Title: A collaborative multi-LLM framework with self-reflection iterative prompting for automated essay scoring

Article References: Johnsi, R., & Kumar, G. B. (2026). A collaborative multi-LLM framework with self-reflection iterative prompting for automated essay scoring. Frontiers in Artificial Intelligence, 9, 1930428. https://doi.org/10.3389/frai.2026.1930428

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1930428

Keywords: automated essay scoring, large language models, prompt engineering, self-reflection, rubric refinement, Quadratic Weighted Kappa, ASAP dataset, multi-LLM consensus, educational assessment, natural language processing, hallucination detection, AI in education