Open-Source AI Grading Platform EvaluAI Turns LLMs Into Classroom Assessment Tools

Grading open-ended exams has long been one of the most stubborn bottlenecks in higher education. Multiple-choice tests can be scored by machines in seconds, but essays, proofs, and algorithm design problems demand human judgment, and human judgment is slow, expensive, and inconsistent. A team at the University of Extremadura in Spain has now released EvaluAI, an open-source platform that uses large language models to grade free-text student work, generate feedback, and hand the results to instructors for review. The software, described in the journal SoftwareX, is released under the Apache-2.0 license and is designed to be deployed on the user’s own infrastructure rather than locked behind a commercial cloud service.

The motivation behind the project is straightforward. Rising student-to-teacher ratios impose a growing administrative burden on faculty, while formative assessment requires continuous feedback to support learning. Manual grading of open-ended assessments is prone to subjectivity, grader fatigue, and significant delays. EvaluAI is designed to address the scalability challenges of qualitative evaluation by supporting timely and standardized feedback for large student cohorts. Unlike optical mark recognition systems, which are limited to multiple-choice questions, the platform leverages large language models to perform semantic analysis of free-text responses, enabling automated yet semantically informed assessment of what students actually wrote.

The system follows a three-stage workflow. First, instructors formalize assessment criteria by defining questions, scoring weights, and criterion descriptions in a machine-readable rubric. Second, students’ submissions are ingested as individual or batch PDF documents, including scanned handwritten exams. Third, EvaluAI generates a quantitative score and qualitative feedback that the instructor can audit, correct, and confirm through a side-by-side review interface, supporting transparency and academic integrity. This human-in-the-loop design distinguishes the platform from fully autonomous grading pipelines: the software distinguishes between so-called temporal exams, which are preliminary AI-generated corrections, and definitive exams, which are audited and confirmed results.

Architecturally, EvaluAI follows a modular client-server pattern organized into three layers. The presentation layer is a single-page application built with React, prioritizing user experience through dynamic state management and asynchronous navigation. The logic layer runs on a FastAPI backend written in Python, where RESTful endpoints validate client requests and delegate processing to specialized components for user, rubric, and exam management, as well as to a dedicated orchestration service that coordinates document processing, prompt construction, and integration with external LLM services. Data persistence is handled by MongoDB Atlas, a cloud-native NoSQL database whose document-oriented model enables flexible schemas for academic rubrics and exam metadata.

Security and account management receive explicit attention. The platform manages individual user accounts, storing credentials as salted bcrypt hashes, and enforces token-based JWT authentication with per-resource ownership checks across all API endpoints. The automated evaluation orchestrator integrates with the Google Gemini 3.6 Flash API, selected as a cloud-based LLM service due to its accessibility and documented relevance in educational contexts. Its modular design isolates LLM provider-specific interactions from the rest of the evaluation workflow, meaning institutions can swap in alternative models, including OpenAI-compatible clients, without rewriting the core grading pipeline. Batch processing allows the system to partition large PDF aggregates into individual student records based on a user-specified number of pages per submission, so an entire cohort can be evaluated in a single operation.

The validation study is where the findings become genuinely interesting, and genuinely nuanced. The team evaluated the platform using real student submissions from an Analysis and Design of Algorithms course at the University of Extremadura. The initial dataset comprised submissions from 14 students, one of which was excluded because its corresponding instructor score could not be retrieved from the original grading records, leaving 13 submissions. Each submission was evaluated under two conditions: without a structured rubric, and with rubric-guided evaluation using the formalized rubric defined within the platform. The same Gemini model and generation configuration were used in both conditions, with a temperature of 0.3 and a maximum output length of 8000 tokens. To account for the non-deterministic behavior of LLMs, each submission was independently evaluated three times under each condition, producing 78 evaluations in total.

The headline result is a paradox that anyone deploying AI grading should understand. Without a rubric, the normalized agreement metrics against the instructor’s grades were a mean absolute error of 0.110, with rubric-guided evaluation yielding a slightly higher MAE of 0.132. In other words, adding the structured rubric did not improve agreement with the original instructor grades. The authors attribute this to a timing mismatch: the reference grades were assigned during the original course assessment, before the structured rubric used by EvaluAI was formalized, and the rubric-guided condition applies explicit penalties, including aspects such as code documentation, that may not have been weighted the same way during the original grading. The AI was not necessarily wrong; it was grading against a different, more explicit standard than the one the human used.

Where the rubric clearly paid off was in consistency. Because LLMs are stochastic, the same submission can receive different scores on repeated runs, a serious problem for any assessment system. Without a rubric, the mean standard deviation of the three repeated total scores across submissions was 0.045 and the intraclass correlation coefficient, computed as ICC(2,1) following the two-way random-effects formulation of Shrout and Fleiss, was 0.910. With rubric-guided evaluation, the mean standard deviation dropped to 0.016 and the ICC rose to 0.986. That is a substantial tightening of run-to-run stability, suggesting that encoding grading criteria into digital rubrics constrains model reasoning in a way that makes scores far more reproducible, even when those scores differ from a particular human grader’s judgment.

The operational economics are equally notable. A complete batch of 13 submissions required approximately 4.5 to 7.5 minutes, and all 78 evaluations completed successfully. Based on the Gemini 3.6 Flash API pricing applicable at the time of the study, the estimated cost was roughly $0.006 to $0.010 per submission evaluation. For instructors facing stacks of hundreds of exams, that translates into near-instant formative feedback at a cost of pennies per student, with the instructor retaining final authority through the audit interface. The authors are careful to note the limitations: the reference grades came from a single instructor and were not produced using the formalized rubric, the experiment does not isolate the effect of rubric structure from the additional grading information provided, feedback quality was not formally assessed, and the instructor did not audit the AI-generated evaluations, so correction rates could not be observed.

Beyond the classroom, the researchers argue the platform opens research questions that were previously constrained by the time and scalability limits of manual grading. These include studying how near-instantaneous feedback affects retention of complex technical concepts compared with traditional delayed grading cycles, examining how rubric granularity influences inter-rater reliability between human experts and autonomous agents, and tracking evolving patterns of student misconceptions across thousands of responses and multiple semesters to identify systemic gaps in a curriculum. Compared with established systems such as e-rater, Gradescope, and Turnitin Feedback Studio, and with recent LLM-based research frameworks like Grade-Like-a-Human, GradeHITL, LLM-Rubric, and CourseEvalAI, EvaluAI stands out by combining open-source licensing, structured rubrics, handwritten and PDF support, batch grading, human audit, and user-controlled deployment in a single platform. Future work will target better interpretation of complex diagrams and handwritten schemas, systematic comparison of different LLMs, and complete per-evaluation provenance recording to make every grading decision fully auditable.

Subject of Research: An open-source LLM-based platform for automated evaluation of student submissions and feedback generation

Article Title: EvaluAI: An open-source LLM-based framework for automated evaluation of student artifacts and feedback generation

Article References: Barriga, J. A., Moruno, J., Barriga, A., Domínguez Arjona, J., & Clemente, P. J. (2026). EvaluAI: An open-source LLM-based framework for automated evaluation of student artifacts and feedback generation. SoftwareX, 36, Article 103136. https://doi.org/10.1016/j.softx.2026.103136

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103136

Keywords: EvaluAI, large language models, automated grading, educational technology, rubrics, formative assessment, learning analytics, open-source software, human-in-the-loop, Gemini, University of Extremadura, SoftwareX