MMAnnotator Integrates RAG Chatbots for Educational Data Annotation

Researchers from the University of Frankfurt and other institutions have introduced MMAnnotator, a web-based software platform designed to streamline the annotation of multimodal data in educational sciences. The tool, published in the journal SoftwareX, integrates a Retrieval-Augmented Generation (RAG) chatbot to assist human annotators in evaluating complex web documents and student-generated content. This development addresses a growing need for flexible annotation environments that can handle diverse data types, including text, images, audio, and video, while maintaining the precision required for scientific research.

The primary motivation for MMAnnotator stems from the limitations of existing annotation frameworks, which are often restricted to specific data modalities or lack advanced automation features. As machine learning models become more powerful, the annotation process has shifted from purely manual tasks to hybrid workflows involving human-in-the-loop mechanisms. However, many current platforms fail to support the multimodal nature of modern educational data, where students interact with web pages, write statements, and consult various sources. MMAnnotator aims to bridge this gap by offering a multi-view annotation system that allows researchers to annotate the same documents from different perspectives without duplicating data.

The software architecture is built upon the Unstructured Information Management Architecture (UIMA), which provides a robust data model for handling multimodal structures. MMAnnotator extends the capabilities of TextAnnotator, a web-based platform for collaborative corpus annotation, and Universal Corpus Explorer (UCE), a tool for searching large annotated corpora. By synchronizing document states with UCE, MMAnnotator enables the use of a RAG-based chatbot that can access the specific content being annotated. This integration allows the chatbot to provide context-aware assistance, such as translating passages or describing visual elements, directly within the annotation interface.

A key feature of MMAnnotator is its support for project-based data management, which facilitates collaboration among multiple annotators. The system introduces the concept of projects as repositories for specific annotation tasks, allowing documents to be shared across different views. This structure supports multi-user environments with optional access levels, ensuring that annotators can work according to their group or project permissions. The platform also includes features for real-time statistical analysis, such as inter-annotator agreement, and supports the export of annotations in standard formats like CSV and CoNLL, which are essential for downstream data processing.

The development of MMAnnotator was driven by specific use cases from the Critical Online Reasoning in Higher Education (CORE) project. In this interdisciplinary study, students from fields such as medicine and economics were tasked with answering online questions about domain-specific scenarios. Their web browser interactions were recorded, generating a dataset of written statements and screenshots of consulted web pages. Annotating this data requires evaluating criteria such as text coherence, the number of sources consulted, and the relevance of the web content. MMAnnotator provides dedicated frontends for these tasks, displaying the student’s response alongside the relevant web page screenshot to guide the annotation process.

The RAG chatbot component is designed to assist annotators with tasks that would otherwise require external tools or significant manual effort. For instance, when students consult web pages in multiple languages, annotators can ask the chatbot to translate specific passages without leaving the annotation view. Similarly, for dense screenshots, the bot can describe visual elements or extract text, aiding in the assessment of the page’s content. The chatbot is integrated into the user interface in a way that allows it to fill in annotation form fields based on user requests. However, the system is designed to keep the final decision with the human annotator; the chatbot acts only on explicit requests, and all values it sets remain visible and editable.

To evaluate the platform, the researchers conducted a field deployment involving 22 participants, including doctoral researchers and computer science students. The participants were asked to annotate a student response and a corresponding web page screenshot using MMAnnotator. They were given the choice between two large language models: Qwen-2.5-VL-7B, which supports both text and images, and Qwen-3-8B, which is text-only. The Qwen-2.5-VL model was preselected due to its multimodal capabilities. After completing the annotation task, participants filled out a survey based on the Usability Metric for User Experience (UMUX) and additional questions regarding their preference for MMAnnotator over Excel and their experience with the chatbot.

The evaluation results indicated that annotators generally agreed that MMAnnotator met their requirements and was easy to use. There was a strong overall preference for using MMAnnotator over Excel for the annotation task, although one participant strongly disagreed. The study noted that this preference measure does not equate to efficiency, as the annotation time was not tracked. Regarding the chatbot, 17 of the 22 participants used it, sending an average of 6.22 messages each. The most common requests were for assessments of criteria values, explanations of annotation guidelines, and descriptions of image content.

The technical implementation of MMAnnotator includes a backend that extends UCE to support locally hosted large language models via the Ollama backend. This allows for the use of models without sending data to third-party servers, addressing data protection concerns. In the evaluation, all models were hosted locally, ensuring that no student data left the operator’s infrastructure. The frontend, built on the React JavaScript framework, provides a login page for authentication and a management interface for administrators to manage projects, repositories, and user access. The system also supports the integration of NLP pipelines through DUUI, enabling automatic pre-annotation and real-time statistical analysis.

Despite its advantages, MMAnnotator has several limitations. The evaluation was a field deployment rather than a controlled experiment, and the sample size was relatively small. The behavioral analyses of chatbot use are therefore exploratory, and it is possible that participants’ requests reflected testing of the platform’s features rather than typical production use. Additionally, the system does not currently track the time spent on individual documents, limiting the ability to make precise efficiency claims. The authors acknowledge that LLM suggestions can anchor human judgments and lead to over-reliance on automated advice. To mitigate this, MMAnnotator allows projects to disable the chatbot entirely, ensuring that gold-standard annotations can be collected without automated interference.

As the need for annotated data grows in fields such as linguistics, legal studies, and biomedical research, tools that can handle multimodal data with flexibility become increasingly important. The platform’s open-source availability under the AGPL v3 license allows the community to use and extend the software, potentially leading to broader applications in other domains. Future work includes improving the chatbot’s history management, enhancing performance, and conducting controlled studies to evaluate the chatbot’s effect on annotation time and quality.

Its design emphasizes the importance of human oversight in the annotation process, ensuring that automated assistance enhances rather than replaces expert judgment. By providing a comprehensive environment for multimodal annotation, MMAnnotator supports the growing complexity of educational data and the need for efficient, accurate annotation workflows. The tool’s integration with existing platforms like TextAnnotator and UCE ensures compatibility with established research practices, making it a valuable resource for researchers in educational sciences and beyond.

Subject of Research: MMAnnotator : Enhancing the annotation of web documents in the field of educational sciences with a RAG chatbot

Article Title: MMAnnotator : Enhancing the annotation of web documents in the field of educational sciences with a RAG chatbot

Article References: Schrottenbacher, P., Baumartz, D., Abrami, G., Mehler, A., & Gombert, S. (2026). MMAnnotator: Enhancing the annotation of web documents in the field of educational sciences with a RAG chatbot. SoftwareX, 36, Article 103188. https://doi.org/10.1016/j.softx.2026.103188

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103188

Keywords: MMAnnotator, Enhancing, annotation, documents, field, educational, sciences, chatbot, scientific research