Researchers investigating automated assessment methods have demonstrated that extending pretrained language models with domain-specific knowledge from an environmental ontology significantly improves the automatic evaluation of students’ written scientific explanations. The primary objective of the experimental investigation was to determine whether integrating transformer models such as Bidirectional Encoder Representations from Transformers, commonly known as BERT, with structured vocabulary mappings could accurately classify undergraduate students’ levels of scientific expertise. Published in Disciplinary and Interdisciplinary Science Education Research, the study focused on constructed responses examining the Food-Energy-Water Nexus within college-level environmental science courses. By bridging natural language processing algorithms with the Environment Ontology, the investigators addressed the persistent challenge of recognizing technical disciplinary terms that are absent from general machine-learning vocabularies, showing that structured ontological frameworks provide an effective mechanism for capturing student systems thinking without requiring large-scale model pretraining from scratch.
Evaluating scientific expertise in higher education increasingly emphasizes knowledge-in-use skills, where learners must apply core conceptual principles rather than merely reciting memorized definitions or identifying isolated surface features. In modern science frameworks, achieving proficiency involves understanding complex systems by identifying interconnections, feedback mechanisms, and resource trade-offs across environmental domains. Open-ended, constructed response assessment items serve as a vital evaluative tool because they require students to generate explanations in their own words, revealing the depth of their cognitive reasoning. However, manual scoring of written constructed responses demands significant time and labor from instructional staff, creating a substantial bottleneck for formative assessment and personalized pedagogical interventions. Automated scoring systems powered by artificial intelligence offer a promising avenue to relieve this grading burden, yet traditional algorithmic scoring approaches often struggle to discern meaningful conceptual relationships embedded within nuanced student explanations.
While natural language processing has advanced considerably with the advent of large pretrained language representation architectures, direct application of these models to educational assessments encounters notable technical hurdles. Standard transformer models, including Google’s BERT, are pretrained on extensive general-purpose corpora such as Wikipedia and Book Corpus, utilizing fixed tokenization vocabularies of approximately 30,522 distinct tokens. When applied to specialized academic domains, these models encounter out-of-vocabulary challenges because students frequently incorporate discipline-specific terminology that standard tokenizers fragment or fail to recognize. Furthermore, educational datasets are characteristically small-scale and class-imbalanced, offering limited data volume for fine-tuning complex neural networks. Consequently, generalized embedding representations often yield suboptimal performance in domain-specific tasks because the underlying algorithms cannot adequately connect open-domain language patterns with the task-specific conceptual structures required for evaluating student scientific reasoning.
To systematically evaluate automated scoring under these challenging educational conditions, the research team analyzed a corpus of 1,293 constructed responses collected from undergraduate students enrolled in introductory interdisciplinary environmental science courses. The data collection was originally conducted for a collaborative research project developing a concept inventory to assess systems thinking surrounding the Food-Energy-Water Nexus. Participants were drawn from introductory courses across seven post-secondary institutions in the United States, intentionally selected to represent the three major categories of four-year institutions defined by the Carnegie Classifications of Institutions of Higher Education as well as the three primary curricular design approaches in interdisciplinary environmental science programs. The investigated responses were gathered from Qualtrics surveys administered pre- and post-course discussion between Spring 2022 and Spring 2023, utilizing two distinct assessment items: an energy reservoir question focused on identifying sources and connections, and a biomass energy question focused on explaining multidimensional system trade-offs.
Scoring the student responses required establishing a rigorous coding framework capable of translating complex reasoning patterns into reliable quantitative labels. The research team first designed dichotomous analytic rubrics aligned with core disciplinary concepts, reading student submissions through an inductive-deductive process based on established knowledge dimensions. Two researchers scored responses independently, resolving discrepancies through consensus discussions across five iterative rounds until achieving an interrater reliability agreement level of 0.85 across all assessment items. Following the analytic phase, the authors synthesized these descriptors into a holistic coding scale ranging from level zero to level four. Level zero designated off-task or nonsensical responses, such as answering with an admission of ignorance, whereas level four represented advanced expertise characterized by coherent multi-component reasoning chains and explicit trade-offs across food, energy, and water domains, regardless of whether students used specific disciplinary terminology.
An initial examination of the 1,293 scored responses revealed a pronounced class imbalance across the five expertise levels, with polar scores containing substantially more student entries than intermediate categories. To prevent skewed distributions from degrading model generalizability during training, the researchers implemented a sentence-level text augmentation procedure using the NLPAUG software package driven by contextual word embeddings. Each student constructed response was augmented five times without altering domain-specific vocabulary, expanding the training corpus to a total of 6,465 augmented responses. A paired t-test comparing model evaluations confirmed that this contextual data augmentation produced a statistically significant improvement in Quadratic Weighted Kappa scores, recording t(4) = 13.43, standard error = 0.017, and p < 0.001. This augmentation strategy provided the diverse linguistic variations necessary to prevent model overfitting while preserving the computational feasibility of training on moderate sample sizes.
The core methodological contribution of the investigation centered on constructing a three-layer model architecture comprising an ontology-based system layer, a BERT encoder layer, and a final text classification layer. For the domain-specific ontology layer, the researchers utilized the Environment Ontology, an established hierarchical knowledge framework capturing biomes, environmental features, and materials. After removing standard English stopwords from student responses, the team mapped the remaining words to nodes within the Environment Ontology using the text2term ontology mapper in Python. Semantic similarity was calculated using term frequency-inverse document frequency vectors, with a cosine similarity threshold set at 0.70 based on prior natural language processing precedents. Terms exceeding this threshold were identified as domain-specific academic vocabulary, while common words already present in the BERT glossary, such as lake, were retained without substitution to avoid unnecessary processing complexity.
When the ontology mapper identified an out-of-vocabulary domain-specific term, the sibling extraction module located three closely related sibling terms within the ontological knowledge graph. Selection criteria prioritized terms exhibiting strong hierarchical proximity within the Environment Ontology while ensuring that candidate sibling terms were present in the standard BERT glossary. For instance, when encountering the domain-specific word photosynthesis, the system identified valid substitutes such as cellular process, light reaction, and dark reaction, deliberately avoiding specialized terms like photosystem that were also absent from the BERT vocabulary. The substitution module then generated three modified sentences, each containing one selected sibling term. During the encoding stage, these modified sentences were processed through BERT to yield three 768-dimensional feature vectors capturing contextual semantic nuances, which were subsequently averaged to produce an enriched composite representation for downstream text classification.
The experimental setup compared the ontology-augmented language models against established natural language processing baselines under standardized computational conditions. The researchers tested the Constructed Response Classifier, an eight-algorithm supervised machine learning ensemble implemented in R, alongside standalone pretrained BERT and SciBERT models. The neural network experiments utilized the bert_base_uncased model from Huggingface, configured with twelve self-attention layers, twelve attention heads, and a hidden dimension of 768. Fine-tuning was executed on an NVIDIA 1080Ti graphics processing unit using the Adam optimizer with a learning rate of 1e-4, batch size of 10, input sequence length of 128, weight decay of 0.0005, dropout rate of 0.3, and 20 training epochs. The dataset was partitioned into a holdout test set of 239 responses and a training and validation set of 1,000 samples evaluated across ten randomly divided cross-validation folds.
Experimental results confirmed the superior performance of transformer architectures over classical machine learning, as well as the clear advantages of incorporating ontological knowledge. The supervised Constructed Response Classifier baseline demonstrated suboptimal predictive accuracy across all metrics, reflecting the inherent limitations of feature-engineered models when handling multi-class text classification with limited training instances. In contrast, both standalone pretrained models achieved Quadratic Weighted Kappa values exceeding the acceptable agreement threshold of 0.70, with SciBERT consistently outperforming base BERT. Integrating the Environment Ontology layer delivered further performance gains across weighted kappa, classification accuracy, and macro-averaged F1 scores for both architectures. The combination of SciBERT and the ontology-based system achieved the highest overall performance, reaching a peak Quadratic Weighted Kappa of 0.853, indicating strong concordance between automated machine scoring and human expert ratings.
Qualitative examination of student responses illustrated that the ontology-enhanced framework excelled at correctly categorizing answers characterized by dense specialized vocabulary, such as responses incorporating terms like ethanol to represent biofuel energy trade-offs. Nevertheless, the authors highlighted clear operational limitations, noting that prediction errors persisted in cases where student answers were extremely brief, contained off-task phrasing, or included typographical errors such as spelling produced as prodouced. The researchers also emphasized that their ex post facto study design could not account for variations in student language proficiency or instructional differences across participating institutions, urging future investigations to integrate fairness checks and equitable assessment frameworks for linguistically diverse cohorts. Ultimately, the findings confirm that combining domain-specific ontologies with contextualized word embeddings provides a robust, computationally efficient mechanism for evaluating complex scientific reasoning in educational settings.
Subject of Research: Ontology-extended language models for automated scoring of student scientific explanations
Article Title: Extending a pretrained language model (BERT) using an ontological perspective to classify students’ scientific expertise level from written responses
Article References: Wang, H., Haudek, K. C., Manzanares, A. D., Romulo, C. L., Royse, E. A., & Azzarello, C. B. (2025). Extending a pretrained language model (BERT) using an ontological perspective to classify students’ scientific expertise level from written responses. Disciplinary and Interdisciplinary Science Education Research, 7(1), Article 26. https://doi.org/10.1186/s43031-025-00149-5
Image Credits: AI Generated
DOI: 10.1186/s43031-025-00149-5
Keywords: automated scoring, BERT, SciBERT, Environment Ontology, constructed response, systems thinking, science education, natural language processing, Extending, pretrained, language, model
Cite Scienmag News
Courtney Benton. (November 11, 2026). Ontology Framework Enhances Language Models for Grading Student Science Responses. Scienmag. https://scienmag.com/ontology-framework-enhances-language-models-for-grading-student-science-responses/
Courtney Benton. "Ontology Framework Enhances Language Models for Grading Student Science Responses." Scienmag, 11 November 2026, https://scienmag.com/ontology-framework-enhances-language-models-for-grading-student-science-responses/. Accessed 18 November 2026.
Courtney Benton. "Ontology Framework Enhances Language Models for Grading Student Science Responses." Scienmag. November 11, 2026. https://scienmag.com/ontology-framework-enhances-language-models-for-grading-student-science-responses/








