Language Models Provide Individualized Feedback for Physics Teacher Reflections

Continuous professional development among science educators remains fundamentally tethered to their capacity to critically evaluate instructional practice and navigate classroom uncertainties. Investigating how computational methods can sustain this process, education researchers Stefan Sorge, Peter Wulff, and Marcus Kubsch explored whether fine-tuned language models can provide automated, individualized diagnostic guidance on written reflections generated by pre-service physics teachers. The investigation focused on transferring a transformer-based machine learning model developed at one educational institution into a separate university setting, probing both the technical generalizability of automated sentence categorization and the observable shifts in student analytical depth across an entire semester. The results indicated that pre-trained representations generalized sufficiently across distinct institutional cohorts while demonstrating marked performance gains when supplemented with targeted local training data.

Science teaching is inherently filled with pedagogical ambiguity, encompassing unpredictable student prior knowledge, variable engagement during experimental activities, and unanticipated classroom events that demand immediate adjustments. Educational researchers conceptualize these critical junctures as catalysts for teacher growth when instructors systematically identify gaps between instructional intentions and actual classroom outcomes. Frameworks such as the ALACT model delineate reflection as a multi-stage progression moving from concrete action and retrospective review to essential awareness, alternative generation, and trial. To establish measurable criteria for reflective quality, science education models integrate structural depth, contrasting descriptive accounts of the learning environment with deeper evaluative stances that formulate alternative strategies, weigh pedagogical justifications, and deduce actionable consequences for future lessons.

Despite widespread recognition that structured reflection constitutes an essential professional competency within international teaching standards, novice educators frequently struggle to achieve substantive analytical depth without explicit feedback. Empirical baseline investigations consistently reveal that pre-service teachers predominantly produce descriptive narratives of observed events rather than rigorous pedagogical critiques or alternative instructional scenarios. While authentic video vignettes and classroom casework offer low-stakes opportunities to practice reflective analysis, observational exposure alone does not consistently foster deeper thinking. Individualized feedback from expert instructors provides the necessary scaffolding to guide reflective progress, yet manually assessing written journals represents an exceptionally resource-intensive task for university faculties, frequently causing formative assessment opportunities to remain underutilized in higher education.

To alleviate these evaluative bottlenecks, natural language processing architectures have emerged as promising mechanisms for automating reflective writing analytics. The research team implemented a semester-long intervention within a physics teacher education Master of Education course at a university in northern Germany, denoted as Context B. The cohort comprised fifteen enrolled pre-service teachers, of whom twenty percent identified as female, with all participants completing the curriculum. The instructional program coupled thirteen paired ninety-minute lectures and application seminars with five sequential reflection activities grounded in real classroom video vignettes. Students composed written reflections of approximately half a page to a page within twenty minutes of viewing each vignette, submitting their coursework to a shared repository for computational analysis.

The underlying machine learning classification framework relied on the Bidirectional Encoder Representations from Transformers architecture, known as BERT, which had originally been fine-tuned using written reflections and video vignette analyses collected in an earlier study at a different institution, denoted as Context A. BERT leverages transformer mechanisms to attend to long-range token interdependencies and positional embeddings across text sequences, making it particularly capable of parsing complex syntax. Automated sentence segmentation split the student reflections into elementary coding units, which the model categorized across structural dimensions derived from the reflection framework: descriptions of circumstances, descriptions of behavior, evaluations of action, formulations of alternatives, and derivations of consequences.

Automated feedback reports were generated and dispatched by email to participating pre-service teachers within forty-eight hours of submission, adhering to feed-back and feed-forward instructional principles. Following an initial baseline assignment and introductory instruction on reflective models, the automated reports communicated the relative percentage of each structural category identified in the text alongside targeted recommendations. For instance, when evaluative segments fell below five percent, the system prompted participants to expand evaluations, alternatives, and consequences in subsequent submissions. Conversely, excessive descriptive narrative exceeding thirty percent triggered prompts urging students to curtail background scene setting in favor of deeper pedagogical critique. Instructors later complemented this structural guidance with qualitative commentary assessing whether evaluative claims incorporated sound educational justifications.

To evaluate cross-institutional model transferability under the primary research question, the investigators compared classification performance across four distinct training and testing configurations. When the model fine-tuned exclusively on Context A training data was deployed directly to categorize unseen Context B test data without any hyperparameter adjustment or local tuning, it achieved an overall weighted F1 score of 0.72. The model exhibited reliable identification across dominant categories, attaining an F1 score of 0.71 for proposed instructional alternatives despite the relative scarcity of that code. However, consequences represented an exceedingly rare category in the test corpus with only eleven labeled instances, yielding a restricted recall of 0.09 and an F1 score of 0.17.

When the researchers augmented the training pipeline by combining Context A and Context B datasets to fine-tune raw BERT parameters before testing on Context B, overall classification performance improved substantially, elevating the weighted F1 score to 0.77. While the persistent rarity of consequence instances kept its category-level recall constrained, precision reached 1.00 when the classifier predicted positive instances. The investigators also demonstrated bi-directional benefits by using the merged multi-site dataset to re-evaluate Context A test benchmarks, observing marked performance gains across all annotated elements compared to models trained on single-site data alone, which underlines the empirical value of collaborative data sharing and open science across educational institutions.

Tracking longitudinal student development across the five sequential assignments revealed discernible shifts in how pre-service teachers allocated prose between descriptive reporting and evaluative analysis. Descriptive segments initially dominated the written submissions, but direct instruction following the baseline submission catalyzed a substantial immediate increase in evaluative proportions. Although a partial reversion occurred in the third reflection, subsequent iterations exhibited a steady upward progression in evaluative thinking. By the fifth writing cycle, evaluative components—comprising evaluations, alternatives, and consequences—constituted approximately fifty percent of total coded segments across participants. Individual trajectories mirrored this overarching pattern, as the student with the lowest initial evaluative baseline progressed from eight percent to sixty percent, while the highest baseline student advanced from fifty-seven percent to fifty-two percent after reaching a mid-semester peak of seventy-one percent.

The investigators emphasized several methodological boundaries when interpreting the pedagogical outcomes, noting that the single-cohort case study design did not incorporate an isolated control group. Consequently, observed gains in analytical depth reflect the combined influence of iterative writing practice, classroom video familiarity, automated structural reporting, and exploratory instructor feedback rather than an isolated causal effect of algorithmically generated guidance. Additionally, low support frequencies for specific subcategories, such as personal consequences, introduce classification uncertainty that future investigations could address through targeted data augmentation techniques. The researchers concluded that while supervised language models like BERT offer robust foundations for structural text analytics, further research is necessary to explore how automated analytical capabilities transfer into live instructional classrooms.

Human annotation established the empirical ground truth for evaluating model accuracy, with interrater reliability assessments yielding Cohen’s kappa values above 0.73 across all reflection elements in Context A and exceeding 0.95 in Context B. During dataset preparation in Context A, the researchers partitioned the text corpus into sixty percent training, twenty percent validation, and twenty percent test subsets through randomized shuffling. The authors noted that this procedure did not stratify data splits by individual student, which could slightly overestimate model classification performance. Model evaluation also had to account for pronounced label imbalances across categories, where the relative scarcity of consequence statements pulled the unweighted macro F1 score down to 0.63 in the combined training configuration despite the higher weighted overall average.

The curricular design strategically paired video vignette materials with targeted seminar topics to stimulate contextual reflection. While all five classroom recordings had been deployed in prior teacher education courses, only two originated from the initial Context A training set, compelling the transferred model to interpret reflections stemming from three unfamiliar classroom scenarios. For example, one vignette featured a teacher attempting to elicit student conceptions regarding optical phenomena, aligning directly with seminar instruction on cumulative learning and instructional intervention strategies. Participation across the semester reflected common longitudinal research patterns, with thirteen students completing the initial baseline reflection, twelve submitting the second and third assignments, nine completing the fourth, and eleven providing the fifth reflection.

Beyond automated structural metrics, the exploratory instructor guidance examined whether student evaluations were grounded in established educational literature. Human instructors assessed whether evaluative statements supplied explicit justifications informed by pedagogical knowledge, content knowledge, or pedagogical content knowledge, directing pre-service teachers toward relevant scholarly research when theoretical foundations were absent. Looking toward future educational technology implementations, the authors suggested that larger generative language models could be integrated into the pipeline to verbalize structural metrics into natural prose or propose relevant learning frameworks. However, because generative architectures do not automatically excel at specialized annotation tasks in science education, the researchers emphasized that automated systems require rigorous human supervision and collaborative cross-institutional validation.

Subject of Research: Automated natural language processing feedback for pre-service science teacher reflections

Article Title: Using a large language model to provide individualized feedback for pre-service physics teachers’ written reflections

Article References: Sorge, S., Wulff, P., & Kubsch, M. (2025). Using a large language model to provide individualized feedback for pre-service physics teachers’ written reflections. Disciplinary and Interdisciplinary Science Education Research, 7(1), Article 25. https://doi.org/10.1186/s43031-025-00145-9

Image Credits: AI Generated

DOI: 10.1186/s43031-025-00145-9

Keywords: Artificial Intelligence, Large Language Models, BERT Architecture, Physics Teacher Education, Pre-service Teachers, Written Reflections, Reflective Thinking, Automated Feedback, Formative Assessment, Science Education Research, Learning Analytics, Model Transferability, Sentence Classification, Cross-Institutional Evaluation, Teacher Professional Development, Educational Technology