Language Models Demonstrate Mixed Precision in Evaluating Science Teacher Pedagogical Knowledge

A proof-of-concept investigation published in Disciplinary and Interdisciplinary Science Education Research has evaluated the capabilities of large language models to assess science teachers’ pedagogical content knowledge on the topic of photosynthesis using both classroom lesson plans and written responses to open-ended pedagogical questions. Conducted by researchers including Arif Rachmatullah, Shaishav Tayde, Nonye Alozie, Soonhye Park, and Selin Capan, the study examined whether automated natural language processing architectures could reliably identify nuanced pedagogical reasoning that human experts traditionally evaluate through intensive qualitative coding. Pedagogical content knowledge represents a specialized synthesis of subject matter expertise and pedagogical practice, enabling educators to anticipate student learning hurdles, interpret misconceptions, and formulate responsive instructional interventions. The research team sought to determine whether artificial intelligence tools could alleviate the non-standardized, labor-intensive, and time-consuming bottlenecks that historically have hindered large-scale empirical research into teachers’ pedagogical reasoning and classroom practice.

The study grounded its evaluation framework in the pentagon model of science teacher pedagogical content knowledge, an empirically validated theoretical structure originally conceptualized by Park and Oliver that models five dynamically interacting knowledge domains rather than isolated instructional skills. The authors focused automated evaluation on four specific components: knowledge of student understanding, knowledge of instructional strategies and representations, knowledge of science curricula, and knowledge of assessment of science learning. The fifth component of the pentagon model, orientations toward teaching science, encompasses broad epistemological beliefs about the purposes of science education that are difficult to infer reliably from concise written assignments or instructional outlines. Consequently, the researchers measured orientations toward teaching science independently using a specialized forty-four-item survey instrument known as Epistemic Orientations toward Teaching Science for Knowledge Generation, which evaluates epistemic alignment, authority relations in learning, views on the nature of knowledge, and beliefs regarding student ability.

To assess instructional artifacts, the investigators collected fifty-three complete, downloadable lesson plans centered on photosynthesis intended for third through twelfth-grade classrooms across open-access educational repositories and teacher community platforms. These instructional materials were evaluated against validated rubrics originally established by Großmann and Krüger, which parse pedagogical content knowledge into twenty-four distinct subcomponents rated on a three-point ordinal scale of not present, sufficient, and excellent. Because five rubrics tailored to pre-service teachers did not suit lesson plans authored by practicing in-service teachers, the researchers concentrated their proof-of-concept automated scoring pipeline on nine essential subcomponents. Two human researchers independently scored all fifty-three lesson plans using the refined rubric, achieving initial reliability that was calculated using Gwet’s AC1 statistic with quadratic weighting, after which a third researcher joined to deliberate discrepancies until full interpretive consensus was established.

Automated lesson plan scoring began with zero-shot prompting using an Explanation-Prediction framework, before transitioning to a few-shot learning paradigm that integrated chain-of-thought prompting with hierarchical decision heuristics to clarify ranking thresholds. When tested against human consensus labels using Gwet’s AC1 with quadratic weighting, the automated scoring models attained strong human-machine reliability across several dimensions, frequently exceeding the baseline human-human interrater agreement. In particular, automated scoring models achieved agreement exceeding 0.90 for subcomponents evaluating knowledge of student understanding, such as anticipating learning difficulties and identifying methodological skills students must acquire. The models similarly surpassed 0.90 reliability when evaluating assessment subcomponents, accurately identifying transparent learning expectations and evidence of student learning. However, the automated models struggled with curricular justification subcomponents and encountered significant difficulty evaluating whether instructional activities progressively developed student competencies through coherent pedagogical transitions.

Evaluating pedagogical knowledge through open-ended written prompts presented distinct recruitment and data collection challenges that shaped the investigation’s methodological pipeline. The researchers adapted an established assessment battery that presented text-based classroom scenarios involving student misconceptions about photosynthesis, requiring teachers to explain student reasoning, propose instructional remediations, and design assessments. When an initial six-scenario instrument was distributed to approximately 5,000 science educators across the United States, demanding cognitive requirements yielded only eighteen completed submissions. In response, the researchers streamlined the protocol to a single scenario with six targeted questions and adopted synthetic data generation procedures aligned with guidance from the United Kingdom Department for Education. Using prompts tailored to varied teaching backgrounds and experience levels, the team generated 188 synthetic datasets comprising 1,128 distinct written responses using language models including Llama, Mistral, and Zephyr.

After expert human coders scored the synthetic responses using the rubric, the researchers fine-tuned and validated four distinct transformer-based bidirectional encoder architectures—RoBERTa, BERTxnli, EduBERT, and FineWeb—using an eighty-twenty training and validation split. On the synthetic evaluation set, model performance varied noticeably across specific pedagogical dimensions and model architectures. EduBERT demonstrated the highest human-machine reliability on the initial question assessing conceptual challenges, closely matching human interrater agreement, whereas BERTxnli performed poorly on that prompt. On the second question regarding reasons behind misconceptions, FineWeb achieved the strongest performance with reliability exceeding 0.70. FineWeb likewise demonstrated superior performance on pedagogical remediation prompts, outperforming human rater agreement with reliability above 0.80, while EduBERT achieved 0.927 human-machine reliability on instructional justification prompts, outperforming human rater reliability of 0.919 despite an observed skew toward lower rubric scores in the synthetic training corpus.

The critical empirical test occurred during phase two, when the models trained solely on synthetic responses were evaluated against authentic written submissions from 115 practicing science teachers across the United States. Across questions measuring assessment design and pedagogical justification, the models sustained high reliability, registering human-machine agreement values hovering near or above 0.90. However, all models exhibited marked performance degradation when evaluating authentic responses concerning student understanding. Reliability coefficients on these diagnostic prompts dropped from synthetic-baseline levels near 0.70 down to values between 0.50 and 0.60, with EduBERT and BERTxnli registering reliability values of only 0.302 and 0.268, respectively, on the initial question. Across the comprehensive assessment, FineWeb emerged as the most resilient and stable architecture, maintaining more consistent performance across contrasting pedagogical sub-constructs than its domain-adapted or multilingual counterparts.

To determine why models struggled when encountering authentic teacher writing, researchers conducted a qualitative thematic analysis on discrepancies where human-machine agreement fell below 0.70. For diagnostic questions addressing student misconceptions, three primary failure modes emerged across the automated systems. First, models frequently awarded higher score levels for vague scientific terms that lacked meaningful pedagogical explanations, erroneously counting broad phrases such as the role of sunlight as defined conceptual challenges, while struggling to separate related physics concepts like light and heat. Second, models exhibited sensitivity to phrasing redundancy, sometimes double-counting a single misconception that was restated across sentences or undercounting distinct misconceptions presented within similar contexts. Third, the systems demonstrated brittle vocabulary recognition, consistently overlooking valid reasoning when teachers utilized non-standard phrasing such as nutrients or food molecules in place of standard terms like glucose.

Qualitative inspection of teacher-suggested instructional strategies revealed parallel linguistic fragilities in automated scoring behavior. When evaluating proposed pedagogical remediations, models routinely assigned lower ratings to ungrammatical or incomplete sentences even when the underlying instructional rationale was pedagogically sound, while rewarding well-structured, grammatical text that contained vague or generic educational advice. Furthermore, models struggled to identify hands-on instructional practices unless teachers explicitly utilized identifying labels like experiment, demonstration, or activity. On items requiring teachers to justify their pedagogical choices, automated models exhibited a persistent architectural bias: they frequently failed to comprehend that a single unified instructional intervention could simultaneously remediate multiple student misconceptions. Consequently, the algorithms incorrectly coded multi-target lessons as distinct, independent teaching strategies, assigning elevated rubric scores that contradicted established coding guidelines.

In the final phase of the investigation, researchers conducted Pearson and Spearman correlation analyses to evaluate whether machine-derived scores mirrored the empirical relationships observed between human-rated pedagogical knowledge and teacher demographic characteristics. The sample of 115 science teachers reported an average of 15.46 years of classroom experience, with fifteen percent holding National Board Certification, and comprised sixty-one percent female educators, predominantly teaching at the high school level. Human evaluations established that teacher knowledge of student understanding correlated positively and significantly with orientations toward teaching science measured via the epistemic survey, as well as with total years of classroom teaching experience. Human-evaluated instructional strategy scores similarly demonstrated statistically significant positive correlations with teaching experience and National Board Certification, while self-efficacy for teaching photosynthesis showed no statistically significant relationship with any pedagogical content knowledge component.

Comparative correlation analyses revealed that FineWeb closely replicated the statistical relationships established by human coders, generating significant positive associations between knowledge of student understanding, epistemic teaching orientations, and classroom experience. FineWeb likewise matched human benchmarks in correlating instructional strategy scores with experience and National Board Certification. Conversely, models like EduBERT and RoBERTa showed weaker, less consistent alignments. Furthermore, none of the four language models successfully replicated the human finding that assessment knowledge correlated significantly with epistemic orientations toward science teaching, with all models yielding non-significant coefficients. The researchers observed that while individual models excelled in isolated sub-domains, no single automated system accurately mirrored human ratings across the entire construct, indicating that solitary model deployments remain insufficient for comprehensive teacher evaluation.

The study authors concluded that while language models show practical promise for evaluating explicit, highly structured criteria in curriculum design and formative assessment planning, significant boundaries remain in automated pedagogical reasoning. The observed vulnerabilities regarding vocabulary sensitivity, sentence structure bias, and inability to trace overarching instructional coherence across lesson phases highlight critical validity and fairness concerns. Because language models tend to exploit superficial linguistic regularities rather than tracking pedagogical substance, uncalibrated automated tools risk distorting evaluations if deployed in high-stakes professional contexts without human oversight. The investigators recommended developing hybrid human-in-the-loop workflows and multi-model ensemble architectures, combining techniques like weighted voting or stacking with broader multimodal datasets, to capture the multifaceted, interconnected nature of science teachers’ pedagogical knowledge responsibly.

Subject of Research: Evaluation of science teachers' pedagogical content knowledge using large language models

Article Title: Exploring large language model’s capabilities in identifying science teacher PCK using lesson plans and open-ended questions

Article References: Rachmatullah, A., Tayde, S., Alozie, N., Park, S., & Capan, S. (2026). Exploring large language model’s capabilities in identifying science teacher PCK using lesson plans and open-ended questions. Disciplinary and Interdisciplinary Science Education Research, 8(1), Article 3. https://doi.org/10.1186/s43031-025-00151-x

Image Credits: AI Generated

DOI: 10.1186/s43031-025-00151-x

Keywords: pedagogical content knowledge, large language models, science education, automated assessment, educational artificial intelligence, lesson plan evaluation, natural language processing, teacher professional development, Exploring, large, language, model