In an exploratory pilot evaluation, researchers found that a locally deployed large language model can reduce the time required to extract implementation determinants from published research by approximately 75 to 95 percent. The investigation, which examined twelve purposively selected published studies, demonstrated that automated processing shortened the time required to code an individual article from several hours down to under twenty minutes. Across the sample, the artificial intelligence system identified implementation barriers and facilitators aligned with the Consolidated Framework for Implementation Research, though the degree of semantic correspondence between human and automated coding depended substantially on the specific prompting approach applied and the structure of the underlying scientific literature.
The research addresses an enduring challenge within implementation science, a discipline dedicated to evaluating targeted strategies designed to promote the systematic adoption, integration, and sustainment of evidence-based clinical interventions, programs, and policies in real-world healthcare settings. Although ongoing scientific advancements consistently produce clinical interventions intended to optimize disease prevention, diagnostics, and patient therapies, persistent gaps separate published clinical evidence from routine healthcare practice. Reviewing the academic literature to identify and code the multifaceted determinants associated with implementation success represents an essential yet notoriously labor-intensive and time-consuming undertaking. To determine whether modern computational models could feasibly scale and accelerate this workflow, the study team assessed whether open-source generative language models could accurately extract relevant implementation determinants within the field of oncology.
To evaluate feasibility, the authors configured a locally deployed, Llama-based large language model and applied an ontology-based schema to structure all extracted information. The analysis drew upon a purposive sample of twelve published papers divided into two distinct literature categories: oncology-focused implementation studies containing structured implementation data, and published clinical intervention studies containing unstructured implementation details. Across these diverse publications, the investigative team tasked the model with isolating and classifying implementation determinants into barriers and facilitators aligned with the established framework. The procedural framework was designed to examine how different textual formats within published biomedical literature influence the reliability and semantic fidelity of automated determinant extraction.
The experimental protocol systematically compared three distinct prompting techniques to establish which instructional format yielded the highest extraction fidelity across different article types. The team evaluated zero-shot prompting, which provided the model solely with direct task instructions, alongside few-shot prompting, which supplemented the foundational task instructions with illustrative extraction examples. Additionally, the investigators tested chain-of-thought prompting, a strategy that explicitly directs the computational model to generate step-by-step intermediate reasoning before producing its final extraction outputs. By testing these three prompting configurations across both structured implementation science reports and unstructured clinical intervention studies, the researchers generated a cumulative total of 144 discrete extraction outputs for comprehensive comparative evaluation.
To quantify extraction accuracy, the researchers measured semantic similarity between the model-generated outputs and human-coded reference standards using two parallel assessment techniques: human evaluator scoring and automated semantic rating via a sentence transformer model. In oncology-focused implementation science papers containing structured data, zero-shot prompting achieved the highest average semantic similarity scores for identifying both barriers and facilitators, outperforming both few-shot examples and chain-of-thought reasoning. Conversely, when processing clinical intervention papers with unstructured implementation information, few-shot prompting delivered superior performance over zero-shot and chain-of-thought strategies. Under few-shot prompting in clinical intervention studies, automated ratings with the sentence transformer yielded average similarity scores of 0.78 for barriers and 0.76 for facilitators, whereas human evaluators assigned average similarity scores of 0.68 for barriers and 0.67 for facilitators.
Across all evaluated prompting approaches and document types, human ratings of semantic similarity remained slightly lower than the automated scores produced by the sentence transformer model. The authors noted that the underlying reasons for this divergence between human judgment and automated rating models were not directly examined in this pilot investigation. Despite this discrepancy, the operational efficiency gains achieved by the locally deployed model were substantial across the board, consistently compressing the multi-hour manual review pipeline into an automated process requiring less than twenty minutes per study. This steep reduction in manual annotation time demonstrates the practical potential of generative models to support evidence synthesis workflows while maintaining recognizable semantic alignment with expert human coding.
Despite these promising operational gains, the investigators emphasized that their findings represent preliminary feasibility evidence rather than definitive validation of extraction accuracy. Because the pilot investigation relied on a small, purposively selected corpus of twelve studies, the external validity and generalizability of the findings remain inherently limited. The authors stressed that the relative efficacy of specific prompting strategies appears sensitive to the structural nature of the source literature being analyzed. Consequently, the research team concluded that substantially larger, systematically sampled datasets featuring independent manual annotations and expert adjudication will be necessary before definitive claims can be established regarding the clinical accuracy, reliability, and broader utility of large language models for evidence synthesis in implementation science.
Subject of Research: Automated extraction of implementation science determinants using large language models
Article Title: Prompt-driven large language model for evidence extraction in implementation science: exploring semantic similarity between manual and automated coding across studies
Article References: Zhang, C., Chan, J. C., Hastings, J., Lin, F., Presseau, J., Delaney, G. P., Michie, S., Fontaine, G., & Taylor, N. (2026). Prompt-driven large language model for evidence extraction in implementation science: exploring semantic similarity between manual and automated coding across studies. Implementation Science Communications. https://doi.org/10.1186/s43058-026-01144-1
Image Credits: AI Generated
DOI: 10.1186/s43058-026-01144-1
Keywords: implementation science, large language models, evidence extraction, semantic similarity, CFIR framework, zero-shot prompting, few-shot prompting, chain-of-thought prompting, sentence transformer, oncology implementation, clinical intervention studies, barriers and facilitators, automated coding, annotation time reduction, ontology-based schema, pilot study, Llama model
Cite Scienmag News
Nathaniel Bowman. (November 18, 2026). Locally Deployed Language Model Cuts Implementation Coding Time in Exploratory Pilot Study. Scienmag. https://scienmag.com/locally-deployed-language-model-cuts-implementation-coding-time-in-exploratory-pilot-study/
Nathaniel Bowman. "Locally Deployed Language Model Cuts Implementation Coding Time in Exploratory Pilot Study." Scienmag, 18 November 2026, https://scienmag.com/locally-deployed-language-model-cuts-implementation-coding-time-in-exploratory-pilot-study/. Accessed 18 November 2026.
Nathaniel Bowman. "Locally Deployed Language Model Cuts Implementation Coding Time in Exploratory Pilot Study." Scienmag. November 18, 2026. https://scienmag.com/locally-deployed-language-model-cuts-implementation-coding-time-in-exploratory-pilot-study/








