AI Pipeline Turns Messy Clinical Notes Into Research-Ready Data With Near-Perfect Accuracy

Electronic medical records are often described as gold mines of clinical information, but most of that treasure is buried. While laboratory values, vital signs, and medication orders arrive neatly coded, the richest details of a patient’s story—the surgical nuances, the procedural variations, the clinical judgment calls—live inside free-text notes written by busy clinicians. For researchers and quality-improvement teams, extracting that information has long meant hours of painstaking manual chart review, a process that is slow, expensive, and nearly impossible to scale. Now, a team at Johns Hopkins All Children’s Hospital has shown that a carefully engineered large language model pipeline can do much of that work automatically, with accuracy so high that it approaches the ceiling of human agreement.

The system, called CLASS—short for Clinical LLM Abstraction & Structuring System—is described in a new feasibility report published in the Journal of Medical Systems. Led by anesthesiologist and informatics researcher Frederick H. Kuo, the team built CLASS as a modular, Python-based pipeline that runs entirely within a secure institutional computing environment, keeping protected health information behind the hospital’s own walls rather than sending it to external services. That design choice reflects a growing consensus in medical informatics: the power of large language models can be harnessed for clinical data without compromising patient privacy, provided the infrastructure is configured correctly.

Technically, CLASS rests on three pillars. The first is a set of concept lists curated by subject matter experts—in this case, surgeons and informatics specialists who defined exactly which procedures and clinical details the system should look for. The second is a task-specific prompt suite, a collection of carefully worded instructions that steers the large language model toward consistent, clinically grounded interpretations of each note. The third is a schema-constrained output format, which forces the model to return its findings in a structured, predictable structure rather than free-flowing prose. The results are then exported to an interactive dashboard built for expert review, allowing clinicians to verify outputs, spot errors, and analyze the extracted data at scale.

One of the most innovative features of CLASS is its handling of the unknown. Rather than simply classifying notes against a fixed list of predefined concepts, the pipeline actively flags potential variants or entirely novel concepts that do not fit the existing schema, surfacing them for expert consideration. In specialized fields where terminology evolves quickly and procedures are often described in non-standard ways, this ability to propose expansions to the concept vocabulary could fundamentally change how clinical registries and research databases are built and maintained.

To test the system, the researchers applied CLASS to a retrospective corpus of pediatric esophageal airway treatment surgery, or EATS, operative notes at their single center. EATS is a demanding test case: these operative notes describe complex, highly individualized procedures in children with airway and esophageal abnormalities, and much of the procedural detail is not captured in standard billing codes. The team compared CLASS outputs against adjudication by an experienced surgeon on the twenty longest notes in the corpus, yielding 3,960 individual note-procedure pairs for evaluation.

The results were striking. Across those thousands of judgments, the observed concordance between the automated pipeline and the surgeon’s adjudication reached an F1 score of 0.9967—a near-perfect measure of precision and recall combined. In practical terms, the model almost never missed a procedure the surgeon identified, and almost never invented one that was not there. For a task as subtle as parsing operative prose about pediatric airway surgery, that level of agreement suggests that large language models, when properly constrained and prompted, can match expert-level abstraction performance in at least some specialized clinical domains.

The novelty-detection results were more nuanced, and arguably more interesting. CLASS proposed 28 candidate procedure variants or additions to the curated concept list, and the clinical team judged 18 of them—64.3 percent—to be genuinely useful. That is a meaningful yield: nearly two out of every three suggestions from the machine were worth a clinician’s time. At the same time, the surgeon identified 12 additional procedures that CLASS failed to surface, a reminder that the system works best as a collaborator rather than a replacement. The human expert still caught things the machine missed, and the machine still surfaced things the human might not have thought to codify.

The authors are careful to frame the study appropriately. This was an exploratory implementation at a single center, focused on a single surgical service, with evaluation limited to a small set of long notes. Generalizability to other institutions, other note types, and other clinical tasks remains unproven, and the team emphasizes that broader validation is needed before such pipelines could be trusted for high-stakes applications. The full production code and clinical data cannot be released publicly because they involve protected health information and institution-specific infrastructure, but the researchers have shared technical implementation details, template code, pseudocode, and de-identified prompt examples in the online supplementary materials, giving other informatics teams a practical roadmap for building similar systems.

Even with those caveats, the implications are considerable. Clinical research has long been throttled by the bottleneck of manual abstraction: cohort studies that could enroll thousands of patients are often limited to hundreds simply because there are only so many hours in a research coordinator’s day. Quality-improvement programs face the same constraint, unable to measure surgical outcomes comprehensively when the relevant data must be pulled by hand from narrative notes. If pipelines like CLASS can reliably convert unstructured text into analyzable data within standard institutional infrastructure, the effective sample sizes of clinical research could expand dramatically, and hospitals could monitor the quality of specialized care in near real time.

The study also highlights a shift in how medical informatics teams may work in the coming years. Instead of writing brittle rule-based extraction algorithms or training bespoke machine-learning models on small labeled datasets, teams can now curate expert concept lists, design prompts, and review machine-generated suggestions—a workflow in which clinicians define what matters and the model handles the linguistic heavy lifting. The CLASS experience suggests this human-machine partnership can work: the model performs the extraction with near-perfect fidelity, proposes useful vocabulary expansions, and leaves final judgment to the experts who bear clinical responsibility. As large language models continue to demonstrate their grasp of medical language, studies like this one offer a concrete, privacy-conscious template for turning the narrative richness of the medical record into structured knowledge—without a single chart being pulled by hand.

Subject of Research: A large language model pipeline for extracting structured data from unstructured clinical notes

Article Title: Exploratory Implementation and Feasibility Report of CLASS (Clinical LLM Abstraction & Structuring System), A Large Language Model Pipeline for Extracting Unstructured Data From Clinical Notes

Article References: Exploratory Implementation and Feasibility Report of CLASS (Clinical LLM Abstraction & Structuring System), A Large Language Model Pipeline for Extracting Unstructured Data From Clinical Notes. (n.d.). https://doi.org/10.1007/s10916-026-02462-6

Image Credits: AI Generated

DOI: 10.1007/s10916-026-02462-6

Keywords: large language models, clinical notes, natural language processing, electronic health records, medical informatics, data extraction, pediatric surgery, artificial intelligence, CLASS pipeline, clinical research, esophageal airway treatment surgery, Johns Hopkins