The study’s design was deliberately rigorous. Rather than asking the models to find important papers on their own, the investigators asked attending surgeons from four subspecialties, emergency general surgery, trauma surgery, surgical critical care and pediatric surgery, to nominate the studies they considered essential for resident education. This expert-driven approach produced a reference set of 26 landmark studies, including six in emergency general surgery, five in trauma surgery, six in surgical critical care and nine in pediatric surgery. Because the goal was to measure how faithfully a model could condense a known paper, not how well it could identify influential literature, curating the list by hand was a critical methodological choice. It meant that every summary could later be checked line by line against a definitive source document chosen by people who teach these papers for a living.

Both models were then given the same standardized task. Each study was uploaded as a PDF in a fresh browser session to eliminate contamination from prior conversation history, and the models received a prompt engineered to simulate expert teaching: they were instructed to act as a subspecialty surgeon teaching residents, to produce a one-page summary covering background, study design, inclusion and exclusion criteria, treatment groups, key findings and clinical applications, and to briefly describe how the data had changed clinical practice since publication. All of the selected studies were publicly available, so no proprietary or restricted content entered the models. The standardized prompt matters more than it might appear, because prompt wording is known to shape the quality and emphasis of large language model output, and a single consistent prompt allowed fair head-to-head comparison between the two systems.

To score the results, the researchers borrowed evaluation machinery from machine learning itself. Each summary was independently assessed by two faculty reviewers from the relevant subspecialty, both blinded to the fact that the summaries were AI-generated and told to evaluate them as if residents had prepared them for educational purposes. Documents were randomly labeled to prevent bias based on perceived source quality. Reviewers classified each of six domains, background, methods, inclusion and exclusion criteria, treatment groups, key findings and clinical relevance, as a true positive, a false positive or a false negative. A true positive meant the AI content was accurate and present in the original paper; a false positive meant the summary contained information absent from the source, the phenomenon better known as a hallucination; a false negative meant the original paper contained essential information the model failed to include. From these labels the team calculated precision, recall and the composite F1 score for each model and domain.

The headline numbers were strikingly strong. ChatGPT-4.0 achieved a composite precision of 0.93, a recall of 0.96 and an F1 score of 0.95. Perplexity Pro performed comparably, with a precision of 0.96, a recall of 0.94 and an F1 score of 0.95. In the machine learning literature, F1 scores above 0.70 are generally considered acceptable and scores above 0.90 excellent, though appropriate thresholds depend on task complexity. By that yardstick, both models delivered excellent performance on a task that demands deep domain knowledge. A Mann-Whitney U test found no statistically significant differences between the two systems on any composite metric, suggesting that the choice between a general-purpose assistant and a web-connected AI search engine mattered less than one might expect for this constrained summarization task.

The weak spot, when the scores were broken down by domain, was instructive. Both models performed best on background, treatment groups, key findings and clinical relevance, but stumbled on inclusion and exclusion criteria, the fine-grained eligibility rules that determine exactly which patients a study applies to. ChatGPT-4.0 posted a precision of 0.93, a recall of 0.84 and an F1 of 0.89 in that domain, while Perplexity Pro achieved a perfect precision of 1.00 but a recall of just 0.81, for an F1 of 0.90. Crucially, the shortfall was driven by omission rather than invention: the models tended to drop detailed eligibility information rather than fabricate it. This reflects a known tendency of large language models to prioritize information they perceive as most salient, retaining study objectives and headline findings while discarding granular methodological detail, a pattern that prior research has linked to overgeneralization of scientific findings in AI summaries.

Qualitative reviewer comments reinforced the quantitative picture. Most errors were omissions, typically involving incomplete reporting of inclusion and exclusion criteria, secondary outcomes or detailed methodological descriptions. True hallucinations were uncommon and most often took the form of overstated clinical implications, with models extrapolating broader practice changes than the original manuscripts supported. Importantly, reviewers rarely identified hallucinations that would meaningfully alter the interpretation of a study’s primary findings, and they frequently noted that the summaries captured the main results accurately while providing concise, readable overviews. Interrater agreement across all reviewer pairs was 84 percent overall, ranging from 77 percent in pediatric surgery to 91 percent in surgical critical care, indicating that the expert scoring itself was reasonably consistent even if percent agreement does not account for chance concordance.

The educational verdict came from the residents themselves. A voluntary survey distributed to 62 general surgery residents across all postgraduate years drew 15 responses, a 24 percent response rate. The results were emphatic: every respondent rated the summaries as clear or very clear, all but one said the summaries captured key findings well or very well, 80 percent judged them accurate or very accurate, and all rated them helpful or very helpful for understanding core concepts. Every respondent agreed the summaries would be a beneficial supplement to regular study materials, and all recommended continued use of LLM-generated summaries. Thirteen of fifteen felt the one-page format struck the right balance between brevity and comprehensiveness. Trust, however, remained calibrated: only a third of residents said they would trust the summaries without additional verification, while the rest wanted review by faculty, by other residents or by themselves against the original papers, a pattern the authors read as trainees valuing the tool while still recognizing the importance of consulting primary literature.

The study is the first, according to its authors, to validate large language models for summarizing key surgical literature, and it extends a body of work showing that models like ChatGPT can pass standardized medical exams, generate realistic clinical vignettes and deliver feedback to trainees. The authors are candid about limitations. Summaries were generated with a single standardized prompt, leaving open whether prompt optimization could improve performance; only four subspecialties were included; accuracy was scored at the section level rather than claim by claim, which could underestimate performance; and rapidly evolving model outputs may not be reproducible over time. They also point to retrieval augmented generation as a promising fix, citing work in which a modified GPT-4 answered clinical guideline questions correctly 84 percent of the time, up from 57 percent for the base model. For now, the message is one of cautious enthusiasm: AI-generated summaries of landmark surgical studies are accurate enough to earn a place in resident education, provided that human oversight, and the original papers, remain close at hand.

Subject of Research: Evaluating large language models for generating accurate summaries of landmark surgical studies for resident education

Article Title: Harnessing large language models to summarize landmark surgical studies for resident education

Article References: Pettigrew, M. F., Tyler, L. A., Gregory, A. L., Nomellini, V., Bhat, S. G., Murphy, J. T., Purcell, L. N., Butler, D., Park, C., Clark, A., & Abdelfattah, K. R. (2026). Harnessing large language models to summarize landmark surgical studies for resident education. Global Surgical Education – Journal of the Association for Surgical Education, 5(1), Article 135. https://doi.org/10.1007/s44186-026-00537-z

Image Credits: AI Generated

DOI: 10.1007/s44186-026-00537-z

Keywords: artificial intelligence, large language models, ChatGPT, Perplexity Pro, surgical education, resident training, medical literature summarization, hallucination, precision and recall, F1 score, graduate medical education, evidence-based surgery

The problem, the researchers argue, is that most existing similarity and difference analysis techniques rely on relatively single analysis criteria. Traditional methods typically consider only the dependency relations between activities in a process model. That works reasonably well for general business workflows, but manufacturing processes have a distinctive feature that such approaches ignore: the execution of every operation is accompanied by the consumption of a series of components and changes in their states. Two processes may look behaviorally identical on paper while consuming entirely different parts, or use identical parts while ordering their operations in critically different ways. When screening large process libraries for similar models, these blind spots cause numerous processes to be mistakenly flagged as highly similar, inflating review workloads and obscuring the real differences that matter for quality control.

To address this, the team first reformulated manufacturing processes as Petri net models, a formalism in which places represent products such as raw materials, items in progress or finished goods, transitions represent operations, and arcs capture the flow between them. Crucially, the model includes a function that assigns sets of components to each operation. Building on this representation, the authors introduce the concept of an operation profile, which characterizes how pairs of operations relate when a process executes. Five relations are distinguished: a strict relation in which one operation always precedes another, an interleaving relation that splits into circulation and parallel subtypes depending on whether the operations loop back to each other, and an exclusive relation in which two operations can never occur in the same execution.

A motivating example in the paper shows why a single criterion fails. When only operation profiles are compared, two processes with identical ordering relations score a similarity of exactly 1.0, implying equivalence, even though the components attached to their operations are not the same. Conversely, two processes sharing the same operation items but differing in whether two operations run strictly in sequence or in parallel are also hard to separate under a parts-only view. The lesson is that neither operation items nor operation profiles alone can deliver reliable judgments; both must be weighed together.

From this insight the researchers derive three similarity measurement methods. The operation items, or OI-based, method scores the similarity of corresponding operations by the Jaccard coefficient of their component sets, capturing how much the parts consumed by matching operations overlap. The operation profile, or OP-based, method instead counts how many operation pairs share the same occurrence relation, normalized by the number of possible pairs. The centerpiece, however, is the operation-aware profile, or COP-based, method, which fuses the two perspectives: for every operation pair it multiplies the pair’s component similarity by a binary indicator of whether the occurrence relation matches, then aggregates across the whole process. In a worked example, two processes that scored a perfect 1.0 under profile-only analysis drop to 0.87 once component differences are counted, and processes differing in both dimensions fall to 0.84, exactly the kind of downward correction that prevents false positives in process screening.

The same machinery extends to difference analysis. Given a newly created process, the method first retrieves all processes in the library whose COP similarity exceeds a threshold, set at 0.8 in the study. The operation profile matrices of these similar variants are aggregated into a single weighted matrix whose entries record the frequency of each occurrence relation across the variant set. Two quantitative indicators then drive an iterative clustering procedure. A separation value, computed with a weighted cosine measure, identifies which operations are most suitable to merge together, while a deviation value, measured against benchmark vectors for each relation type, determines which occurrence relation best characterizes the merged pair. Because different relations carry different constraint strength, from the weakly constrained interleaving relation to the strongly constrained exclusive relation, the weights are calibrated by combining domain rules with data-driven optimization rather than manual assignment. The result is a candidate operation profile matrix that summarizes the typical behavior of the whole variant family, against which the new process is compared entry by entry to expose every structural difference.

To test the approach, the team assembled a dataset of low-cost rapid manufacturing temperature sensors, establishing 12 reference processes with 4 variants each, drawn largely from the published literature and converted into Petri net form. A benchmark for true similarity was set with the help of eight experienced modelers, and the methods were evaluated on precision, recall, F1-score and accuracy. The COP method achieved a perfect precision of 1.0 in every case where similar variants existed, correctly rejecting dissimilar processes that the single-criterion methods accepted. The OP method, by contrast, reached recall of 1.0 in most cases, but its broad matching strategy came at the cost of substantial noise, with accuracy in some cases falling as low as 0.25. The COP method attained an average accuracy of 0.83 while the older OTV approach, which considers only part of the profile information, trailed behind in precision.

Statistical validation reinforced the picture. Paired t-tests at a significance level of 0.05 showed that the COP method’s improvements over the OI and OP methods were significant across both precision and accuracy, with all p-values below 0.05. Against the advanced OTV method, COP retained a statistically significant advantage in precision, while the difference in accuracy was not significant, with a p-value of approximately 0.1039, indicating at least comparable overall performance. The authors note that COP nonetheless offers greater theoretical completeness and interpretability, since it fully integrates both operation-component relations and the complete set of occurrence relations rather than a subset.

Perhaps the most practically significant finding concerns the method’s strictness toward small deviations. Rapid manufacturing processes are sensitive to environmental factors, and minor non-standard changes in operation sequence or execution logic frequently arise in production. The t-test results confirmed that when such substandard structural changes occur, the COP method assigns markedly lower similarity scores, flagging potential quality risks that looser methods gloss over. The authors therefore position the OI method for production resource arrangement and equipment scheduling, the OP method for macroscopic process planning and template construction, and the COP method for compliance verification and production quality control, where catching hidden deviations matters most.

In terms of difference mining, simulation experiments on grouped process models demonstrated that the iterative aggregation and clustering procedure correctly recovers candidate operation profiles, even when one process variant dominates the population with a 70 percent share, and that comparing candidate operation items against a new process pinpoints exactly which components deviate from established practice. The work was supported by the National Natural Science Foundation of China and several Anhui provincial programs. The authors acknowledge that, constrained by the availability of real industrial data and privacy limits, the evaluation relied on benchmark datasets from the literature; future work will apply the model to actual production logs with noise in real manufacturing workshops. If those trials succeed, factories could gain a rigorous, quantitative tool for managing sprawling process libraries, promoting reuse, and standardizing production on a scale that manual review has never been able to match.

Subject of Research: A Petri net-based method for measuring similarity and detecting differences between manufacturing processes by jointly analyzing operation profiles and operation-component relations.

Article Title: Analysis of similarity and difference in manufacturing processes based on petri nets

Article References: Analysis of similarity and difference in manufacturing processes based on petri nets. (n.d.). https://doi.org/10.1007/s10586-026-06494-y

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06494-y

Keywords: Petri nets, manufacturing processes, similarity measurement, operation profile, process mining, difference analysis, behavioral profiles, process optimization, Cluster Computing, t-test, rapid manufacturing, precision and recall