DeepSeek AI with Guideline Knowledge Base Boosts Junior Doctors’ Arthroplasty Decisions

Artificial intelligence models built by the Chinese company DeepSeek can measurably improve the preoperative decision-making of junior orthopedic physicians, according to a retrospective study published in the journal Arthroplasty. The research, conducted at Qilu Hospital of Shandong University, tested four configurations of the DeepSeek large language model against 292 anonymized medical records of patients scheduled for hip or knee replacement, and found that a reasoning-enhanced version paired with a curated guideline knowledge base outperformed both simpler model versions and unassisted human judgment.

The study addressed a persistent problem in elective joint replacement: most patients are elderly and carry multiple chronic conditions such as cardiovascular disease, diabetes, and obesity, making preoperative assessment both vital for surgical safety and genuinely difficult. Senior surgeons rely on years of accumulated experience to weigh medical history, physical examination findings, laboratory results, and imaging data, but junior physicians often face information overload when synthesizing these factors for complex patients. The research team, led by Haibo Wu and colleagues with Qunshan Lu as corresponding author, set out to determine whether large language models could serve as a practical auxiliary tool in exactly these high-stakes, detail-heavy judgments.

Between January and August 2025, the researchers screened 323 hospitalized patients who met surgical indications for total hip or total knee arthroplasty. After excluding 31 cases with incomplete records, they included 292 fully de-identified patient records. From these they built a domain-specific knowledge base using 21 perioperative management guidelines drawn from PubMed and UpToDate, covering cardiopulmonary assessment, comorbidity management, thromboprophylaxis, and infection control. Six of the guidelines were arthroplasty-specific and fifteen were general perioperative documents. The guidelines were converted to plain text, split into semantic chunks, embedded into a local vector database, and retrieved through retrieval-augmented generation (RAG) technology, with the three most semantically similar chunks appended to each model prompt.

Four model configurations were compared: DeepSeek-Chat, DeepSeek-Chat plus the knowledge base, DeepSeek-R1, and DeepSeek-R1 plus the knowledge base. All models were invoked through the DeepSeek API with web search disabled, running on the open-source Dify platform with fixed inference parameters. Each model was asked to perform American Society of Anesthesiologists (ASA) scoring, revised cardiac risk index (RCRI) scoring and risk stratification, surgical tolerance judgment, and surgical risk prediction. Model outputs were compared against reference standards documented in the medical records, and a panel of five senior orthopedic surgeons evaluated the models’ technical performance in comprehension, retrieval, and reasoning while blinded to which model had produced each response.

The results revealed a clear architectural divide. In ASA scoring, none of the four models exceeded 60 percent exact-match accuracy, and the differences among them were not statistically significant, with DeepSeek-R1 plus the knowledge base achieving the best result at 58.6 percent. Notably, the researchers found that the reference standard itself was subjective: when two independent senior anesthesiologists re-scored a random subset of 50 patients, Fleiss’ kappa was 0.473, indicating only moderate agreement among the original anesthesiologist and the two reviewers. For RCRI scoring and risk stratification, however, the R1 series performed dramatically better, reaching approximately 79.4 percent scoring accuracy and 79.7 percent stratification accuracy, significantly surpassing the Chat series at 56.5 to 66.0 percent.

Surgical tolerance judgment showed a similar pattern. DeepSeek-R1 plus the knowledge base achieved the best performance, with 58.6 percent accuracy, a true positive rate of 59.1 percent, and a false negative rate of 40.9 percent that was significantly better than the Chat models at seizing surgical opportunities and reducing missed diagnoses. Yet the same model carried a false positive rate of 45.7 percent, and no significant differences emerged among the four models in specificity or false positive rate, meaning all remained at the same level in safely excluding high-risk surgical cases. The authors cautioned that the reference standard for tolerance was the actual surgery status recorded in the charts, an imperfect benchmark since postponements may occur for non-medical reasons that retrospective records cannot capture.

In surgical risk prediction the gap widened further. The DeepSeek-R1 plus knowledge base group and the basic DeepSeek-R1 group achieved average prediction accuracy rates of 87.2 percent and 86.4 percent respectively, while the Chat plus knowledge base and basic Chat groups managed only 63.6 percent and 61.8 percent. Both R1 models reached 100 percent accuracy in predicting infection and venous thromboembolism, and the knowledge base-enhanced R1 model exceeded 90 percent accuracy for myocardial infarction and poor wound healing. However, every model struggled with the residual category of other risks, with prediction accuracy not exceeding 35 percent in any group, a limitation the authors attributed to the low incidence, diverse presentation, and sparse guideline coverage of these less common complications.

The most consequential finding came from the human-machine collaboration experiment. Using a 5 by 5 balanced Latin square crossover design, the researchers divided the 292 records into five case sets and had five junior orthopedic physicians with less than three years of experience evaluate them both independently and with assistance from each of the four models. A generalized linear mixed model adjusting for physician variability and patient case mix showed that DeepSeek-R1 plus the knowledge base significantly improved junior physicians’ overall assessment accuracy compared with independent decision-making, with an odds ratio of 1.58 (95 percent confidence interval 1.16 to 2.15, p = 0.004), and it also outperformed the DeepSeek-R1, DeepSeek-Chat, and DeepSeek-Chat plus knowledge base conditions. No significant period effect was detected, ruling out learning or fatigue across the five assessment rounds. In surgical tolerance judgment specifically, junior physicians assisted by the model improved from 76.0 percent to 88.0 percent accuracy.

Senior surgeons’ qualitative evaluations reinforced the quantitative results. The R1 series scored significantly higher than the Chat series in comprehension, retrieval, and reasoning, and the knowledge base-enhanced R1 model achieved the best overall performance with 69.4 percent comprehension accuracy, 82.3 percent retrieval accuracy, and 66.3 percent reasoning accuracy, all significantly better than its base model. Adding the knowledge base to the Chat model, by contrast, produced no significant enhancement and even slightly reduced its ASA scoring accuracy, from 55.1 percent to 52.7 percent. The authors suggested that the Chat model tends to reproduce guideline content without deeply aligning it with the patient’s clinical data, so the additional retrieved text becomes noise, whereas the R1 model’s chain-of-thought reasoning actively verifies consistency between guideline provisions and the actual medical record. In the Likert-scale evaluation, significant differences among the models appeared only in the helpfulness and completeness dimensions, with the R1 models earning higher upper-quartile scores.

The researchers were emphatic that these models remain auxiliary tools rather than autonomous decision-makers. Even the best-performing configuration achieved only 58.6 percent accuracy in surgical tolerance judgment, and the authors warned that junior physicians are particularly vulnerable to automation bias, the tendency to over-rely on algorithmic outputs without sufficient independent verification. They recommended deployment exclusively within a human-in-the-loop framework in which a senior physician retains final authority over all surgical decisions and can override model recommendations, and they stressed that prospective validation with real-time safety monitoring is essential before any clinical translation. The study’s limitations include its single-center retrospective design, the imperfect reference standard for surgical tolerance, and a knowledge base restricted to perioperative guidelines without orthopedic textbooks or expert consensus statements. Within those bounds, the findings suggest that pairing reasoning-capable language models with curated guideline knowledge offers a cost-effective pathway toward better-supported preoperative assessment, provided human judgment remains the final safeguard.

Subject of Research: Evaluation of DeepSeek large language models with a retrieval-augmented guideline knowledge base for preoperative assessment in joint replacement surgery

Article Title: Evaluation of DeepSeek with a RAG-based knowledge base in preoperative assessment for arthroplasty to improve the decision-making accuracy of junior physicians

Article References: Wu, H., Hou, M., Chen, G., Wen, Z., Yang, C., Li, S., Meng, X., Liu, P., & Lu, Q. (2026). Evaluation of DeepSeek with a RAG-based knowledge base in preoperative assessment for arthroplasty to improve the decision-making accuracy of junior physicians. Arthroplasty, 8(1), Article 77. https://doi.org/10.1186/s42836-026-00437-z

Image Credits: AI Generated

DOI: 10.1186/s42836-026-00437-z

Keywords: DeepSeek, large language models, preoperative assessment, arthroplasty, retrieval-augmented generation, knowledge base, junior physicians, decision support, ASA score, RCRI, surgical risk prediction, joint replacement