Model Comparison and Methodology

The experimental framework compared several models, including a base Llama-3-8B model from Meta, which was subjected to domain adaptation using a mixed corpus of general, medical, and radiology text before undergoing fine-tuning using both small (n = 1,280) and large (n = 499,814) sets of report-impression pairs (designated DA + FTs and DA + FTl). Similarly, an OpenBio-8B model was tested with identical fine-tuning regimens (Bio + FTs and Bio + FTl). Furthermore, the commercial model Gemini-1.5-Flash-002 was included in the comparison used out-of-the-box, providing a benchmark for commercially available, pre-trained capabilities. The clinical assessment was conducted independently by four radiologists, who rated the generated radiology impressions using six distinct 5-point Likert items, ensuring a multi-reader evaluation approach to capture clinical consensus and variance.

Performance in Data-Scarce Regimes

The results indicated a clear advantage for domain adaptation when labeled clinical data were scarce. Specifically, the domain-adapted model (DA + FTs) achieved a mean score of 24.3 ± 2.8, significantly outperforming the fine-tuned open-source model (Bio + FTs) with a mean score of 22.8 ± 3.5 (P = 0.0004). This finding suggests that domain adaptation serves as a highly effective initial step for institutions operating with limited access to large, curated, paired report-impression datasets. By pre-training the model on broader, yet related, medical and radiological language, the model gains necessary domain expertise even when specific clinical examples are few.

Diminishing Returns with Large-Scale Fine-Tuning

However, the study also highlighted the concept of diminishing returns when scaling up the fine-tuning process. When both models were subjected to large-scale fine-tuning (DA + FTl and Bio + FTl), the performance gap narrowed considerably. The scores for DA + FTl (24.9 ± 4.0) and Bio + FTl (25.0 ± 3.8) were statistically indistinguishable (P = 0.89). This suggests that while massive datasets can push performance toward a plateau, the incremental benefit derived from large-scale fine-tuning over a well-adapted model is minimal. This observation is critical for resource allocation in clinical AI deployment, suggesting that the effort and computational cost of acquiring and processing massive fine-tuning datasets may not yield proportional improvements in expert-rated quality.

Comparison to Commercial Models

A notable comparative finding involved the performance of open-source models versus the commercial Gemini-1.5-Flash-002. The research demonstrated that the large-scale fine-tuned open-source models (DA + FTl and Bio + FTl) surpassed the commercial model in both expert-rated quality and in the rate of generating harmful outputs. This outcome suggests that, under optimal fine-tuning conditions, open-source LLMs can be tailored to meet or exceed the performance benchmarks set by proprietary, closed-source models, particularly regarding safety and clinical accuracy.

Limitations in Inter-Rater Agreement and Automated Metrics

Despite the technical improvements observed, the study underscored significant practical limitations in the clinical adoption of these AI tools. The agreement among the four independent radiologists reviewing the generated impressions was low, as evidenced by an Intraclass Correlation Coefficient (ICC) ranging from 0.22 to 0.49. This low inter-rater agreement highlights the inherent variability in clinical judgment and the difficulty in establishing a single ‘ground truth’ for subjective impression generation. Furthermore, the automated NLP metrics utilized in the study showed a weak correlation with the human expert evaluations, with a correlation coefficient (ρ) less than 0.32. This discrepancy is a crucial finding, indicating that standard, automated NLP metrics are insufficient proxies for clinical relevance and quality in radiology.

Clinical Implications and Deployment Threshold

The findings provide a framework for defining the data-efficiency threshold for LLMs in clinical practice. For institutions dealing with limited labeled clinical data, the strategy of domain adaptation followed by light fine-tuning appears to be the most cost-effective and efficient path to achieving near-ground-truth radiology impressions. Conversely, institutions with access to vast archives might prioritize large-scale fine-tuning, although the marginal gains are small. Crucially, the study reinforces the necessity of human oversight, as the low correlation between automated metrics and human judgment mandates that expert-informed evaluation remains the gold standard for clinical validation.

Conclusion on AI in Radiology

In summary, this research provides a nuanced perspective on the deployment of LLMs in medical imaging interpretation. Domain adaptation proves to be a powerful tool for bootstrapping performance in data-constrained environments, while large-scale fine-tuning offers only marginal, diminishing returns. The low correlation between automated metrics and radiologist scores underscores a fundamental challenge in medical AI: the inability of current automated evaluation tools to capture the complexity and nuance of expert clinical reasoning. Ultimately, while LLMs show potential, their integration into clinical workflows requires robust, human-centric validation protocols.

Subject of Research: Evaluating domain adaptation and fine-tuning strategies for large language models in generating radiology reports.

Article Title: Domain Adaptation of a Large Language Model for Radiology Impression Generation – Multi-Reader Evaluation

Article References: Sorin, V., Reynolds, M. C., Hara, A. K., Gupta, A. C., Collins, J. D., Jensen, Z. D., Vassilieva, N., Wolfe, J., Rezaei, S., Callstrom, M. R., & Korfiatis, P. (2026). Domain Adaptation of a Large Language Model for Radiology Impression Generation – Multi-Reader Evaluation. Journal of Medical Systems, 50(1), Article 156. https://doi.org/10.1007/s10916-026-02484-0

Image Credits: AI Generated

DOI: 10.1007/s10916-026-02484-0

Keywords: LLMs, Domain Adaptation, Radiology, Medical AI, Natural Language Processing, Fine-Tuning, Llama-3, Clinical NLP, Domain, Adaptation, Large, Language