Structured prompts improve Chinese LLM radiology impressions in retrospective validation

Structured prompts improved the quality of diagnostic impressions generated from liver MRI findings by four Chinese large language models in a retrospective validation study. The research included 15,127 primary-center liver MRI reports, with additional evaluation in 1,009 liver MRI reports from an external center and 1,692 pulmonary thin-section CT reports. The study measured report-level performance rather than patient outcomes or the models’ ability to replace radiologists.

Investigators supplied imaging findings, while withholding consensus diagnostic conclusions confirmed by two associate-senior or senior radiologists, to Kimi-K2, DeepSeek-V3, ByteDance-Seed-OSS-36B-Instruct and Qwen3-235B-A22B-Instruct-2507. They compared prompts containing only a role definition or basic task instructions with prompts that varied in structure, composition and number of examples. Generated impressions were assessed with a Multi-Dimensional Quality Assessment framework, or MDQA, covering semantic coherence, diagnostic correctness, clinical prioritization alignment, top-1 matching with the reference conclusion and a composite score.

Structured prompts produced better diagnostic correctness, prioritization and overall MDQA scores than role-only or basic task prompts. Increasing the number of examples had a more selective effect, mainly improving semantic coherence and top-1 matching. The highest composite score reported for the primary cohort was 0.7696 for Kimi-K2 with Prompt 11, while DeepSeek-V3 reached 0.7613 with Prompt 10. These are scores under the study’s MDQA framework, not measures of accuracy against individual patient diagnoses.

In the external-center cohort, a locally adapted Prompt 11 V3 achieved the highest MDQA scores for Kimi-K2 and DeepSeek-V3. The cross-disease analysis applied the approach to pulmonary thin-section CT findings: the MDQA score increased from 0.4773 ± 0.1701 with Prompt 0 to 0.7109 ± 0.1676 with Prompt 10. These results show that the tested prompts and assessment approach were used beyond the original liver MRI dataset, although the external analysis remained retrospective and the pulmonary cohort involved CT rather than MRI.

Radiologist ratings followed trends similar to the MDQA measurements. Together, the findings support structured prompt design and multidimensional evaluation as methods for examining consistency and report-level quality in LLM-generated radiology impressions. The study does not establish that the approach improves clinical decisions, patient care or diagnostic performance in prospective practice. Its conclusions are limited to the tested models, prompts, retrospective report cohorts and evaluation framework, including the transfer from liver MRI to pulmonary CT reports.

Subject of Research: Evaluation of structured prompts and multidimensional quality assessment for LLM-generated radiology diagnostic impressions

Article Title: Prompt optimization and multidimensional quality assessment for Chinese large language model-generated diagnostic impressions from liver MRI findings: external-center and cross-disease validation

Article References: Wang, Q., Sun, X., Chen, Y., Cheng, J., Liu, Y., Zhang, X., Zhang, Z., Liu, C., Li, X., & Chen, W. (2026). Prompt optimization and multidimensional quality assessment for Chinese large language model-generated diagnostic impressions from liver MRI findings: external-center and cross-disease validation. BMC Medical Imaging. https://doi.org/10.1186/s12880-026-02849-4

Image Credits: AI Generated

DOI: 10.1186/s12880-026-02849-4

Keywords: large language models, prompt optimization, radiology reports, liver MRI, quality assessment, external validation, pulmonary CT, diagnostic impressions, Chinese AI, Prompt, optimization, multidimensional