LLMs underperform specialized platform in 475-variant clinical classification

General-purpose large language models were substantially less consistent than a specialized clinical interpretation platform when classifying genetic variants, even after receiving detailed instructions. In an analysis accepted for publication in BMC Bioinformatics , Claude, ChatGPT, Gemini and Perplexity achieved a mean concordance of 69.0% with reference classifications established by expert variant scientists. The commercial non-LLM platform tested reached 81.1% concordance, highlighting the limits of conversational prompting for autonomous clinical use.

The study examined 475 variants across 185 genes. Each language model was given a detailed structured prompting strategy, and its classifications were compared both with expert references and with the specialized platform. The mean performance across the language models included 70.2% precision and 68.2% recall, whereas the specialized platform achieved 80.2% precision and 84.5% recall.

Detailed prompting generally improved language-model performance, but it did not eliminate the gap between the models and the dedicated interpretation tool. Results varied markedly by variant type. Unadjusted mean concordance ranged from 87.2% for intronic variants to 56.0% for splice-region variants.

The researchers also evaluated adjusted associations between performance and variant characteristics and found them to be model-specific rather than uniform across the four systems. These findings show why the overall 69.0% mean does not describe every variant equally well: the models’ agreement with expert classifications depended on both the model and the type of variant being assessed. The evaluation went beyond final classifications to examine performance on individual evidence criteria.

Sensitivity exceeded 80% for BP7, PP3, PVS1 and PM2, indicating that the language models frequently identified or applied these selected criteria. Their performance was less consistent at other levels of the task. Criterion-call precision and exact-strength agreement were substantially lower for several criteria, showing that recognition of a criterion did not reliably translate into a correct final classification or the correct strength of that criterion.

Overall, the study supports a narrow conclusion about the tested approach: general-purpose language models operating through direct or structured prompting remained substantially limited relative to a specialized non-LLM interpretation platform. Instead, it directly tests their support for clinical genetic variant classification and finds that asking a general-purpose model to perform the task, even with detailed instructions, is insufficient for autonomous clinical classification. The results also underscore the difference between detecting selected evidence signals and consistently reproducing expert classifications across variants and criteria.

Subject of Research: Benchmarking general-purpose large language models for clinical genetic variant classification

Article Title: Prompting alone is insufficient for large language model-based clinical variant classification

Article References: Marek-Yagel, D., Lev, M., Kfir, K., Shomer, I., Kosowsky, J., Greenberg, R., Balicer, R., Ben-Shachar, S., & Isakov, O. (2026). Prompting alone is insufficient for large language model-based clinical variant classification. BMC Bioinformatics. https://doi.org/10.1186/s12859-026-06720-9

Image Credits: AI Generated

DOI: 10.1186/s12859-026-06720-9

Keywords: large language models, clinical genetics, genetic variant classification, variant interpretation, Claude, ChatGPT, Gemini, Perplexity, prompt engineering, bioinformatics, Prompting, alone