PROBE benchmark exposes gaps in AI protein-function annotation

General-purpose large language models can connect natural-language descriptions of experimentally characterized proteins with biologically related Gene Ontology terms, but a new multi-organism benchmark shows that their exact annotations remain too unreliable for autonomous use. In the PROBE evaluation, the best of 10 tested open-source models achieved a macro-F1 score of just 9.3%, while a simple heuristic that selected the five most frequently predicted terms outperformed four of the models. The study, published in BMC Bioinformatics on 20 October 2026, concludes that current systems show meaningful semantic alignment with protein function while frequently missing the precise Gene Ontology terms required for rigorous annotation.

PROBE—PRotein functiOn Benchmark Evaluation—contains 1,000 proteins drawn from UniProt Swiss-Prot across five organisms. Every reference annotation in the benchmark is supported by experimental evidence, providing a gold-standard test of whether models can translate descriptions of characterized protein functions into the correct Gene Ontology terms without task-specific training. The authors evaluated 10 open-source large language models using five prompt formats under zero-shot and inference-only conditions. Across those combinations, the models generated 150,000 predictions. The scale of the evaluation exposed both occasional biological competence and a persistent gap between apparently plausible language-model output and exact, evidence-based functional labeling.

Exact-match performance was weak. Mistral Large 123B produced the highest macro-F1 among the open-source systems, reaching 9.3%, while a most-frequent-five baseline performed better than four of the 10 models. A major source of error was the tendency to predict Gene Ontology terms absent from the benchmark annotations: 41.3% of predicted terms were unmatched. Researchers checked this result against a temporal holdout using current UniProt records and found that only 1.25% of those unmatched predictions had newly gained support. This check suggests that most unmatched terms represented genuine errors rather than proteins that had simply been annotated after the benchmark was assembled, although it does not establish that every unmatched prediction is incorrect.

Prompt design nevertheless changed the results substantially. The candidate-selection format, designated P5, reduced the unmatched-prediction rate by 33.5 percentage points relative to the P1 format and achieved the highest F1 score, at 8.2%. This improvement indicates that giving models a constrained set of candidate terms can make their outputs more compatible with the task, even when exact annotation accuracy remains limited. The findings also varied across organisms: the yeast Saccharomyces cerevisiae unexpectedly had the strongest score, with an F1 of 8.6%. Because the benchmark covered only five organisms, however, the authors said the pattern could not be tied reliably to evolutionary distance. Neither literature density nor the size of the available annotation pool reached statistical significance, leaving organism-level performance differences as an open question rather than an explained biological effect.

The analysis also revealed information that exact-match scores alone could not capture. A Gene Ontology semantic-similarity assessment found that the models’ predictions were 2.1 to 2.6 times more biologically similar to correct answers than random predictions across all namespaces. In other words, the systems often selected terms related to the intended protein function even when they did not reproduce the reference annotation exactly. A preliminary comparison involving 50 proteins and four closed-source models produced F1 scores from 0.081 to 0.127, compared with 0.099 for the best open-source model. The difference of 0.028 points suggests, but does not establish, that similar limitations may affect commercial systems as well. Overall, the research indicates that general-purpose language models can extract some functional signal from protein descriptions, but their high rate of unsupported or incorrectly specific predictions prevents them from replacing curated annotation workflows. The benchmark, evaluation pipeline and all 150,000 experimental model-response records were released publicly to support reproducibility.

Subject of Research: Benchmarking large language models for Gene Ontology protein-function annotation across five organisms

Article Title: PROBE: a multi-organism benchmark for evaluating general-purpose large language models on Gene Ontology protein function annotation

Article References: Naha, K., & Jamil, H. (2026). PROBE: a multi-organism benchmark for evaluating general-purpose large language models on Gene Ontology protein function annotation. BMC Bioinformatics. https://doi.org/10.1186/s12859-026-06697-5

Image Credits: AI Generated

DOI: 10.1186/s12859-026-06697-5

Keywords: PROBE benchmark, large language models, protein function, Gene Ontology, UniProt Swiss-Prot, computational biology, annotation accuracy, semantic similarity, prompt engineering, bioinformatics, PROBE, multi-organism