Prompting boosts multimodal models modestly in tongue-diagnosis benchmark

A prompt-based evaluation of nine large multimodal models (LMMs) found that adding Traditional Chinese Medicine knowledge and instructions to check contradictions produced modest aggregate improvements in reconstructing expert-annotated tongue signs, but the benefit varied substantially among models and did not establish clinical utility. The TongueBench study evaluated API-accessed LMMs on 2,694 tongue images carrying 22 labels across tongue-body shape, body color, coating texture and coating color. Direct prompting was compared with knowledge-guided prompting and knowledge-guided reflective prompting, or KGRP, which added contradiction review, mutual-exclusion checks, conservative rollback and hard-case judgment before labels were emitted. No task-specific model training was performed, making prompt construction the principal experimental intervention.

The setting is unusually demanding because tongue diagnosis is not a conventional single-label image-classification task. A model must reconstruct several visually related features at once, including fissures, tooth marks, moisture, coating thickness, body color and coating color. Some labels are mutually exclusive, others commonly coexist, and several depend on subtle boundaries or color distinctions. The researchers therefore evaluated not only conventional accuracy but also whether models could recover a complete 22-label set, respect the ontology, and remain consistent when asked to generate an answer more than once. Three generations were requested for each image-model-prompt condition, with strict-majority voting across retained attempts used for primary task metrics.

Across the fixed set of nine models, KGRP increased mean Micro-F1 from 47.57% under direct prompting to 48.51%, Macro-F1 from 32.41% to 33.60% and sample-level Jaccard from 34.45% to 35.32%. Mean macro balanced accuracy rose from 58.75% to 59.38%, while Hamming loss fell from 26.39% to 25.54%. An exploratory paired bootstrap over images placed the mean KGRP-minus-direct-prompt difference in Micro-F1 at 0.94 percentage points, with a 95% percentile interval of 0.77 to 1.10 points; the corresponding Macro-F1 difference was 1.19 points, with an interval of 0.93 to 1.44. These intervals are conditional on the selected models, archived outputs and scoring protocol, and they assume independent images, an assumption the researchers could not verify because participant identifiers were unavailable.

The small average gain concealed substantial model-level differences. KGRP produced the highest Micro-F1 in seven of the nine systems, but direct prompting performed better for GPT-5.3 and Qwen3-Omni-30B-A3B. Gemini-3.1-Pro recorded the largest Micro-F1 increase, rising 3.65 percentage points, followed by Kimi-K2.6 at 2.35 points and Doubao-Seed-2.0-Pro at 2.09 points. In the other direction, Qwen3-Omni-30B-A3B declined by 1.19 points and GPT-5.3 by 1.11 points. Doubao-Seed-2.0-Pro and Gemini-3.1-Pro had the strongest KGRP profiles, but MiniMax-M2.7 was an extreme outlier: even with reflective prompting, its Micro-F1 was only 0.36% and its parse success rate was 10.79%. Excluding that model left the ordering of the prompt strategies unchanged, supporting the conclusion that the aggregate gain was modest rather than an artifact of the outlier alone.

Performance also differed sharply among individual diagnostic labels. The highest mean F1 values across model-prompt combinations occurred for white tongue coating at 72.53%, thin tongue coating at 71.69%, light-red tongue at 68.31% and tongue with cracks at 62.61%. Several labels remained difficult, including thin tongue at 5.29%, no tongue coating at 5.61%, blue-purple tongue at 11.05%, gray-black tongue coating at 11.46% and ecchymosis at 12.72%. Reflective prompting produced the largest mean gains for tender tongue, no tongue coating, tough tongue, dark-red and red tongue, blue-purple tongue and thick and thin coating, but declines occurred for tongue with teeth marks, pale tongue, gray-black coating and ecchymosis. Domain-level results suggested that aggregate improvements were concentrated mainly in body color and coating texture, while shape and coating color were comparatively stable.

A notable limitation was the difficulty of reconstructing complete label sets. Under KGRP, the mean exact-match ratio rose only from 0.40% to 0.49%. For 2,607 images, representing 96.77% of the benchmark, none of the nine models under KGRP exactly recovered the full reference label set. Many images contained five to seven positive labels, so any omitted or additional feature invalidated exact agreement. The study also found low detection sensitivity for several labels, although it did not determine the causes. Under KGRP, thin tongue had a mean F1 of 5.28% and recall of 3.12%; blue-purple tongue had 12.65% F1 and 7.53% recall; and gray-black coating had 10.31% F1 and 5.77% recall. The study did not assess how individual errors might affect a clinician’s diagnosis or treatment decisions.

Repeated generations also showed that cleaner aggregate scores did not necessarily mean more consistent behavior. Mean pairwise Jaccard consistency was 70.97% under direct prompting and 70.03% under KGRP, while the proportion of attempt pairs with identical label sets declined from 26.02% to 23.66%. Dynamic switching between mutually exclusive labels increased slightly, from 2.73% to 3.07%. A separate raw-output audit found ontology violations in 851 of 194,813 successfully parsed generations, or 0.437% overall. Pooled conflict incidence was 0.425% for direct prompting, 0.810% for knowledge-guided prompting and 0.077% for KGRP, suggesting fewer raw contradictions under reflection. However, post-processed conflict rates were zero under every condition because deterministic evaluator rules removed violations, meaning the cleaned output did not independently demonstrate that the models had mastered the ontology. Parse failures, valid empty outputs and missing generation records were also tracked separately rather than treated as successful non-empty responses.

Three supervised computer-vision systems provided task-specific reference points on the same 270-image test set: Faster R-CNN, LGAN and IF-RCNet. All three had higher F1 and Jaccard scores and lower Hamming loss than every KGRP LMM. Faster R-CNN recorded 73.00% Micro-F1, 64.15% Macro-F1 and 59.86% sample-level Jaccard, compared with a KGRP LMM range of 0.48% to 59.11% for Micro-F1. This is a descriptive comparison rather than a definitive ranking, because the data were split by image and repeated acquisitions from the same participant could not be grouped. Participant overlap between training and test subsets could not be excluded, potentially making the supervised results optimistic, while the LMMs were not trained on the benchmark and their potential pretraining exposure to related tongue-image data could not be ruled out.

Several additional limitations constrain the findings. The retained files lacked validated participant-to-image links, patient counts and per-participant image counts, so the image-level bootstrap may be too narrow and patient-level generalizability cannot be assessed. Original independent annotations and adjudication records were also unavailable, preventing calculation of inter-rater agreement and leaving the reliability of the expert reference standard unquantified. The three attempted generations provide only a preliminary stability probe, and provider-side decoding parameters, retries and endpoint updates were not consistently recorded, so the historical API calls cannot be reproduced exactly. Incomplete acquisition, color-calibration and demographic information also prevented evaluation of device robustness, demographic fairness or cross-institutional performance. The study did not establish visual grounding, universal prompt superiority or readiness for autonomous diagnosis, but its controlled protocol shows that prompt design can measurably shift multimodal-model behavior while leaving substantial failures in parsing, ontology adherence, repeated-output consistency and complete multi-label reconstruction visible.

Subject of Research: Evaluation of prompt-engineered multimodal models for multi-label Traditional Chinese Medicine tongue-image assessment

Article Title: TongueBench: performance of prompt-engineered large multimodal models in Traditional Chinese Medicine tongue diagnosis

Article References: Meng, Y., Cui, N., Yao, J., Kong, S., Liu, P., Zhang, Z., & Liang, H. (2026). TongueBench: performance of prompt-engineered large multimodal models in Traditional Chinese Medicine tongue diagnosis. Frontiers in Artificial Intelligence, 9, Article 1943004. https://doi.org/10.3389/frai.2026.1943004

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1943004

Keywords: TongueBench, large multimodal models, Traditional Chinese Medicine, tongue diagnosis, prompt engineering, medical AI evaluation, multi-label classification, model benchmarking, ontology-constrained inference, performance, prompt-engineered, large