AI Learns to Check Its Own X-Ray Reports by Searching Past Cases Before Writing

Chest X-rays are the workhorse of modern medical imaging. They are cheap, quick, widely available, and deliver a low radiation dose, which is why they account for roughly 40 percent of the 3.6 billion diagnostic X-ray examinations performed worldwide every year. Each of those images, however, needs a written report, and producing one is a slow, labor-intensive task that depends heavily on the experience of the radiologist and the availability of trained staff. A team of researchers from India and Romania now reports a new artificial intelligence framework that could ease that burden by making automated report writing more accurate, and their approach does something surprisingly simple: it lets the AI look up similar cases from a library of past reports before committing its findings to paper.

The study, published in Frontiers in Digital Health, tackles a well-known weakness of vision-language models, the class of artificial intelligence systems designed to connect what a machine sees with what it can say. When these models generate radiology reports, they rely entirely on the representations they learned during training. If the relevant contextual information is not captured, the model may produce statements that are incorrect, unsupported, or outright fabricated, a phenomenon researchers call hallucination. Even state-of-the-art systems evaluated by trained radiologists have been shown to generate reports containing missing or false references, a serious problem in a domain where precision of diagnostic terminology matters.

The new framework, developed by Anandhu Saji, Fousia M. Shamsudeen, Chinnu Jacob, Cristina Popirlan, Claudiu-Ionut Popirlan, and D. Jude Hemanth, builds on an established vision-language model called R2Gen, which was designed specifically for medical report generation. The workflow operates in three stages. First, R2Gen processes the input chest X-ray through a ResNet-101 convolutional encoder that converts the image into a dense visual feature vector, and its language decoder produces a preliminary report. Second, a retrieval stage searches an external repository of radiology reports for cases similar to the one at hand. Third, a refinement stage powered by the Zephyr 7B large language model combines the preliminary report with the retrieved textual context and produces the final, polished report.

The retrieval mechanism is the technical heart of the system. The researchers used FAISS, an efficient library for similarity search developed for exactly this kind of vector-based lookup. For every image-report pair in the repository, the visual feature vector extracted by the R2Gen encoder is stored as the searchable vector in the FAISS index, alongside a fixed-dimensional semantic embedding of the corresponding report generated by the sentence transformer model all-MiniLM-L6-v2. At inference time, the query image yields both an image embedding and, via its preliminary report, a text embedding. A joint scoring function based on cosine similarity combines visual and semantic similarity, and the top-k most similar cases are retrieved. Their associated report information is then handed to the refinement model as textual context.

One of the most practical aspects of the design is what it does not require: retraining. The retrieval and refinement stages operate entirely at inference time, meaning the pretrained R2Gen component is left untouched. There is no additional fine-tuning of the underlying report generation model, and the retrieval repository can be extended or updated independently simply by rebuilding the retrieval index when needed. In a field where most recent advances have come from computationally expensive fine-tuning of large language models, this modular, plug-in approach offers a flexible alternative. The authors describe the two specialized modules, a Retrieval Agent and a Report Generation Agent, as functionally specialized components within a predefined pipeline rather than an autonomous multi-agent reasoning system.

To test the framework, the team used two benchmark datasets. IU X-Ray contains 7,470 chest X-ray images and 3,955 radiology reports, each structured into indication, comparison, findings, and impression sections. CheXpert Plus is far larger, with 223,000 image-report pairs from 64,700 patients and annotations for 14 chest conditions. In both cases the data were split 80:20, with the larger portion used to build the FAISS vector database and the remaining 20 percent reserved for evaluation. The retrieval databases were constructed independently for each dataset, patient-level separation was maintained, and the ground truth reports of the evaluation cases were used only for scoring, never as input to the generation process.

The results were evaluated with standard natural language generation metrics, each capturing a different aspect of text quality. BLEU measures the precision of n-gram matches, ROUGE-L focuses on recall-oriented overlap of the longest common sequences, METEOR accounts for stemming and synonym matching, and token-overlap precision, recall, and F1-score quantify how closely generated tokens match the reference. The gains over the R2Gen baseline were substantial. On CheXpert Plus, the baseline achieved a BLEU-4 score of 0.081, while the proposed framework reached 0.2085, with ROUGE-L rising from 0.246 to 0.3622 and METEOR from 0.318 to 0.3917. On IU X-Ray, BLEU-4 climbed from 0.165 to 0.3939, ROUGE-L from 0.371 to 0.5070, and METEOR from 0.187 to 0.5382.

The comparison extended to previously published state-of-the-art methods, including R2GenGPT, R2GenCSR, MambaXray-VL-L, METTransformer, Token-Mixer, and BootstrappingLLM. On CheXpert Plus, the proposed method’s BLEU-4 of 0.2085 and F1-score of 0.4546 exceeded the 0.112 and 0.335 reported for MambaXray-VL-L, the strongest prior result in that comparison. On IU X-Ray, the framework’s BLEU-4 of 0.3939, ROUGE-L of 0.5070, and METEOR of 0.5382 surpassed the 0.206, 0.401, and 0.215 reported for R2GenCSR, while also achieving precision, recall, and F1 values of 0.5997, 0.6976, and 0.6338. Qualitative analysis reinforced the numbers: the generated reports captured key diagnostic elements such as clear lung fields, absence of effusions or consolidations, a normal cardiac silhouette, and intact bony structures.

The work sits within a rapidly growing body of research on retrieval-augmented generation, or RAG, in radiology. A recent scoping review of 45 RAG-based studies found that retrieval consistently improves report generation and question answering performance, but also flagged hallucinations, inconsistent evaluation practices, and limited clinical validation as persistent challenges. Other recent approaches have explored semantically guided retrieval, factual evidence fusion, and multi-agent conversational systems. The present study distinguishes itself less by inventing a new retrieval algorithm and more by its modular organization: it shows that simply pairing an established report generator with inference-time retrieval and a separate language model for refinement can deliver measurable gains without touching the trained model.

The authors are careful about the scope of their claims. Automatic metrics like BLEU and ROUGE measure textual similarity, not clinical correctness, and improvements in these scores should not be read as direct evidence of diagnostic accuracy. The retrieval stage is also only as good as the repository behind it, so the characteristics of the stored reports inevitably shape the context supplied during refinement. Future work, they suggest, should incorporate dedicated clinical factuality measures, extend evaluation to additional datasets and imaging modalities, and explore more clinically informed retrieval strategies. Still, the message is compelling: sometimes the best way to make an AI a better writer is not to train it harder, but to give it a reference library and let it check its work against the accumulated experience of the reports that came before.

Subject of Research: Retrieval-augmented chest X-ray report generation using an agent-based framework combining R2Gen, FAISS similarity search, and Zephyr 7B report refinement

Article Title: Enhanced vision language framework for medical report generation using agent-based retrieval

Article References: Saji, A., Shamsudeen, F. M., Jacob, C., Popirlan, C., Popirlan, C.-I., & Hemanth, D. J. (2026). Enhanced vision language framework for medical report generation using agent-based retrieval. Frontiers in Digital Health, 8, Article 1989756. https://doi.org/10.3389/fdgth.2026.1989756

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.1989756

Keywords: chest X-ray, radiology report generation, retrieval-augmented generation, vision-language models, R2Gen, FAISS, Zephyr 7B, medical imaging, artificial intelligence, hallucination, natural language processing, health informatics