Multimodal RAG framework improves question answering over lengthy documents

Researchers at South China Normal University have unveiled MMRDoc, a new retrieval-augmented generation framework designed to help artificial intelligence systems read, search and reason over long, visually dense documents such as financial reports, scientific papers and multi-page contracts. The study, published in the Journal of Intelligent Information Systems by Shengxu Xu and Hongsong Wang of the School of Artificial Intelligence, reports state-of-the-art results on three major document question answering benchmarks — MMLongBench-Doc, PaperTab and FetaTab — while remaining robust on the extreme-length LongDocURL benchmark.

At the heart of the work lies a problem that has quietly limited the practical usefulness of multimodal AI systems: the way documents are physically divided into pages does not match the way humans actually reason about them. A financial table that stretches across three pages, or an argument that begins in one chapter and concludes in another, is fragmented by pagination into logically disconnected visual chunks. Most current multimodal retrieval-augmented generation (RAG) systems retrieve entire pages as images and feed them to a large vision-language model. This page-level strategy, the authors argue, suffers from two intertwined failures they call “feature dilution” and “logical fragmentation.”

Feature dilution occurs when a page is packed with heterogeneous content — dense text, charts, footnotes, headers and tables competing for the model’s attention. When such pages are retrieved as single visual units, the genuinely relevant evidence is diluted by a sea of irrelevant visual detail, degrading the precision with which the system can locate the fine-grained facts needed to answer a question. Logical fragmentation, meanwhile, arises because the semantic units that matter — a complete table, a coherent argument, a referenced figure — often span page boundaries. Retrieval at the physical page level severs these semantic connections, making cross-page reasoning fragile and error-prone.

MMRDoc addresses these weaknesses through a three-stage architecture built on structural awareness and multi-granularity synergistic retrieval. The first stage, Semantic Integrity Reconstruction (SIR), tackles fragmentation before any retrieval takes place. Rather than treating each page as an atomic unit, the framework reconstructs semantically coherent segments that bridge cross-layout and cross-modal boundaries — stitching together a table split across pages, for example, or reuniting a figure with the text passage that discusses it. By repairing these broken semantic links at the document-structure level, SIR gives the downstream retrieval machinery evidence that is complete rather than arbitrarily truncated.

The second stage is a Synergistic Dual-path Retrieval mechanism, which deliberately hedges between two complementary levels of granularity. In parallel, the system retrieves the Top-K macro-pages containing global visual information — preserving layout, spatial relationships and visual context that OCR-based pipelines tend to discard — while simultaneously retrieving fine-grained semantic features from micro-elements such as individual paragraphs, table cells, or figure captions. This dual-path design means the answering model can draw both on the “where it looks right” signal of page-level imagery and on the “what precisely was said” signal of element-level text. Neither alone is sufficient: page images alone drown the model in detail, while element-level text alone loses the visual grounding that modern OCR-free document models depend on.

Multi-granularity retrieval, however, creates its own difficulties, and the third stage of MMRDoc is designed to resolve them. Retrieving candidates at two levels simultaneously produces a heterogeneous candidate pool plagued by information redundancy and renewed logical fragmentation — the same evidence may appear in multiple forms, and micro-elements can arrive stripped of their surrounding context. To manage this, MMRDoc dynamically aggregates the heterogeneous candidate sets and passes them through a Structural-aware Reranking module that evaluates all candidates in a unified manner. This module enables what the authors describe as a logical transition from coarse-grained recall to fine-grained filtering: broad nets are cast first, and only the most structurally coherent, question-relevant evidence survives to reach the generation stage.

The empirical results reported in the paper are striking. On MMLongBench-Doc, a benchmark specifically designed to test long-context document understanding with visualizations, on PaperTab, and on FetaTab, MMRDoc achieves state-of-the-art performance, outperforming existing visual RAG baselines and multi-agent systems. Critically, the gains are concentrated exactly where the framework’s design predicts they should appear: in cross-page table understanding and long-range reasoning, the two categories of questions most punishing for page-level retrieval. On LongDocURL, a benchmark that pushes document lengths to extremes, MMRDoc maintains strong robustness, suggesting the multi-granularity strategy scales rather than collapsing as context grows.

The work arrives at a moment when the tension between long context windows and retrieval has become one of the central debates in applied AI. Simply extending the context length of large vision-language models — systems such as Qwen2.5-VL, InternVL3, GPT-4o and Llama 3 — is tempting but expensive and, as prior research on the “lost in the middle” phenomenon has shown, not reliably effective: models demonstrably underuse information buried in the middle of long contexts. RAG offers an alternative, feeding only relevant evidence to the model, but conventional RAG was built for plain text and struggles when the evidence lives in images, tables and complex layouts. Vision-based document retrieval systems such as ColPali and VisRAG extended retrieval to page images, and multi-agent frameworks such as MDocAgent distributed the work across cooperating components, but the granularity mismatch problem has persisted.

MMRDoc’s contribution is to show that granularity itself can be treated as a first-class design dimension rather than a fixed choice. By synergizing macro-level visual recall with micro-level semantic filtering, and by binding both to an explicit model of document structure, the framework avoids the false dichotomy between “retrieve whole pages” and “parse everything into text.” The Structural-aware Reranking step is particularly notable because it performs unified evaluation across candidate types that are intrinsically different in nature — a page image is not commensurable with a table cell — by scoring them against a shared structural representation of the document.

The practical implications extend well beyond academic benchmarks. Enterprises, governments and research institutions sit on vast archives of visually rich documents — insurance claims, clinical records, engineering drawings, regulatory filings — where answers routinely require combining a number from page 47 with a clause from page 112. Systems that fragment such evidence across page boundaries fail silently and confidently, producing plausible answers grounded in incomplete context. A framework that explicitly reconstructs semantic integrity and then retrieves at multiple granularities promises more trustworthy evidence localization, which is a prerequisite for deploying document question answering in high-stakes settings.

The authors acknowledge that the field is moving quickly. Companion efforts published in the same venue and elsewhere — including FactualRAG for fact-checking, MoLoRAG for logic-aware retrieval, URaG for unified retrieval and generation, and VRAG-RL for reinforcement-learning-enhanced visual retrieval — indicate a broad convergence on the idea that document understanding requires structure-aware, retrieval-centric architectures rather than monolithic long-context models. MMRDoc’s multi-granularity approach represents one of the most complete articulations of that principle to date, tested against three major benchmarks simultaneously.

The datasets used in the evaluation — MMLongBench-Doc, PaperTab, FetaTab and LongDocURL — are publicly available, and the authors state that the source code and associated models will be released publicly upon acceptance of the manuscript, which should allow other research groups to reproduce and extend the results. The research was conducted at the School of Artificial Intelligence at South China Normal University in Guangzhou, with Xu performing the implementation and experiments and Wang supervising the research. The authors declare no competing interests and no external funding.

As multimodal AI systems are increasingly asked not merely to describe a single image but to interrogate documents of hundreds of pages, the MMRDoc results suggest a clear lesson: the physics of the page must yield to the logic of the document. Systems that reconstruct what a document means — rather than merely retrieving what it looks like — appear set to define the next generation of document intelligence.