A multimodal framework named CREST effectively improves the extraction of chemical reaction information from scientific literature on a self-built evaluation dataset, according to a research study published in the Journal on Image and Video Processing. The development of large language models for the chemical domain relies heavily on access to high-quality structured data, but key experimental details are frequently distributed across published documents in varied formats. The newly reported system demonstrates that an end-to-end pipeline can achieve extraction performance comparable to certain closed-source models while relying on a relatively small model size.
The research identifies a major operational challenge in chemical informatics: crucial experimental information in chemical literature is scattered across scientific PDF documents in multimodal forms. Within these documents, experimental records are presented through visual reaction schemes, dense experimental tables, figure captions, and supplementary footnotes. This diverse presentation makes automated structured information extraction substantially more difficult than traditional single-modality text extraction.
The authors outline several specific problems that have limited automated extraction in previous workflows. Chief among these issues are the technical difficulty of finely segmenting reaction schemes located within complex publication layouts, the core challenge of conducting unified semantic parsing across multiple modalities simultaneously, and the widespread lack of an end-to-end workflow designed for local deployment and continuous model optimization.
To address layout segmentation hurdles directly, the CREST framework combines two tools, VisualHeist and MinerU, to carry out fine-grained segmentation of PDF document content. In addition to segmenting visual elements, this processing pipeline systematically extracts document titles, descriptive text blocks, and footnotes. These extracted textual components are subsequently utilized as auxiliary text prompts to support downstream interpretation.
At the core of the framework is an end-to-end information extraction pipeline built upon the Qwen3 multimodal language model. By implementing Low-Rank Adaptation (LoRA) fine-tuning, the research team trained the Qwen3 architecture to jointly parse graphical reaction schemes, experimental data tables, and related textual descriptions within a unified semantic parsing workflow.
To validate the system’s extraction accuracy, the researchers developed an evaluation mechanism that integrates automated evaluation metrics with independent expert verification. In experimental trials conducted on the self-built chemical literature dataset, CREST demonstrated effective improvements in multimodal reaction information extraction, achieving performance levels comparable to closed-source models despite its smaller model size.
The authors emphasized that these experimental findings confirm the practical value and application potential of the framework for chemical research. Moving beyond experimental testing, the team also deployed a functional online information extraction service powered by CREST, making the multimodal extraction pipeline accessible to the broader scientific community through a dedicated online portal.
Subject of Research: Multimodal chemical reaction information extraction from scientific literature
Article Title: CREST: Chemical Reaction Extraction from Scientific Texts
Article References: Li, X., Liang, H., Wang, X., Yang, F., & Xu, F. (2026). CREST: Chemical Reaction Extraction from Scientific Texts. Journal on Image and Video Processing. https://doi.org/10.1186/s13640-026-00704-8
Image Credits: AI Generated
DOI: 10.1186/s13640-026-00704-8
Keywords: chemical reaction extraction, scientific literature, multimodal framework, CREST, Qwen3, LoRA fine-tuning, VisualHeist, MinerU, PDF segmentation, reaction schemes, experimental tables, figure captions, chemical domain language models, semantic parsing, expert verification, automated evaluation, structured data extraction
Cite Scienmag News
Gavin Prescott. (November 17, 2026). Multimodal Framework Improves Chemical Reaction Extraction From Complex Scientific Texts. Scienmag. https://scienmag.com/multimodal-framework-improves-chemical-reaction-extraction-from-complex-scientific-texts/
Gavin Prescott. "Multimodal Framework Improves Chemical Reaction Extraction From Complex Scientific Texts." Scienmag, 17 November 2026, https://scienmag.com/multimodal-framework-improves-chemical-reaction-extraction-from-complex-scientific-texts/. Accessed 17 November 2026.
Gavin Prescott. "Multimodal Framework Improves Chemical Reaction Extraction From Complex Scientific Texts." Scienmag. November 17, 2026. https://scienmag.com/multimodal-framework-improves-chemical-reaction-extraction-from-complex-scientific-texts/








