AI Framework Weaves Reports, Images and Old Maps into Conflict Knowledge Graphs

Conflict zones generate an overwhelming torrent of information, and it rarely arrives in a single, tidy format. A humanitarian situation report may describe a clash in prose, a news photograph may capture armored vehicles on a road, and an archived scanned map may chart the terrain around a contested town. For analysts trying to reconstruct who did what, where, and when, these fragments live in separate silos: text databases, image libraries, and map repositories that do not speak to one another. A new study published in Scientific Reports proposes an integrated computational framework that pulls these heterogeneous sources together into a single multimodal spatiotemporal knowledge graph, in which entities, events, geographic locations, temporal attributes, and multimedia evidence are represented as nodes and edges in one connected structure.

The research, conducted by Li Guo, Wanchen Zhang, Nan Jia, and Caixuan Wang of Information Engineering University in Zhengzhou, China, addresses a long-standing bottleneck in event-oriented information extraction. Textual reports contain rich narrative detail but no explicit coordinates. Web images show visual scenes but no machine-readable references to place. Scanned historical maps encode precise geographic extents, yet their coordinate annotations are locked inside pixels that standard text-processing pipelines cannot touch. The team’s framework chains together textual entity, relation, and event extraction; visual object detection; scanned-map coordinate recognition; geographic entity grounding; cross-modal entity fusion; provenance-aware storage; and graph visualization, so that each modality contributes its own strengths to a unified whole.

One of the most technically demanding pieces of the pipeline is the recovery of geographic information from scanned maps. Old or digitized maps often carry coordinate annotations along their margins, but these numeric labels sit amid decorative cartographic elements, gridlines, and noise. The researchers designed a cascaded procedure that pairs two established open-source tools: YOLOv5, a convolutional neural network for object detection, and PaddleOCR, a text recognition engine. In the first stage, YOLOv5 detects and crops the coordinate annotation regions directly from the scanned image. Only then does PaddleOCR read the numbers within those focused crops, rather than attempting to transcribe the entire cluttered map sheet. The recognized coordinates are subsequently used to reconstruct the geographic extent of the map, effectively giving the digital archive a machine-readable bounding box in the real world.

The quantitative gains from this cascade are striking. On a dataset of 287 scanned maps, YOLOv5 achieved a region-detection F1-score of 0.977, indicating that the detection stage rarely misses a coordinate block. In a paired evaluation across 278 maps using an out-of-fold protocol, the proportion of maps whose coordinates were completely and correctly recognized rose from 64.03 percent with direct PaddleOCR alone to 87.05 percent with the cascaded procedure. Because the same maps were processed both ways, the team could apply an exact McNemar test to the paired outcomes; the improvement was statistically significant, with p less than 0.001. For anyone who has watched OCR systems flounder on dense cartographic layouts, the lesson is clear: detecting where to look before reading what is there pays substantial dividends.

Textual processing forms the narrative backbone of the framework. Entity, relation, and event extraction models applied to conflict-event reports achieved F1-scores between 0.76 and 0.84 across the evaluated tasks, meaning the system can pull out actors, actions, and locations from prose with reasonably high precision and recall. Extracted place entities are then geocoded against GeoNames, the widely used open geographic database, converting place names into latitude-longitude anchors. The crucial connection to maps happens next: textual place entities are linked to relevant scanned maps through point-polygon spatial intersection. If a geocoded location falls inside the reconstructed extent of a map, the report and the map become associated in the graph, allowing an analyst who reads about a village attack to jump straight to the cartographic coverage of that area.

Visual evidence enters the pipeline through modern object detection. The team evaluated YOLOv8 on three categories of web imagery and reported mean average precision at an intersection-over-union threshold of 0.5 — mAP@0.5 — of 0.935 for aircraft, 0.890 for ships, and 0.922 for general web-image detection. These are demanding visual recognition problems: aircraft appear at varying scales and angles in satellite or aerial imagery, and ships occupy few pixels against reflective water. The high scores suggest that the detection components are mature enough to feed reliable visual nodes into the graph, each image contributing detected objects that can be associated with events, times, and places.

Perhaps the most consequential step is cross-modal entity fusion, the process of deciding that a place name in a report, a detected object in a photograph, and a coordinate range on a map all refer to the same real-world thing. Naive matching fails badly in conflict contexts, where place names repeat across regions and abbreviations abound. The framework therefore applies a constrained fusion method, and the evaluation design matters as much as the method itself: the team used an event- and source-document-grouped protocol, which prevents information leakage between training and test cases that share events or documents. Under this strict protocol, the complete constrained cross-modal entity-fusion method achieved an F1-score of 0.884, a strong result for a task that spans modalities as different as narrative text and raster graphics.

Provenance-aware storage is the quiet innovation that makes the resulting graph trustworthy. Every extracted fact, whether it came from a sentence, a detected object, or a recognized coordinate label, carries a record of its source document and extraction method. This matters in conflict analysis, where contradictory reports are common and the credibility of a claim often depends on where it originated. By storing provenance alongside the fused entities and events, the graph does not flatten disagreement into a single answer; it preserves the evidentiary chain so that analysts can audit why the system linked a photograph to a place or an event to a date. Graph visualization then renders the structure interactively, letting users traverse from an event node to its actors, locations, timestamps, source texts, images, and associated maps.

To demonstrate practical applicability, the authors include a conflict-event case study showing the full pipeline in action: reports are mined for entities and events, web images are scanned for relevant objects, scanned maps are georeferenced through the cascaded OCR procedure, and the resulting graph ties narrative, visual, and cartographic evidence together for a real analysis scenario. The case study positions the framework as more than a collection of benchmark scores; it is an end-to-end system aimed at the daily workflow of analysts who currently juggle disconnected tools.

The broader significance of this work lies in how it treats multimodality as a first-class requirement rather than an afterthought. Knowledge graphs have become a standard substrate for intelligence and crisis informatics, but most operational systems still ingest text primarily, with images and maps handled as attachments rather than integrated knowledge. By demonstrating statistically validated improvements on the hardest step — reading coordinates off scanned maps — and a rigorous, leakage-resistant evaluation of cross-modal fusion, the Zhengzhou team has provided both a reusable method and an honest yardstick for future systems. As archives of scanned maps and multimedia reporting continue to grow, frameworks of this kind may determine how quickly raw, scattered evidence becomes structured, queryable knowledge about the world’s most urgent events.

Subject of Research: Multimodal spatiotemporal knowledge graph construction for conflict-event analysis

Article Title: Multimodal spatiotemporal knowledge graph construction for conflict-event analysis: an integrated framework with geographic entity-map association

Article References: Guo, L., Zhang, W., Jia, N., & Wang, C. (2026). Multimodal spatiotemporal knowledge graph construction for conflict-event analysis: an integrated framework with geographic entity-map association. Scientific Reports. https://doi.org/10.1038/s41598-026-73358-1

Image Credits: AI Generated

DOI: 10.1038/s41598-026-73358-1

Keywords: knowledge graph, multimodal, spatiotemporal, conflict events, geographic grounding, entity fusion, OCR, YOLOv5, YOLOv8, PaddleOCR, geocoding, information extraction