AI Learns to Balance Text and Images to Fill Gaps in Knowledge Graphs

Knowledge graphs sit quietly beneath many of the technologies people use every day. When a search engine answers a question directly, when a voice assistant explains who wrote a novel, or when a recommendation system connects a film to its director, a vast network of entities and relationships is being queried behind the scenes. These graphs store facts as triples: a head entity, a relation, and a tail entity, such as Paris, capital of, France. Yet even the largest and most carefully curated graphs are riddled with gaps. Facts are missing, connections are incomplete, and entire regions of the network remain uncharted. The task of predicting those missing triples, known as knowledge graph completion, has become one of the central challenges in modern data mining, and a new study published in Data Mining and Knowledge Discovery proposes a fresh way to tackle it by confronting a problem that most existing methods have quietly ignored.

The research, led by Ronghua Tian and Hong Yu of the Chongqing Key Laboratory of Computational Intelligence at Chongqing University of Posts and Telecommunications, together with Yaogang Geng and Xiaoling Wang of East China Normal University, addresses multimodal knowledge graph completion, abbreviated MMKGC. Unlike classical knowledge graph completion, which relies purely on symbolic triples, multimodal completion also exploits the rich information attached to entities in other forms: images, textual descriptions, and numerical attributes. A knowledge graph describing a famous painting might carry not only the fact that it was created by a particular artist but also a digital image of the canvas and a descriptive paragraph about its style. The premise of multimodal completion is that these additional signals can help a model infer missing links that the structure alone cannot reveal.

There is, however, a catch that the authors identify as a fundamental weakness in the field. Real-world multimodal knowledge graphs are profoundly uneven in how much information they provide for each entity. Some entities are lavishly documented, with high-resolution images and lengthy descriptions, while others may have only a name and a single relation. Existing methods, the paper argues, tend to overlook this imbalance of modal information among entities. The consequence is that models develop an overreliance on the dominant modality, typically the visual one when images are plentiful or the textual one when descriptions dominate. When a model leans too heavily on one channel, the weaker channels contribute little, and for entities whose strong modality is missing or uninformative, completion performance degrades sharply. The imbalance is not a marginal nuisance; it is baked into the structure of the data itself.

To confront this problem, the researchers propose a method called MI-MKGC, which attacks modality imbalance from two complementary directions. The first component is a relation knowledge-guided dynamic modal fusion module. The key insight here is that the type of relation connecting two entities carries prior knowledge about which modality is likely to matter most. Consider a relation such as has image or looks similar to: for these, visual features are clearly decisive. For a relation like was born in or is the author of, textual and structural information carries far more weight. Rather than fusing modalities with fixed weights that apply uniformly across all triples, MI-MKGC dynamically adjusts the weight assigned to each modality depending on the relational type between the entities involved. The relation acts as a guide, telling the fusion mechanism where to look before it decides how to blend the available signals.

This dynamic fusion stands in contrast to a long line of earlier approaches. Foundational embedding models such as TransE, which translated entities through relation-specific vectors in a continuous space, and ComplEx, which extended the idea into complex-valued spaces, established the template of learning purely from graph structure. Later systems incorporated literals and images, and more recent transformer-based architectures such as MEAformer and optimal-transport-based methods like OTKGE pushed multimodal fusion further. But in most of these designs, the fusion weights are either learned globally or computed without reference to the specific relation at hand. MI-MKGC’s use of relational types as priors for modality weighting represents a targeted response to the observation that different relations draw on different kinds of evidence, and that a one-size-fits-all fusion strategy systematically shortchanges entities whose informative modality happens to be the weaker one.

The second component of the method is a joint modality adversarial training module, which addresses the imbalance from the data side rather than the fusion side. Inspired by generative adversarial networks, a framework introduced by Goodfellow and colleagues in 2014, this module generates negative samples specifically designed to enhance weak modality information. In adversarial training, a generator produces challenging examples while a discriminator learns to distinguish them from genuine data, and the two improve in tandem. Here, the generator crafts negative triples that are hard to reject, with a particular focus on strengthening the representations of modalities that are underrepresented in the training signal. By forcing the model to grapple with adversarially constructed examples that probe its weak channels, the training process gradually amplifies the contribution of those channels, reducing the dominance of the strong modality and producing more balanced entity representations.

The choice of adversarial training is grounded in a substantial body of prior work. Wasserstein generative adversarial networks and their improved variants provided more stable training dynamics for generative models, and adversarial techniques have already been applied to knowledge graph settings, for instance in KBGAN, which used adversarially generated negative samples to sharpen knowledge graph embeddings, and in GAN-based approaches to text classification and information retrieval. MI-MKGC adapts this lineage to the specific problem of modality imbalance, using the adversarial game not merely to improve discrimination between true and false triples but to deliberately rebalance the informational diet of the model across modalities.

The empirical evaluation is notably thorough. The authors tested MI-MKGC on three public multimodal knowledge graph datasets against sixteen state-of-the-art baseline methods, spanning the range from purely structural embedding models to the latest multimodal fusion architectures. The results showed consistent advantages, with the most striking gain appearing in the Hit@1 metric, which measures the proportion of cases where the model’s top-ranked prediction is exactly correct. On this metric, MI-MKGC improved by up to 5.2 percent over the best existing compared method. In a field where incremental gains of fractions of a percent are common, an improvement of that magnitude on the strictest ranking metric is a meaningful advance, and it suggests that directly modeling modality imbalance, rather than treating it as background noise, unlocks performance that fusion tweaks alone cannot reach.

The practical implications extend well beyond benchmark datasets. Multimodal knowledge graphs underpin question answering systems that combine language models with structured knowledge, recommendation engines that blend product images with user behavior, and video captioning systems that must reconcile visual and textual evidence. In all of these applications, the entities involved vary enormously in how well documented they are, and a system that silently favors well-illustrated or well-described entities will perform unevenly across the long tail of less prominent items. By dynamically reweighting modalities according to relational context and actively training up weak channels through adversarial sampling, MI-MKGC offers a blueprint for more equitable and robust multimodal reasoning, particularly for the rare entities where completion is hardest and most valuable.

The team has released its implementation code publicly on GitHub, lowering the barrier for other researchers to build on the approach. The work was supported in part by the National Natural Science Foundation of China, and the authors report no conflicts of interest. As knowledge graphs continue to expand into multimodal territory, absorbing images, text, and structured facts at scale, the problem of imbalance will only grow more pronounced, since no curation process can guarantee uniform coverage across billions of entities. What this study demonstrates is that the solution lies not in richer data alone but in algorithms that recognize, at the level of individual relations and individual entities, which signals to trust and which to strengthen. In teaching models to listen to their quietest channels, the researchers may have pointed the way toward knowledge systems that are not only more accurate but fundamentally fairer in how they represent the world.

Subject of Research: Multimodal knowledge graph completion using relation-guided fusion and adversarial training to address imbalanced modality information

Article Title: Multimodal knowledge graph completion method for addressing imbalance of modal information

Article References: Tian, R., Yu, H., Geng, Y., & Wang, X. (2026). Multimodal knowledge graph completion method for addressing imbalance of modal information. Data Mining and Knowledge Discovery, 40(6), Article 106. https://doi.org/10.1007/s10618-026-01272-1

Image Credits: AI Generated

DOI: 10.1007/s10618-026-01272-1

Keywords: knowledge graph completion, multimodal learning, modality imbalance, adversarial training, knowledge graph embeddings, generative adversarial networks, link prediction, dynamic modal fusion, negative sampling, data mining, machine learning, representation learning