The research, conducted by Jiaxu Li and Wei Liu of the School of Information and Electronic Engineering at Shandong Technology and Business University in Yantai, China, introduces a model called VMMD, short for Variational Autoencoder-Based Multi-Granularity Missing-Aware Dual-Path Fusion Network. Published on 17 September 2026, the work addresses what the authors describe as the dual challenges of accuracy and robustness in complex scenarios, where incomplete modality information degrades performance and undermines sentiment polarity inference.

The scale of the problem becomes clear when considering how these systems are deployed. Emotion-aware interfaces for virtual assistants, mental health screening tools and human-computer interaction platforms all rely on fusing information from multiple sensors. Earlier approaches to missing modality problems have included generative adversarial networks that attempt to impute missing views, modality translation techniques that map one channel onto another, and transformer-based feature reconstruction networks. Yet each strategy carries trade-offs: generative imputation can hallucinate plausible-looking but emotionally misleading features, while translation methods depend heavily on statistical correlations that may not hold across speakers or recording conditions.

At the heart of VMMD lies a component the authors call the Modal-Token Missing-Aware Module, or MT-MAM. This module generates what the researchers describe as robust proxy features—stand-in representations that can substitute for missing data. It does so by integrating core semantic information from each modality, guided by two distinct levels of modeling. At the modal level, an uncertainty weighting scheme assesses how reliable each modality is in a given instance, drawing on the principles of variational autoencoders, which encode data as probability distributions rather than fixed points and can therefore express how confident the model is about what it has learned. At the token level, the module applies missing-aware modeling of the dominant modality, typically the text stream, which in most sentiment benchmarks carries the strongest and most consistent emotional signal.

This dual granularity matters because emotions are expressed at different scales simultaneously. A single word can flip the polarity of an entire sentence, while a sustained facial expression shapes the interpretation of a whole video clip. By weighting whole modalities according to their uncertainty and then modeling token-level absence within the dominant modality, MT-MAM produces proxy features that preserve the core semantics needed for sentiment discrimination even when, for example, the video channel is completely unavailable. The variational autoencoder backbone allows the network to treat missingness as an uncertain condition to be reasoned about probabilistically, rather than a hard failure to be patched over with zeros or averages.

Proxy features alone, however, are not enough. The authors note that single proxy representations have limitations in sentiment discrimination, precisely because they compress an entire modality into a compact stand-in. To compensate, VMMD incorporates a second path built on the Transformer architecture, the same attention-driven design that underpins modern large language models. This second path integrates semantic clues from the non-dominant modalities—audio prosody, visual expressions, and any residual tokens that survived data loss—and supplies supplementary evidence to the fusion process. The dual-path design therefore runs two complementary streams of information in parallel: one robust and condensed, one rich and detailed, with each compensating for the other’s blind spots.

Running two paths creates its own subtle hazard, and it is here that the paper’s third innovation enters. Because the proxy features and the supplementary information are built at different granularities, their semantic representations can drift apart, becoming inconsistent in ways that confuse the final classifier. To address this, the researchers introduce the Cross-Path Semantic Consistency Constraint, or CSCC, a mechanism driven by contrastive learning. Contrastive learning, popularized in frameworks such as supervised contrastive loss, trains a model by pulling semantically similar representations closer together in an embedding space while pushing dissimilar ones apart. CSCC applies this idea across the two paths of the network, explicitly aligning the heterogeneous representations so that the proxy features and the supplementary clues tell a coherent story about the same emotional content.

The authors report that experiments on multiple datasets demonstrate the effectiveness and robustness of the proposed model. The experimental work, described in the paper’s contribution statement, included simulations of modality missing, data preprocessing on the widely used CMU-MOSI and CMU-MOSEI English benchmark corpora and the Chinese CH-SIMS dataset, model training and comprehensive result analysis. These benchmarks are the standard proving grounds for the field: CMU-MOSI contains short monologue video clips rated for sentiment intensity, CMU-MOSEI extends the format to thousands of speakers in the wild, and CH-SIMS provides fine-grained annotations of each modality separately, allowing researchers to test how well models handle disagreement between what is said, how it is said and how the speaker looks.

The broader significance of the work lies in a shift of philosophy. Rather than treating missing data as an anomaly to be repaired, VMMD treats uncertainty as a first-class property of the input, to be weighted, modeled and constrained throughout the fusion process. This mirrors a wider trend in machine learning toward reliable representation learning for incomplete multi-view data, an area that has seen growing attention as multimodal systems move out of the laboratory and into noisy deployment environments such as mobile devices, video conferencing platforms and social media monitoring.

For the field of affective computing, the implications extend to any application where machines must understand people through imperfect sensors. Depression screening tools that combine speech and facial analysis, sentiment monitoring of customer service calls, and social robotics all face the reality that modalities drop out unpredictably. An architecture that maintains accurate sentiment polarity inference under uncertain missingness, while explicitly reconciling the semantic tension between different levels of representation, offers a template for building systems that degrade gracefully rather than catastrophically. The research was supported by Shandong Technology and Business University, and the authors state that the data underlying the study will be made available on request.

Subject of Research: Robust multimodal sentiment analysis under uncertain missing modalities using a dual-path fusion network

Article Title: Dual-path fusion network for multimodal sentiment analysis under uncertain missing modalities

Article References: Li, J., & Liu, W. (2026). Dual-path fusion network for multimodal sentiment analysis under uncertain missing modalities. International Journal of Machine Learning and Cybernetics, 17(10), Article 468. https://doi.org/10.1007/s13042-026-03312-0

Image Credits: AI Generated

DOI: 10.1007/s13042-026-03312-0

Keywords: multimodal sentiment analysis, missing modality, variational autoencoder, contrastive learning, Transformer, cross-modal fusion, deep learning, affective computing, robustness, CMU-MOSI, CMU-MOSEI, CH-SIMS