Shared Transformer Brings Video, Audio and Text Together for Microvideo AI

Short-form video has become the dominant language of the internet, and the machines that must read it are struggling to keep up. A microvideo lasting only seconds can carry a spoken narration, background music, on-screen captions, scene changes and subtle visual cues all at once, and understanding any one of those streams in isolation tells only part of the story. In a study published in Neural Computing and Applications, researchers at Northeastern University in Shenyang, China, present a new architecture called the Multimodal Transformer Attention model, or MTA, which is designed to extract and fuse video, audio and text features jointly rather than treating them as separate problems stitched together at the end. The work, led by Zhaokai Zhong with Wei Zhang, Hai Yu and Zhiliang Zhu, addresses one of the most persistent bottlenecks in multimedia artificial intelligence: how to build a single representation of a clip that is genuinely informative and discriminative when the underlying data streams are heterogeneous and constantly evolving.

The core problem the team set out to solve is a familiar one in multimodal machine learning. Most existing pipelines process each modality independently, using a vision network for frames, an audio network for sound and a language model for captions or titles, and only combine the outputs in a late fusion stage. That design is convenient, but it comes at a cost. Information that only makes sense across modalities, such as the way a piece of music reinforces the mood of a scene, or the way a caption disambiguates a visually ambiguous object, can be lost before fusion ever happens. The separate encoders may also learn feature distributions that are poorly aligned with one another, so that the fusion layer receives streams that are difficult to reconcile. The result, the authors argue, is inefficient representation learning and a ceiling on classification accuracy for tasks such as microvideo category recognition.

MTA departs from that recipe by building extraction and fusion into a shared Transformer structure. Transformers, which rely on self-attention to weigh relationships between elements of a sequence, have become the backbone of modern artificial intelligence, powering everything from large language models to video vision transformers such as ViViT and multimodal self-supervised systems such as VATT. What distinguishes MTA from a conventional Transformer is what happens in its fusion layer. Instead of applying a static attention mechanism that treats all inputs uniformly, MTA incorporates a dynamic modality-aware attention mechanism. In practice, this means the model learns to adaptively attend to modality-specific information, shifting its focus depending on what a particular clip demands. For a clip in which the soundtrack carries most of the semantic signal, the audio stream can receive greater weight; for a clip driven by on-screen text or visual content, the mechanism can redirect attention accordingly. The dynamic weighting also helps align the heterogeneous feature distributions produced by different encoders, reducing the mismatch that plagues late-fusion designs.

A second conceptual innovation in the paper is the notion of categorical modality. In many multimodal classifiers, the fused representation is compared against class labels only at the very end, through a final classification layer that treats labels as inert symbols. MTA instead facilitates direct interaction between the fused multimodal features and label semantics during prediction. By giving the label side a learned semantic representation that can exchange information with the fused features, the model can exploit the relationships between categories themselves, for example when two microvideo categories share visual or textual vocabulary. The authors report that this end-to-end interaction improves learning effectiveness, because the prediction stage is no longer a passive readout but an active participant in shaping the shared representation.

The third pillar of the study is an investigation of parameter-sharing strategies. Sharing parameters across the branches of a multimodal network is a double-edged sword: it can regularize the model, reduce its size and encourage features that generalize across modalities, but excessive sharing can blur modality-specific distinctions. The researchers examined two sharing strategies within the MTA framework and found that both further boosted model performance, suggesting that a carefully chosen degree of weight sharing complements the dynamic attention mechanism rather than competing with it. This finding is practically significant for deployment, because parameter sharing reduces memory footprint and training cost, which matters when models must process the enormous volumes of short video that platforms handle daily.

Evaluation was carried out on a category-balanced benchmark dataset, a deliberate choice that removes the distorting effect of imbalanced class distributions, in which a model can appear accurate simply by favoring frequent categories. Across this benchmark, MTA consistently outperformed state-of-the-art multimodal fusion methods in classification accuracy. The authors present this as evidence of the robustness and scalability of the approach for real-world microvideo understanding tasks, and they have released their code publicly on GitHub, allowing other researchers to reproduce the results and build on the architecture.

The significance of the work becomes clearer when set against the broader landscape of multimodal research. Recent years have produced spectacular large multimodal models, from Flamingo and BLIP-3 to Llama 3 and the Qwen vision-language family, which achieve broad capabilities through massive pretraining. Yet specialized tasks such as microvideo category recognition, scene recognition and venue classification still benefit from architectures tailored to the specific structure of the problem, where clips are short, modalities are tightly interleaved and label sets are well defined. Prior work on microvideos has explored joint sequential-sparse modeling, neural multimodal cooperative learning, gated fully convolutional sequence models and attention-enhanced joint learning networks, each attacking pieces of the fusion problem. MTA’s contribution is to unify extraction and fusion within a single Transformer backbone while making the fusion itself adaptive, rather than fixed.

The applications that motivated the research are far-reaching. Content recommendation systems depend on accurate representations to surface clips that match user interests, and better multimodal understanding translates directly into more relevant feeds. Regulatory compliance is an equally pressing driver, since platforms must detect content that violates policies even when the violation is conveyed by the combination of audio and imagery rather than by either alone. Multimedia search, meanwhile, requires representations that capture what a clip is about, not merely what it looks like. A model that can weigh modalities dynamically and reason over label semantics offers a stronger foundation for all three use cases than pipelines that lose cross-modal information before fusion.

Technically, the design reflects several lessons from the Transformer literature. Self-attention allows every element of a sequence to attend to every other element, which makes it a natural fit for aligning features across modalities that unfold at different rates and granularities. Prior multimodal Transformers, such as the Multimodal Transformer for unaligned language sequences, demonstrated that cross-modal attention can bridge streams that are not temporally aligned. MTA extends this line by making the attention itself modality-aware and dynamic, so the alignment process is conditioned on the content being fused. Combined with the categorical modality mechanism, the architecture treats classification not as an afterthought but as an integrated stage of representation learning, echoing the end-to-end philosophy that has driven progress across deep learning.

The study, received in April 2025 and accepted in July 2026, was supported by the 111 Project, the Natural Science Foundation of Liaoning Province and the National Natural Science Foundation of China. Its limitations are those inherent to the field: the benchmark is category-balanced, while real-world feeds are anything but, and the reported data are available only on request due to privacy and ethical restrictions. Nevertheless, the consistent gains over existing fusion methods, the public code release and the architectural clarity of the approach position MTA as a meaningful step toward short-video AI systems that genuinely watch, listen and read at the same time. As platforms grapple with billions of daily microvideo views, architectures that fuse modalities adaptively within a shared Transformer structure are likely to shape how the next generation of recommendation, moderation and search systems makes sense of the world’s shortest, densest media.

Subject of Research: Multimodal microvideo understanding using a shared Transformer architecture with dynamic modality-aware attention

Article Title: Multimodal microvideo understanding based on the shared transformer structure

Article References: Zhong, Z., Zhang, W., Yu, H., & Zhu, Z. (2026). Multimodal microvideo understanding based on the shared transformer structure. Neural Computing and Applications, 38(19), Article 758. https://doi.org/10.1007/s00521-026-12409-0

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12409-0

Keywords: microvideo understanding, multimodal fusion, transformer, attention mechanism, deep learning, video classification, multimedia search, content recommendation, neural networks, representation learning, Neural Computing and Applications, Multimodal