The research, conducted by Lalruatkimi and Lenin Laitonjam of the National Institute of Technology Mizoram, evaluates multimodal fusion strategies for offensive meme classification across two widely used benchmark datasets: MultiOFF, a collection of offensive political memes, and the Facebook Hateful Meme dataset, which was designed specifically to contain examples where hate speech emerges only from the interaction between image and text. Rather than proposing a single new architecture, the authors treat fusion itself as the object of study, asking which way of merging image and text representations works best, and under what conditions.

The study examines three broad families of fusion. Early fusion combines features from the image and the text before classification begins, merging them into a single representation that a classifier then processes. Late fusion keeps the two modalities separate throughout processing, allowing dedicated image and text models to make their own assessments before their outputs are combined, in some cases through mechanisms such as fuzzy logic or correlation maximization that weigh how much each modality should contribute. Hybrid fusion blends elements of both, integrating intermediate representations while still preserving modality-specific decision paths.

To test these strategies, the researchers paired them with a range of underlying architectures. On the conventional side, they used Bi-LSTM and Stacked LSTM networks, which process text sequences in both directions or through layered recurrence, alongside convolutional neural networks for text and the VGG16 architecture for images. On the transformer side, they combined BERT and RoBERTa, two influential text encoders, with Vision Transformers, which apply the attention mechanism that revolutionized language modeling to images. They also tested CLIP, a model trained from the start on paired images and texts and therefore naturally attuned to cross-modal relationships.

The results reveal that fusion performance is highly dependent on both the chosen architecture and the chosen fusion strategy, with no single combination dominating across all settings. Among the conventional models, the strongest result on the MultiOFF dataset came from pairing a CNN text encoder with VGG16 for images, using summation-based early fusion and fuzzy logic-based late fusion, which achieved an F1 score of 0.6727. This figure underscores how challenging the task remains even for well-established deep learning components: the offensive meaning of a meme often hinges on subtle cultural or political context that neither modality exposes on its own.

Transformer-based approaches pushed performance further. The best result reported in the study came from combining RoBERTa for text with a Vision Transformer for images and applying correlation maximization in the late fusion stage, which reached a weighted F1 score of 0.75 on the Facebook Hateful Meme dataset. The authors attribute this advantage to the richer cross-modal relationships that transformer representations capture. Because both BERT-style text encoders and Vision Transformers are built on attention mechanisms, their internal representations are more compatible, allowing the fusion stage to align visual and textual signals that refer to the same entities, actions, or implied narratives.

The choice of dataset mattered as much as the choice of model. The Facebook Hateful Meme dataset was constructed so that neither the image nor the caption alone is sufficient to identify hate speech, making it a deliberately adversarial test of multimodal understanding. MultiOFF, focused on offensive political memes, presents a different distribution of content and a different balance of difficulty. The finding that the same fusion technique can excel on one dataset and underperform on another suggests that practitioners deploying moderation systems cannot simply adopt a universal recipe; they must validate fusion choices against the specific content distribution they face.

Beyond the headline numbers, the study carries practical implications for platforms grappling with harmful content at scale. Late fusion strategies that adaptively weight each modality, such as the fuzzy logic and correlation maximization approaches tested here, offer a form of robustness: when one modality is ambiguous or uninformative, the system can lean more heavily on the other. This flexibility is valuable in real-world settings, where memes vary enormously in how much of their meaning is carried by the image versus the caption. The public availability of both benchmark datasets, and of the authors’ custom data split for the Facebook Hateful Meme dataset, also supports reproducibility and further comparison by other research groups.

The work sits within a rapidly growing literature on multimodal machine learning, spanning hate speech detection, sarcasm and misogyny identification, sentiment analysis, and even medical imaging, where fusing information from multiple sources has proven similarly consequential. By isolating fusion strategy as a variable and testing it systematically across architectures and datasets, the study provides a kind of empirical map for researchers and engineers deciding how to build multimodal classifiers. Its central lesson is twofold: transformer-based representations meaningfully improve the understanding of how image and text interact, and the fusion strategy must be matched carefully to both the architecture and the task. As memes continue to evolve as a medium of online expression, the systems that police them will need exactly this kind of nuanced, evidence-based understanding of how machines can learn to read pictures and words together.

Subject of Research: Multimodal fusion techniques for automated classification of offensive internet memes

Article Title: Empirical analysis of multimodal fusion techniques for offensive meme classification

Article References: Lalruatkimi, & Laitonjam, L. (2026). Empirical analysis of multimodal fusion techniques for offensive meme classification. Multimedia Tools and Applications, 85(10), Article 789. https://doi.org/10.1007/s11042-026-21935-x

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21935-x

Keywords: meme classification, multimodal fusion, hate speech detection, transformers, BERT, RoBERTa, Vision Transformer, CLIP, early fusion, late fusion, content moderation, deep learning