The system produced a weighted F1 score of 0.940, compared with 0.920 for a directly evaluated learnable-weight baseline, although that difference did not reach statistical significance in the reported paired test (p = 0.0625). The experiment used MUStARD, a public dataset of synchronized video, speech and transcripts extracted from television sitcoms including Friends, The Big Bang Theory and The Golden Girls. Each sample is associated with an utterance labeled as sarcastic or non-sarcastic, with preceding dialogue retained as context.

The 690 samples are evenly divided between the labels, comprising 345 clips in each class. Utterances were split at the sample level into 414 training examples, 138 validation examples and 138 test examples, with stratified sampling intended to preserve class balance. The researchers reported three runs with different random seeds and presented the mean and standard deviation for the principal metrics.

This was a benchmark study, not a clinical trial or a field deployment of a sarcasm-detection system. Sarcasm was treated as a cross-channel inference problem because a transcript may omit irony carried by a voice, while a facial expression or change in intonation can alter the apparent meaning of ordinary words. The text branch used RoBERTa-Base to produce contextual embeddings for the target utterance and its conversational history.

Preprocessing retained negation, intensifiers and emphasis markers because they can change pragmatic meaning, while informal forms such as “gonna” and “wanna” were normalized and inputs were tokenized with byte-pair encoding. The final hidden representation came from the model’s classification token and entered a binary sarcasm classifier. On its own, this branch achieved 0.82 accuracy, with recall and precision both reported as 0.82 and a weighted F1 score of 0.82.

Sound supplied a different view of the same utterance. The audio pipeline converted speech to waveform files, resampled them to 16 kHz, normalized amplitude, filtered background noise, removed non-speech intervals with a voice-activity detector and standardized utterance duration through truncation or zero-padding. A pretrained HuBERT-Base checkpoint converted raw waveforms into frame-level features with a convolutional encoder, then used stacked self-attention layers to model contextual acoustic patterns.

Lower layers were frozen while upper layers were fine-tuned, and mean pooling produced a 768-dimensional utterance representation. The resulting audio-only model reached 0.88 accuracy, recall and precision, with a weighted F1 score of 0.88. Its class-level errors were nearly balanced: the confusion matrix contained 63 correct non-sarcastic decisions and 59 correct sarcastic decisions, alongside eight errors in each direction.

The visual branch used ViViT-Base to model the timing as well as the appearance of facial and gestural cues. Video was sampled at 25 frames per second, with faces detected and cropped to 224-by-224-pixel regions. Pixel values were normalized, histogram equalization reduced some illumination differences, and augmentation included horizontal flipping, rotation and brightness adjustment.

Sixteen-frame clips were supplied to the transformer, whose patch embeddings and self-attention captured spatial relationships within frames and temporal dependencies across frames. This model alone obtained 0.90 accuracy and a weighted F1 score of 0.89, making it the strongest unimodal result in the study. It was also conservative about labeling utterances as sarcastic: 61 non-sarcastic and 63 sarcastic examples were correctly classified, with seven false positives and seven false negatives.

The central technical contribution was a compact, derivative-free search over the relative contribution of the three representations. Each particle in the PSO population represented a candidate vector containing text, audio and video weights. The weights were constrained to be non-negative and to sum to one.

Particles moved through the search space using personal and global best positions, and their positions were renormalized at each update. Candidate vectors were evaluated on the validation set, after which the best configuration was applied globally to the fused feature representation. The implementation used 30 particles and allowed up to 50 iterations, with convergence reported around 30 to 40 iterations.

The encoder parameters were held fixed during this stage, so the optimization changed three modality weights rather than the millions of parameters inside the three pretrained transformers. The method is feature-level rather than decision-level fusion. Modality embeddings are projected into a shared feature space and combined as a weighted sum before the final classifier makes a sarcastic or non-sarcastic prediction; individual class probabilities are not averaged or voted together.

It also differs from sample-adaptive reliability weighting because the optimized weights are learned globally from the validation set and remain fixed for all test samples and subsequent inference. The authors describe the approach as model-agnostic because the fusion module can be attached to different pretrained encoders without changing their internal architecture, but its reported benefits were demonstrated only with RoBERTa, HuBERT and ViViT on MUStARD. Controlled fusion experiments showed that the optimization procedure contributed to the gain.

With the same encoders and experimental protocol, simple concatenation produced a weighted F1 score of 0.88, equal-weight fusion produced 0.90 and learnable weighted fusion produced 0.92, while PSO-based fusion reached 0.94. The reported difference from the directly evaluated learnable baseline was two percentage points, but the paired statistical comparison did not pass the study’s 0.05 significance threshold. Ablation results also showed that the three channels were complementary.

Audio plus video was the strongest two-channel combination, with accuracy around 91.1% to 91.5% across the reported summaries, while audio plus text reached about 83.5% and text plus video about 84.5%. The full audio, text and video model reached approximately 94.4% in the modality comparison and 94.0% in the main evaluation. The results must be interpreted within the benchmark’s limitations.

Although the paper initially describes a speaker-independent protocol, the data were ultimately partitioned at the utterance level, so speakers and dialogue scenes could recur across the training, validation and test sets. The authors explicitly acknowledge that complete speaker- and dialogue-independent generalization was not guaranteed. The study did not quantify parameter count, floating-point operations, peak memory, training time or inference latency, making claims of real-time efficiency or lightweight deployment unverified.

Robustness tests nevertheless showed a gradual performance decline as modalities were omitted or corrupted, with accuracy ranging from 0.878 under 30% text corruption to 0.940 with intact inputs, underscoring the continued value of complementary evidence. Among the eight final test errors, the researchers classified three as context-dependent sarcasm, two as subtle or weak visual cues, two as cross-modal inconsistency and one as weak acoustic or prosodic cues. These categories identify the remaining challenge: some sarcastic intentions are not expressed through any conspicuous channel and must be inferred from conversational history, shared knowledge or culturally specific references.

The study supports a clear but bounded conclusion: on one balanced benchmark of television dialogues, explicitly optimizing the contribution of text, speech and video produced the strongest reported performance, with 94.0% accuracy and a 0.940 weighted F1 score. The evidence does not establish that the same gain will occur in larger, more diverse or fully speaker-independent datasets, nor does it show that the advantage over learnable weighting is statistically conclusive. Within the tested setting, however, the results show that sarcasm recognition benefits from combining complementary modalities and that the method of weighting them is itself an important experimental variable.

Subject of Research: Multimodal sarcasm detection using transformer encoders and Particle Swarm Optimization-based feature fusion

Article Title: Enhanced multimodal transformer framework for sarcasm detection with particle swarm optimization based feature fusion

Article References: Payala, V. K., Jeet, A. A., & Abinaya, S. (2026). Enhanced multimodal transformer framework for sarcasm detection with particle swarm optimization based feature fusion. Frontiers in Artificial Intelligence, 9, Article 1947291. https://doi.org/10.3389/frai.2026.1947291

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1947291

Keywords: sarcasm detection, multimodal learning, Particle Swarm Optimization, RoBERTa, HuBERT, ViViT, feature fusion, MUStARD dataset, transformer models, conversational AI, Enhanced, multimodal

A new study published in Information Systems Frontiers by Jasleen Kaur and Amarjit Gill of the Edwards School of Business at the University of Saskatchewan tackles this problem for Punjabi, a language spoken by well over a hundred million people yet strikingly underrepresented in sarcasm research. The work is significant for two reasons. First, it introduces PunjSarc, a new labelled dataset of sarcastic and non-sarcastic Punjabi text drawn from blogs, news websites and social media platforms. Second, it proposes a hybrid deep learning architecture, Hybrid mT5-LSTM, that combines a multilingual transformer encoder with a recurrent sequence model, and demonstrates that this pairing outperforms existing benchmark methods for sarcasm detection in the language.

The researchers assembled 5,427 instances of Punjabi text, each labelled as either sarcastic or non-sarcastic, creating what they describe as the first substantial resource of its kind for the language. Because sarcastic examples are inherently rarer than straightforward statements in most collections, the dataset suffered from class imbalance, a well-known pitfall that biases classifiers toward the majority class. To counter this, the team applied a back-translation-based data augmentation strategy, in which sentences are machine-translated into another language and back again, producing paraphrases that preserve meaning while adding linguistic variety. This process expanded the dataset to 5,775 instances and allowed the authors to rigorously test whether augmentation actually helps or hurts downstream classification.

The technical pipeline draws on three families of textual features. The simplest is the bag-of-words representation, in which a document is reduced to the multiset of words it contains, weighted either by raw term frequency or by term frequency-inverse document frequency, a scheme that down-weights common words and highlights distinctive ones. These sparse representations have long been the workhorses of text classification because they are fast and interpretable. The second family consists of dense word embeddings, continuous vectors that place semantically similar words near each other in a learned geometric space. The third family is the transformer encoder at the heart of the hybrid model, which processes entire sequences of tokens at once and captures long-range dependencies that older models miss.

mT5, the multilingual variant of the T5 text-to-text transformer, was pretrained on a vast multilingual corpus and therefore carries useful linguistic knowledge about low-resource languages such as Punjabi. On its own, however, a transformer produces contextualized token representations that still need to be distilled into a single decision. The authors addressed this by feeding the transformer’s output into a long short-term memory network, a recurrent architecture introduced in 1997 that uses gated memory cells to decide what information to retain or discard as it moves through a sequence. The LSTM layer, in effect, reads the transformer’s contextual representations sequentially, capturing directional patterns in how sarcastic cues unfold within a sentence before a final classification layer renders its verdict.

To contextualize the hybrid model’s performance, the study ran four families of experiments on both the original and the augmented datasets: classical baseline machine learning models such as decision trees and support vector machines, ensemble learning methods, standalone deep learning models including LSTM and bidirectional LSTM, and the proposed hybrid architecture. The results on augmentation were notably mixed. Back-translation augmentation produced marginal improvements for the deep learning sequence models, which benefited from the additional training examples, but it slightly degraded performance for decision trees and, more surprisingly, for the hybrid model itself. This finding is a valuable caution for practitioners: augmentation is not a free lunch, and its effects depend heavily on how a model represents and generalizes from text.

The headline result concerns the hybrid model trained on the original, unaugmented data. Hybrid mT5-LSTM achieved 94.29 percent accuracy in distinguishing sarcastic from non-sarcastic Punjabi text, surpassing all benchmark models tested in the study and setting a new state of the art for the language. Because a single accuracy figure can be misleading, the authors validated the superiority of their model using a statistical t-test, providing formal evidence that the performance gain is not an artifact of random variation across runs. For a language that until now had virtually no dedicated sarcasm detection research, the jump is a substantial leap forward.

The broader significance of the work extends well beyond the leaderboard. Sarcasm fundamentally distorts sentiment analysis: a sarcastic review that reads as praise on the surface often encodes deep dissatisfaction underneath. Prior research has shown that sentiment classifiers that ignore sarcasm misclassify a meaningful fraction of opinionated text, and marketing scholars have documented how media sentiment shapes firm sales growth and how social media has become central to consumer behaviour and new product development. Accurate sarcasm detection therefore functions as a quality-control layer for consumer analytics, and the authors argue that business owners who deploy it can uncover hidden customer dissatisfaction that would otherwise remain invisible, enabling tailored recommendations that strengthen satisfaction and loyalty.

There is also a cultural and linguistic equity dimension. Most sarcasm detection systems have been built for English, with growing bodies of work on Arabic, Chinese, Hindi, Bengali, Kannada, Telugu, Tamil and Indonesian. Punjabi, despite its enormous speaker base and vibrant social media presence, has been left largely on the sidelines, in part because building labelled datasets is expensive and requires native-speaker annotation. The release of the PunjSarc dataset, together with publicly available code, lowers the barrier for other researchers and sets the groundwork for future academic work on figurative language in Punjabi, including irony, humour and code-mixed text that blends Punjabi with English.

The study also contributes a nuanced empirical lesson about the interaction between data augmentation and model architecture. The finding that transformer-LSTM hybrids perform best on unmodified data while simpler recurrent models gain from augmentation suggests that the capacity of a model mediates how much it benefits from additional paraphrased examples. High-capacity pretrained encoders may already encode the variability that augmentation injects, making synthetic data redundant or even noise-inducing for such models, whereas weaker learners gain genuine signal from the expanded sample. As organizations increasingly fine-tune large multilingual models for niche languages and specialized sentiment tasks, this interaction between augmentation strategy and architecture choice is exactly the kind of practical knowledge that determines whether a deployed system performs as promised.

For the growing community of researchers in computational linguistics and information systems, the message of this research is twofold. Sarcasm detection in low-resource languages is both tractable and commercially valuable, and the path forward runs through careful dataset construction, honest ablation of techniques such as augmentation, and hybrid architectures that marry pretrained multilingual knowledge with sequence-aware classification. Kaur and Gill’s hybrid model, validated statistically and benchmarked against a broad set of machine learning, ensemble and deep learning competitors, offers a template that can be adapted to other underserved languages. As social media conversation shifts decisively toward regional languages, tools that can read between the lines will become essential instruments for understanding what consumers everywhere are really saying.

Subject of Research: Sarcasm detection in Punjabi social media text using a hybrid mT5-LSTM deep learning model.

Article Title: From Sarcasm to Consumer Insight: Leveraging mT5-LSTM for Analyzing Punjabi Social Media Behavior

Article References: Kaur, J., & Gill, A. (2026). From Sarcasm to Consumer Insight: Leveraging mT5-LSTM for Analyzing Punjabi Social Media Behavior. Information Systems Frontiers. https://doi.org/10.1007/s10796-026-10812-5

Image Credits: AI Generated

DOI: 10.1007/s10796-026-10812-5

Keywords: sarcasm detection, Punjabi, natural language processing, deep learning, mT5, LSTM, data augmentation, sentiment analysis, consumer insight, machine learning, transformer-based learning, social media analytics