Friday, September 5, 2025
Science
No Result
View All Result
  • Login
  • HOME
  • SCIENCE NEWS
  • CONTACT US
  • HOME
  • SCIENCE NEWS
  • CONTACT US
No Result
View All Result
Scienmag
No Result
View All Result
Home Science News Social Science

New AI framework enhances emotion analysis

June 26, 2024
in Social Science
Reading Time: 3 mins read
0
Overall architecture of the two-stage stacked transformer framework
66
SHARES
597
VIEWS
Share on FacebookShare on Twitter
ADVERTISEMENT

Social media enthusiasts tend to spice up their text posts with emojis, images, audio, or video to attract more attention. Simple as it is, this technique makes scientific sense: multimodal information is found to be more effective in conveying emotions, as different modalities interact and enhance one another. To advance the understanding of these interactions and improve the analysis of emotions expressed through modality combinations, a Chinese research team introduced a novel two-stage framework using two stacked layers of transformers, state-of-the-art AI models for multimodal sentiment analysis. This study was published May 24 in Intelligent Computing, a Science Partner Journal.

Overall architecture of the two-stage stacked transformer framework

Credit: GUOFENG YI ET AL.

Social media enthusiasts tend to spice up their text posts with emojis, images, audio, or video to attract more attention. Simple as it is, this technique makes scientific sense: multimodal information is found to be more effective in conveying emotions, as different modalities interact and enhance one another. To advance the understanding of these interactions and improve the analysis of emotions expressed through modality combinations, a Chinese research team introduced a novel two-stage framework using two stacked layers of transformers, state-of-the-art AI models for multimodal sentiment analysis. This study was published May 24 in Intelligent Computing, a Science Partner Journal.

Current research in multimodal sentiment analysis often focuses on either fusing different modalities or addressing intricate interactions or adaptations between different types of fused information. Either approach alone can lead to information loss. This team’s framework, on the other hand, fuses information in two stages to effectively capture information on both levels. It was tested on three open datasets—MOSI, MOSEI, and SIMS—and performed better than or as well as the benchmark models.

The general workflow of this framework includes feature extraction, two stages of information fusion, and emotion prediction. First, text, audio, and video signals taken from source video clips are processed through their corresponding feature extractors and then encoded with additional context information into context-aware representations. Next, the three types of representations fuse for the first time: the text representations interact with the audio and video representations, allowing each modality to adapt to the others during the process, and the results further integrate with the original text representations. The text-centered output from the first stage then fuses with the adapted non-text representations so that they can enhance each other before the final, enriched output is ready for the emotion prediction stage.

The core of the team’s framework is stacked transformers and consists of bidirectional cross-modal transformers and a transformer encoder. These components correspond to two functional layers: the bidirectional interaction layer allows cross-modal interaction and is where the first-stage fusion occurs, and the refine layer addresses the more nuanced second-stage fusion.

To enhance the performance of the framework, the team implemented an attention weight accumulation mechanism that aggregates the attention weights of the text and non-text modalities during fusion to extract deeper shared information. Attention, a key concept in transformers, enables the model to identify and focus on the most relevant parts of the data. The team’s stacked transformers adopt two types of attention mechanism: the bidirectional cross-modal transformers use cross-attention, and the transformer encoder uses self-attention.

The future work of the team will focus on integrating more advanced transformers to improve computational efficiency and mitigate the inherent challenges associated with the self-attention mechanism.



Journal

Intelligent Computing

DOI

10.34133/icomputing.0081

Article Title

A Two-Stage Stacked Transformer Framework for Multimodal Sentiment Analysis

Article Publication Date

24-May-2024

Share26Tweet17
Previous Post

A chip-scale Titanium-sapphire laser

Next Post

Targeting human coatomer subunit zeta-1 by a marine natural product derivative represents a new therapeutic strategy to inhibit enterovirus A71 replication

Related Posts

blank
Social Science

Digital Cognitive Twins Transform Mental Health Care

September 5, 2025
blank
Social Science

Exploring Well-Being in Vocational vs. General Students

September 4, 2025
blank
Social Science

Study Reveals Soothing Methods Matter More Than Speed, Highlighting Cultural Differences in Parenting

September 4, 2025
blank
Social Science

Exploring Teacher-Child Dynamics in Head Start Programs

September 4, 2025
blank
Social Science

Prenatal SSRI Risks and Benefits: Maternal, Child Impact

September 4, 2025
blank
Social Science

How Stress and Social Challenges Drive America’s Obesity Epidemic

September 4, 2025
Next Post
Chemoproteomics enables identification of coatomer subunit zeta-1 targeted by a small molecule for enterovirus A71 inhibition

Targeting human coatomer subunit zeta-1 by a marine natural product derivative represents a new therapeutic strategy to inhibit enterovirus A71 replication

  • Mothers who receive childcare support from maternal grandparents show more parental warmth, finds NTU Singapore study

    Mothers who receive childcare support from maternal grandparents show more parental warmth, finds NTU Singapore study

    27544 shares
    Share 11014 Tweet 6884
  • University of Seville Breaks 120-Year-Old Mystery, Revises a Key Einstein Concept

    958 shares
    Share 383 Tweet 240
  • Bee body mass, pathogens and local climate influence heat tolerance

    643 shares
    Share 257 Tweet 161
  • Researchers record first-ever images and data of a shark experiencing a boat strike

    510 shares
    Share 204 Tweet 128
  • Warm seawater speeding up melting of ‘Doomsday Glacier,’ scientists warn

    313 shares
    Share 125 Tweet 78
Science

Embark on a thrilling journey of discovery with Scienmag.com—your ultimate source for cutting-edge breakthroughs. Immerse yourself in a world where curiosity knows no limits and tomorrow’s possibilities become today’s reality!

RECENT NEWS

  • Hairy Black Holes: Scrambling Cosmic Past
  • Impact of Vegetable Waste on Labeo Rohita Growth
  • Ultra-Compact Plasmonic Nanocavity Boosts Magnetic SHG
  • Microwave-Assisted Synthesis of Biomass-Derived N-Doped Carbon Dots Advances Metal Ion Sensing Technology

Categories

  • Agriculture
  • Anthropology
  • Archaeology
  • Athmospheric
  • Biology
  • Blog
  • Bussines
  • Cancer
  • Chemistry
  • Climate
  • Earth Science
  • Marine
  • Mathematics
  • Medicine
  • Pediatry
  • Policy
  • Psychology & Psychiatry
  • Science Education
  • Social Science
  • Space
  • Technology and Engineering

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 5,183 other subscribers

© 2025 Scienmag - Science Magazine

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • HOME
  • SCIENCE NEWS
  • CONTACT US

© 2025 Scienmag - Science Magazine

Discover more from Science

Subscribe now to keep reading and get access to the full archive.

Continue reading