The rationale for fusing four modalities rests on a simple observation: each channel carries information the others cannot supply. Video shows externally observable match content such as player positions and ball trajectories. Audio conveys commentary semantics and crowd atmosphere that no visual frame contains. EEG captures internal emotional and cognitive states through alpha and beta band-power dynamics linked to arousal and attention. Eye tracking reveals where, within the wide VR field of view, the viewer’s attention is actually allocated. A video-only model cannot detect a viewer who is physically watching but mentally disengaged, while an EEG-only model cannot tie an emotional spike to a specific match event. Crucially, the authors did not treat this complementarity as an assumption; they designed a battery of validation experiments to confirm empirically that each modality encodes the specific information type attributed to it.

The dataset behind the study is substantial. Thirty adult volunteers aged 18 to 35, stratified by prior VR experience and football viewing frequency, each watched 24 sixty-second VR clips drawn from publicly available broadcasts of the UEFA Champions League, La Liga and the English Premier League. The clips covered four scenario types: goals, free kicks, corner kick attacks and defenses, and midfield ball possession. Participants wore a Meta Quest Pro headset with built-in 120 Hz eye tracking and a wireless 32-channel dry-electrode EEG cap sampling at 256 Hz. After each clip they rated presence, attentional engagement and emotional involvement on seven-point scales, and three trained evaluators normalized these scores into four equally spaced immersion levels. Segmenting each clip into overlapping one-second windows produced 86,400 multimodal analysis samples with approximately balanced class labels.

Technical synchronization posed a serious challenge because the four streams arrive at wildly different rates: video at 30 frames per second, audio at 16 kHz, EEG at 256 Hz and gaze at 120 Hz. The team solved this with a unified hardware timestamp mechanism in which a trigger is broadcast simultaneously to the headset clock and the EEG amplifier synchronization port, achieving alignment within ±5 milliseconds. Each modality then passes through a dedicated encoder: a three-dimensional spatiotemporal convolutional backbone for video, a pre-trained acoustic encoder fed log-Mel spectrograms for audio, a one-dimensional convolutional network over five EEG frequency bands for brain signals, and fully connected layers over fixation, saccade and pupil features for gaze. Linear projection layers map all four outputs into a common semantic space so that attention computations operate on dimensionally compatible and semantically comparable features.

The fusion architecture itself contains three purpose-designed innovations. First, instead of a fully connected cross-attention topology that computes all six pairwise interactions among four modalities, CMAFNet uses a hub-and-spoke design in which video acts as the semantic hub interacting bidirectionally with each auxiliary modality, cutting cross-attention parameters by 50%. The choice is domain-motivated: commentary describes visual events, EEG reactions are triggered by visual stimuli, and eye movements track visual objects, so non-visual modalities relate to each other mainly through their shared visual reference. Representational similarity analysis supports this, with video showing the highest similarity to each auxiliary modality (0.38 with audio, 0.34 with EEG, 0.36 with gaze) while similarities among the non-visual channels stay lower, between 0.17 and 0.22.

Second, a quality-aware adaptive gating mechanism adjusts each modality’s fusion weight on a per-sample basis rather than learning one fixed weight per channel. This matters because EEG quality varies dramatically across users due to cap fit and within sessions due to sweat accumulation, and stadium audio can be degraded by background noise. A lightweight gating network reads each modality’s enhanced feature vector, whose norm and cross-modal consistency implicitly encode signal quality, and outputs a weight between zero and one. When the authors injected Gaussian noise into a single modality at test time, the gating value of the corrupted channel fell monotonically as signal-to-noise ratio dropped: EEG’s mean weight fell from 0.53 to 0.19 at the worst noise level, while the clean modalities held steady or rose slightly to compensate. This is direct evidence that the gates track reliability rather than something spurious.

Third, the loss function introduces an ordinal regression objective that decomposes the four-level classification into three binary threshold decisions, preserving the fact that immersion levels are ordered rather than unordered categories. The payoff shows in the confusion patterns: misclassifications concentrate between adjacent levels, with severe errors spanning two or more levels accounting for only 1.3% of test samples. Ablation experiments quantify each component’s contribution: removing cross-modal attention costs 7.3 percentage points of accuracy, replacing adaptive gating with equal weighting costs 3.8 points, and dropping the ordinal loss costs 2.4 points of accuracy while raising the mean absolute error from 0.274 to 0.348. A controlled experiment re-training key baselines with identical encoders and preprocessing confirmed that the fusion strategy itself, not implementation details, drives the advantage.

The validation of modality-specific information is perhaps the study’s most rigorous section. Linear probes trained on frozen features showed each modality achieving its highest decoding accuracy on its attributed attribute, with aligned probing accuracies ranging from 74.2% to 88.7% and margins of at least 20.6 percentage points over the next best modality. Semantically targeted perturbations that removed only the attributed information caused degradations 4.1 to 17.4 times larger than control perturbations on the same modality. A striking qualitative case illustrates why the physiological channels matter: in one high-action counterattack clip, video motion energy and audio energy resembled a thrilling goal sequence, and a video-only variant predicted high immersion, but the viewer’s EEG showed no arousal peak and gaze lingered on the scoreboard and stands for over eleven seconds. The full model correctly predicted medium-low immersion, matching the viewer’s self-report.

Practical deployment metrics look feasible. On a single NVIDIA RTX 4090 consumer GPU, CMAFNet runs at 22.4 frames per second with 44.6 milliseconds of latency and fits within 2.8 GB of memory, comfortably within the 10 to 20 Hz update rate sufficient for adaptive broadcast decisions while leaving headroom for concurrent VR rendering. Leave-one-subject-out cross-validation across all 30 participants yielded a mean accuracy of 79.3 ± 2.1%, only 2.3 points below the standard split and above every baseline’s standard-split score, and accuracy stayed above 68% even when any single modality was entirely absent. In a separate subjective session, full-modality fusion increased presence by 32.6% and emotional involvement by 42.4% relative to a video-only baseline, with differences reaching statistical significance, while visual fluidity ratings remained essentially unchanged.

The authors are candid about limitations. The cohort of 30 tech-familiar adults aged 18 to 35 cannot represent elderly users or VR novices, the clip corpus omits long-tail scenarios such as penalty shootouts and VAR controversies, and wearing a headset plus EEG cap for more than 20 minutes strains comfort and ecological validity. The probing analysis relies partly on clip-level self-reports, and the perturbation experiments establish association rather than strict causal attribution. Deployment on standalone headsets without an external GPU would require a distilled variant not yet characterized. Still, the study demonstrates that real-time, physiologically grounded immersion assessment in VR sports broadcasting is technically achievable, opening a path from post-hoc questionnaires toward feedback loops that could dynamically shape rendering and broadcast decisions as the match unfolds.

Subject of Research: A multimodal deep learning algorithm for assessing user immersion in VR football broadcasting using fused video, audio, EEG and eye-tracking data

Article Title: Research on algorithm optimization for immersive football match broadcasting user experience assessment based on VR multimodal data fusion

Article References: Shen, W., & Xia, M. (2026). Research on algorithm optimization for immersive football match broadcasting user experience assessment based on VR multimodal data fusion. Discover Artificial Intelligence, 6(1), Article 1641. https://doi.org/10.1007/s44163-026-02511-6

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02511-6

Keywords: virtual reality, football broadcasting, multimodal fusion, EEG, eye tracking, cross-modal attention, immersion assessment, deep learning, user experience, ordinal regression, adaptive gating, real-time inference