The work, published in Frontiers in Neuroinformatics by YuGuo Wang of Jilin Business and Technology College and colleagues at Changchun University of Science and Technology, tackles three stubborn problems in multimodal emotion recognition. First, EEG and fNIRS do not contribute equally to every decision, so a fixed fusion rule wastes information. Second, supervised deep learning usually demands large annotated datasets, which are expensive and slow to collect for brain signals. Third, most existing pipelines pre-train each modality separately, discarding the correlations between the electrical and the hemodynamic face of the same neural event.

DSSER’s answer is a two-stage design. In the pre-training stage, each modality flows through its own branch of a Siamese architecture, the same twin-network idea that underpins modern self-supervised methods such as BYOL. Each branch contains an online network with an encoder, a projector, and a predictor, and a target network that mirrors the online network but drops the predictor. The target network’s parameters are updated as an exponential moving average of the online network’s parameters, which stabilizes training. The online network learns by predicting the target network’s representations of augmented views of the same signal, so the method needs only positive pairs and no negative samples at all, avoiding the sampling bias that plagues contrastive learning.

The encoder itself is a cascade of multi-scale one-dimensional convolutions with 64 and 128 output channels, batch normalization, ReLU activations, max pooling, and dropout of 0.3. A Squeeze-and-Excitation module after the second convolutional block adaptively re-weights feature channels, strengthening informative ones and suppressing redundancy. The outputs of two parallel convolutional branches with different temporal kernel sizes are concatenated, passed through positional encoding, and fed into a Transformer encoder with a model dimension of 128, a feedforward dimension of 32, and two attention heads, letting the model capture long-range temporal dependencies that convolutions alone would miss.

Crucially, the two streams are not trained in isolation. A co-training strategy aligns their latent representation spaces through a normalized, bidirectional cross-modal alignment loss. Before computing the alignment objective, the projected EEG and fNIRS representations are L2-normalized, so the loss measures directional discrepancy between the modalities rather than being dominated by feature magnitudes. A stop-gradient operation keeps the target representation as a stable regression target. The total pre-training loss combines the intra-modal BYOL losses for EEG and fNIRS with these alignment terms, balanced by a hyperparameter λ.

Self-supervised learning of this kind lives or dies by its data augmentations. The researchers tested six transformations of the raw signals: adding Gaussian noise, scaling amplitudes, inverting polarity, reversing the signal in time, permuting non-overlapping segments, and warping time by stretching and compressing randomly chosen segments with linear interpolation. Noise and scaling mimic realistic signal interference, while permutation and time warping introduce temporal variability. Polarity inversion and temporal reversal are stronger, physiologically less interpretable perturbations, and the authors note that future work will explore physiologically constrained augmentation methods. Experiments showed the model’s performance remained relatively consistent across all six augmentation settings.

Once pre-training is complete, the encoder from the target network is kept as the feature backbone, the projector and predictor heads are discarded, and a new fusion module and classification head are attached for supervised fine-tuning. The fusion module, called Dynamic Gated Cross-Attention Fusion (DGCAF), is the second key innovation. A gating mechanism computes global average-pooled features from each modality, passes them through learnable weight vectors, and produces sample-level weights for EEG and fNIRS. These weights dynamically decide which modality acts as the primary one and which as the auxiliary for each input sample. The primary modality’s features serve as queries while the auxiliary modality’s features serve as keys and values in a multi-head cross-attention module, so information flows between the modalities and inconsistencies and redundancy are reduced before classification.

The experimental evidence comes from a purpose-built dataset. Sixteen right-handed participants aged 23 to 27 watched twelve emotionally evocative movie clips, four each for positive, neutral, and negative categories, drawn from the validated SEED and SEED-IV stimulus sets plus additional online videos. EEG was recorded from 13 electrodes at 1,000 Hz while the LABNIRS system monitored oxyhemoglobin, deoxyhemoglobin, and total hemoglobin concentrations in the prefrontal cortex through 32 measurement channels at 27.7 Hz, using near-infrared light at 780, 805, and 830 nanometers. After careful preprocessing, including artifact correction for fNIRS with the TDDR algorithm and independent component analysis for EEG, the continuous recordings were segmented with a 5-second sliding window into 9,216 paired EEG-fNIRS samples.

Under five-fold cross-validation with a within-subject, window-level protocol, DSSER reached 98.03 percent accuracy, outperforming re-implemented supervised baselines such as DBJNet (84.80 percent on fNIRS), CNN-Transformer (86.70 percent on fNIRS), EEGNet (90.76 percent), and a plain CNN (91.31 percent), as well as self-supervised competitors including GANSER (94.53 percent), ACRNN (94.45 percent), SGMC (95.59 percent), TFSCL (96.84 percent), and MMV-SSTMA-Att (94.89 percent). The authors are careful to state that this result reflects within-subject performance on randomly split windows and should not be read as generalization to unseen participants; in a cross-subject transfer experiment with target-subject fine-tuning, average accuracy dropped to 67.76 ± 14.33 percent, underscoring that inter-subject variability remains the field’s central challenge. Ablation studies confirmed that both collaborative pre-training and the DGCAF module matter: removing either lowered accuracy, and fixed primary-modality settings consistently trailed the dynamic fusion.

Visualization with t-SNE showed that fine-tuning transforms blurry, overlapping class boundaries into compact, well-separated clusters of positive, neutral, and negative emotion. The bigger picture is that EEG and fNIRS are genuinely complementary: EEG captures fast neural oscillations with high temporal resolution, while fNIRS tracks slower hemodynamic responses with higher spatial resolution, and their fusion consistently beat either modality alone across all six augmentation scenarios. By learning modality-specific features while aligning them in a shared space, and by letting each sample decide how much weight each modality deserves, DSSER offers a template for brain-inspired affective computing that needs far fewer labels. If future work extends the dynamic gating to time-resolved modality weights and physiologically grounded augmentations, systems like this could bring emotion-aware interfaces, mental-health monitoring, and adaptive services closer to everyday reality.

Subject of Research: EEG-fNIRS self-supervised emotion recognition method based on dual-stream Siamese network

Article Title: EEG-fNIRS self-supervised emotion recognition method based on dual-stream Siamese network

Article References: Wang, Y., Wang, H., Wang, Z., & Wu, Y. (2026). EEG-fNIRS self-supervised emotion recognition method based on dual-stream Siamese network. Frontiers in Neuroinformatics, 20, Article 1894506. https://doi.org/10.3389/fninf.2026.1894506

Image Credits: AI Generated

DOI: 10.3389/fninf.2026.1894506

Keywords: title, html, EEG-fNIRS, self-supervised, emotion, recognition, method, based, dual-stream, Siamese, network, scientific research