AI Reads Faces and Voices to Spot Depression with Directional Causal Fusion

Depression is among the most common psychiatric disorders, and early, accurate diagnosis remains an unmet clinical need. A research team at SRM Institute of Science and Technology in Tamil Nadu, India, now reports a new artificial intelligence framework that reads acoustic and visual cues together and explains its own decisions. The system, called Adaptive Spatio-Temporal Directional Causal Fusion BiLSTM-Mamba, or AST-DCF BiLSTM-Mamba, was described in a research article published on 28 October 2026 in Frontiers in Computer Science by Shaik Rasheeda Begum and Saad Yunus Sait. On the widely used D-Vlog multimodal dataset, the model reached 81.86 percent accuracy, which the authors report as higher than the baseline and state-of-the-art methods they compared against.

The central novelty lies in how the framework fuses information from two modalities. Most existing cross-modal fusion methods, including cross-attention transformers, mutual transformers, and Mamba-based fusion schemes, model the relationship between acoustic and visual streams symmetrically, capturing correlations between modalities at the same timestamp. The new approach adds a Directional Causal Fusion module inspired by Granger causality, a statistical concept for testing whether the past of one time series helps predict the future of another. In this design, lightweight self-predictors estimate the next state of each modality from its own history, while cross-predictors estimate the same quantity using the history of both modalities. If the cross-predictor error drops below the self-predictor baseline, the model treats this as evidence of a directed, lagged predictive influence from one modality to the other rather than a mere correlation.

A subtle technical safeguard underpins this causal estimate. The self- and cross-predictors operate only on forward-direction, causally restricted encodings produced by the temporal branch, because a bidirectional encoding could already contain information from future time steps and would contaminate the prediction-error comparison with leakage. The authors also applied a temporal shuffle control, randomly permuting one modality’s history along the time axis before passing it to the cross-predictor. Under this control, the causal-influence gate values converged on the self-predictor baseline, indicating that the predictive advantage observed in the unshuffled setting reflects a genuine temporal link across modalities rather than leakage or spurious correlation.

The architecture itself combines several blocks. A Spatial Mamba block extracts spatial feature representations from audio and video inputs using layer normalization, convolution, depth-wise convolution, and a Mamba module for long-range spatial dependencies. A Temporal Mamba-BiLSTM block runs a bidirectional long short-term memory network in parallel with a state space model, the sequential backbone made popular by the Mamba architecture, and merges the two outputs through a gated weighting mechanism. According to the authors, this pairing captures fine-grained short-term dependencies through the BiLSTM and coarse-grained long-range temporal structure through the state space model. A Cross-modal Interaction Mamba Encoder then captures symmetric co-occurrence relationships between audio and visual states at the same time step, complementing the directional module. An adaptive attention mechanism weighs the two streams, and a fully connected layer with a sigmoid activation produces the final depression probability.

Training data posed a familiar obstacle. The D-Vlog dataset contains 961 samples, 555 depression-related videos and 406 non-depression videos, with 161 features per sample, 25 derived from audio and 136 from video. The authors note that the small sample size and class imbalance can limit the diversity of training data and the generalization of deep learning models. To address this, they introduced a hybrid TimeGAN-Diffusion augmentation pipeline. TimeGAN, a generative adversarial network designed for time-series data, generated roughly 10,000 synthetic samples from the 672-sample training split, and a diffusion model, first trained on the original data, was then used to remove noise and artifacts from the synthetic samples. The final training set contained 10,672 samples, while the original validation and test sets of 96 and 193 samples were kept untouched.

The augmentation paid off in controlled comparisons. Using a BiLSTM classifier to evaluate augmentation strategies, the team reported 68.39 percent accuracy without augmentation, 69.43 percent with TimeGAN alone, 70.46 percent with a diffusion model alone, and 72.53 percent with the hybrid TimeGAN-Diffusion approach. For the full framework, the proposed model achieved 81.86 percent accuracy, 83.92 percent precision, 84.68 percent recall, and an 84.29 percent F1 score on the augmented test set, with a standard deviation of 0.019 across ten independent runs. Without augmentation, the same model still reached 78.23 percent accuracy and an 81.24 percent F1 score. Baseline models tested under identical settings, including the same 7:1:2 data split, 200 training epochs, early stopping, and ten runs, included a standard Mamba model at 73.05 percent accuracy, STCM-Mamba at 73.57 percent, and CAF-Mamba at 75.12 percent.

Comparisons with other published Mamba-based depression detection methods showed a similar pattern. The proposed model’s 81.86 percent accuracy exceeded Emo-Mamba at 75.54 percent, CAF-Mamba at 72.17 percent, DepMamba at 68.87 percent, and STCM-Mamba at 72.64 percent as reported in the article’s comparison table. In the confusion matrix on the D-Vlog test set, the proposed model correctly classified 94 depression samples and 64 non-depression samples, with 17 and 18 misclassifications respectively. A separate evaluation on the AVEC 2017 challenge setup, which uses audiovisual clinical interview data from the DAIC-WOZ corpus, yielded 85.10 percent accuracy and a 75.85 percent F1 score, which the authors present as evidence of consistent performance across dataset protocols.

Ablation studies dissected which components matter. On unimodal inputs, the audio-only configuration reached 75.12 percent accuracy while the video-only configuration reached 71.50 percent, and the combined multimodal fusion lifted performance to 81.86 percent. Separating the two feature-extraction blocks, the Spatial Mamba block alone achieved 77.72 percent accuracy and an 81.69 percent F1 score, the Temporal Mamba-BiLSTM block alone reached 80.31 percent accuracy and an 83.17 percent F1 score, and the combination produced the best results. Testing the fusion modules independently, the Directional Causal Fusion module alone achieved 80.82 percent accuracy, the Cross-modal Interaction Mamba Encoder alone 79.79 percent, and the two together 81.86 percent, which the authors interpret as evidence that synchronous interaction modeling and directional influence estimation contribute complementary information.

Interpretability received its own dedicated module, the Cross-Modal Consistency-Guided SHAP-LIME framework. SHAP provides global feature importance by estimating each feature’s marginal contribution to the model output, while LIME generates local explanations by perturbing inputs and fitting a surrogate model, but LIME is sensitive to perturbation and SHAP cannot fully reveal local decision boundaries. The new module computes the cosine similarity between SHAP and LIME explanations for each modality and uses this consistency score to weight the two explanation sources adaptively, with a stabilizing constant preventing degenerate weighting when agreement is low. Across the assessed samples, the agreement between SHAP and LIME was higher for the audio modality, at 0.84, than for the video modality, at 0.76, and all three explanation methods consistently ranked audio as more important than video, with the fused framework showing lower standard deviations and thus more stable importance estimates.

The team also validated explanation faithfulness with a perturbation-based Area Over the Perturbation Curve evaluation. Top-ranked features were sequentially deleted or inserted while confidence changes on the true class were recorded. The consistency-guided framework produced the lowest deletion scores and the highest insertion scores across both modalities, outperforming SHAP alone and LIME alone on all four metrics, indicating that its feature rankings align more closely with the features that actually drive the model’s predictions. The authors caution that their claims are their own and that future work will extend the framework to text, physiological, and behavioral modalities, pursue lightweight self-supervised learning for real-time use, and add uncertainty-aware and causal explanation methods.

Subject of Research: A multimodal deep learning framework combining Mamba, BiLSTM, directional causal fusion and SHAP-LIME explainability for depression detection from audio and video data.

Article Title: An adaptive spatio-temporal cross-modal learning with directional causal fusion for explainable multimodal depression detection

Article References: Begum, S. R., & Sait, S. Y. (2026). An adaptive spatio-temporal cross-modal learning with directional causal fusion for explainable multimodal depression detection. Frontiers in Computer Science, 8, 1948529. https://doi.org/10.3389/fcomp.2026.1948529

Image Credits: AI Generated

DOI: 10.3389/fcomp.2026.1948529

Keywords: depression detection, multimodal fusion, Mamba, BiLSTM, Granger causality, D-Vlog dataset, data augmentation, TimeGAN, diffusion model, SHAP, LIME, explainable AI