The study tackles a long-standing mismatch in the field. Most music emotion recognition research has been developed and evaluated on general datasets such as AMG1608, MagnaTagATune, and DEAM, which consist overwhelmingly of Western popular music or multi-instrument classical compositions. Models trained on those collections infer emotional cues from vocal content or rich instrumental arrangements, and they struggle when handed a solo violin recording, where emotion lives in subtle timbral differences and dynamic variations rather than harmonic complexity or lyrical content. The authors note that this domain mismatch leads to significant performance degradation when existing systems are applied to violin music. Even the violin-specific effort that preceded this work—a CNN-BiGRU-Attention baseline built alongside the VioMusic dataset—relied on end-to-end supervised learning and conventional convolutional architectures, which the new study argues have limited capability in capturing long-range temporal dependencies and generalizing to subtle emotional variation.

To frame the task itself, the researchers adopt the Russell Valence-Arousal model, a dimensional approach that treats emotion as a point in a two-dimensional plane rather than a discrete label. Valence captures how positive or negative the emotion feels, while arousal captures how calm or intense it is. Defining recognition as continuous regression along these two axes suits violin music well, because emotional transitions in a solo performance can be gradual, unevenly distributed over time, and difficult to squeeze into categories like happy or sad. The VioMusic dataset, publicly released by Ma and Zhou, provides the phrase-level annotations the approach requires: approximately 7.5 hours of recordings, 1926 segmented musical phrases, performed by three violinists across Western and Chinese classical works, folk music, and pop. Each phrase was annotated by five listeners, and the annotation consistency—measured by Cronbach’s alpha of 0.775 for arousal and 0.809 for valence—indicates good agreement among annotators.

The backbone of the new framework is HuBERT, or Hidden-Unit BERT, a self-supervised audio model pre-trained on 100,000 hours of speech. Self-supervised pre-training means the model learned hierarchical acoustic patterns and long-range dependencies from raw audio without anyone labeling a single sample. But speech and solo violin are not the same domain, and the study’s central move is a two-stage training strategy. In the first stage, the HuBERT encoder—initialized with its speech pre-trained weights—undergoes domain-adaptive continued pre-training on unlabeled VioMusic audio through contrastive learning. In the second stage, the domain-adapted encoder is fine-tuned with a Bidirectional Gated Recurrent Unit and an attention mechanism to predict continuous valence and arousal values from the emotion annotations.

The contrastive learning stage works by teaching the model what makes two versions of the same recording emotionally equivalent and what makes different recordings distinct. For each input, two randomly sampled data augmentations are applied to the Mel-spectrogram—a visual representation of the audio’s frequency content over time—generating two correlated views that form a positive sample pair. The augmentation set includes three operations: time masking, which hides continuous regions along the time axis so the model learns global temporal structures; frequency masking, which hides regions along the frequency axis to build robustness to partial information loss; and gain adjustment, which randomly scales the spectrogram amplitude to simulate volume changes under different recording conditions. All augmented views from different samples in the same batch are treated as negative pairs, so the model learns discriminative emotion-related representations without relying on any emotion labels at all.

Adapting HuBERT for spectrogram input required an architectural modification. The original HuBERT Base model uses a one-dimensional convolutional front-end of seven layers, designed for raw 16-kilohertz waveform input, with a total time-domain downsampling factor of 320. The researchers removed this front-end and replaced it with a single two-dimensional convolutional layer that projects the 80-band Mel-spectrogram into the 768-dimensional hidden space of the HuBERT Transformer. The Transformer layers carry the pre-trained weights, while the new projection layer is randomly initialized. During continued pre-training, the Transformer layers were frozen for the first 10 epochs to let the front-end adapt to the spectrogram distribution, after which all parameters were optimized jointly. A small projection head, borrowed from the SimCLR contrastive learning design, maps the encoder’s output into the contrastive loss space during pre-training and is removed before fine-tuning, where the model predicts valence and arousal through fully connected layers with a Sigmoid activation.

Under the optimal configuration, CL-HuBERT achieves a Pearson correlation coefficient of 0.718 for arousal and 0.701 for valence, with mean absolute errors of 0.089 and 0.095 respectively. The Pearson coefficient measures how closely the model’s predictions track the true emotional ratings, with values closer to 1 indicating stronger agreement, while the mean absolute error measures the average size of the prediction mistakes. The experiments used a nested evaluation strategy with 5-fold cross-validation at the song level to prevent data leakage, holding out 20 percent of songs for testing in each fold, tuning hyperparameters on a validation set, and reporting results as the mean across the five folds. The optimal feature fusion setting combines the HuBERT deep features with traditional hand-crafted acoustic features—Mel-frequency cepstral coefficients, chroma features, spectral contrast, and zero-crossing rate—through weighted summation, with the deep features receiving a weight of 0.7 and the low-level features 0.3.

The comparisons reveal how much the contrastive continued pre-training matters. A classical machine learning baseline using Support Vector Regression with hand-crafted features achieved the lowest performance, with arousal and valence correlations of only 0.352 and 0.338, showing that shallow models cannot capture the complex emotional patterns in violin music. Among self-supervised audio models, MERT, a music-domain multi-task Transformer encoder, reached 0.651 for arousal and 0.618 for valence, while SSAST achieved the second-best performance at 0.638 and 0.594. WavLM showed strong arousal prediction at 0.625 but weaker valence prediction at 0.531, which the authors attribute to a pre-training objective focused more on speech content than emotional valence. Compared with the strongest violin-specific baseline, CBA, CL-HuBERT improved arousal correlation by 7.7 percent; compared with directly fine-tuning HuBERT without contrastive pre-training, the improvement was 9.4 percent for arousal and 13.3 percent for valence. Against MERT, the gains were 6.7 percent for arousal and 8.3 percent for valence. Paired t-tests across the cross-validation folds confirmed that all improvements over CBA and MERT were statistically significant at the p < 0.05 level, meaning the advantage is reliable rather than random fluctuation.

The systematic ablations add technical texture to the result. Under the same contrastive pre-training framework, HuBERT Base outperformed VGG16, ResNet50, DenseNet121, and a standard four-layer Transformer encoder, which the authors attribute to HuBERT’s self-supervised pre-training on large-scale unlabeled audio having learned rich hierarchical representations. Mel-spectrograms proved the best input feature, beating MFCC, whose limited frequency resolution struggles to capture subtle timbral changes, and raw spectrograms, which retain complete frequency information but carry heavy redundancy. Combining data augmentation with feature normalization produced the strongest preprocessing outcome, and a leave-one-performer-out evaluation—training on two violinists and testing on the third—achieved arousal and valence correlations of 0.685 and 0.672, only slightly below the performer-dependent setting, confirming reasonable generalization to unseen players. The temperature coefficient of the contrastive loss, which controls the sharpness of the similarity distribution, proved optimal at 0.2: too small a value forces the model to focus excessively on hard negatives, while too large a value weakens the discriminative power of the objective. Mean squared error also beat Huber and mean absolute error as the fine-tuning regression loss, suggesting that a square-level penalty on large errors helps the model fit the overall trend of emotional change.

The authors are candid about the limits of the work. The VioMusic dataset is small at roughly 7.5 hours and 1926 segments, with constrained diversity of repertoires and performers, which may affect generalization; the study focuses exclusively on the violin, so the framework may not transfer to other solo instruments without cross-instrument transfer learning; and the experiments address only continuous emotion prediction, not discrete emotion classification. Future directions include larger and more diverse datasets, cross-instrument strategies, and more advanced architectures such as pure Transformer models or graph neural networks for encoding musical structure. Still, the core finding stands on its own: a model that learned to understand human speech can, through contrastive self-teaching on unlabeled recordings, learn to hear the quiet emotional language of a solo violin—and, in the tests reported in this study, predict what listeners feel more accurately than the baselines and state-of-the-art systems it was compared against.

Subject of Research: Contrastive learning with HuBERT for violin music emotion recognition

Article Title: A contrastive learning framework with HuBERT for violin music emotion recognition

Article References: Liu, X. (2026). A contrastive learning framework with HuBERT for violin music emotion recognition. Discover Artificial Intelligence, 6(1), Article 1024. https://doi.org/10.1007/s44163-026-02123-0

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02123-0

Keywords: music emotion recognition, contrastive learning, HuBERT, self-supervised learning, violin, valence-arousal, VioMusic dataset, audio deep learning, domain adaptation, affective computing, music information retrieval, feature fusion