CLAIP-Emo: Parameter-Efficient Adaptation of Language-supervised models for In-the-Wild Audiovisual Emotion Recognition
Abstract
Audiovisual emotion recognition (AVER) in the wild is still hindered by pose variation, occlusion, and background noise. Prevailing methods primarily rely on large-scale domain-specific pre-training, which is costly and often mismatched to real-world affective data. To address this, we present CLAIP-Emo, a modular framework that reframes in-the-wild AVER as a parameter-efficient adaptation of language-supervised foundation models (CLIP/CLAP). Specifically, it (i) preserves language-supervised priors by freezing CLIP/CLAP backbones and performing emotion-oriented adaptation via LoRA (updating \ensuremath{\le}4.0\% of the total parameters), (ii) allocates temporal modeling asymmetrically, employing a lightweight Transformer for visual dynamics while applying mean pooling for audio prosody, and (iii) applies a simple fusion head for prediction. On DFEW and MAFW, CLAIP-Emo (ViT-L/14) achieves 80.14\% and 61.18\% weighted average recall with only 8M training parameters, setting a new state of the art. Our findings suggest that parameter-efficient adaptation of language-supervised foundation models provides a scalable alternative to domain-specific pre-training for real-world AVER. The code and models will be available at \href{https://github.com/MSA-LMC/CLAIP-Emo}{https://github.com/MSA-LMC/CLAIP-Emo}.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of CLAIP-Emo, a novel framework that efficiently adapts language-supervised foundation models for in-the-wild audiovisual emotion recognition, achieving state-of-the-art results with minimal parameters. This work represents a meaningful advancement in the field, addressing key challenges in emotion recognition and offering a pathway for future research in scalable affective computing solutions.
Methodology
The methodology presented in CLAIP-Emo is innovative and well-structured, focusing on parameter-efficient adaptation of language-supervised models for audiovisual emotion recognition. The use of frozen CLIP and CLAP backbones with LoRA for lightweight adaptation is a significant advancement, allowing the model to maintain rich semantic priors while adapting to specific tasks. The asymmetric temporal modeling approach effectively leverages the strengths of both visual and audio modalities, which is a thoughtful design choice that addresses the inherent differences in how these modalities convey emotional information. The framework's architecture is modular, allowing for flexibility and potential future enhancements.
Results & evaluation
The experiments conducted on DFEW and MAFW datasets demonstrate the effectiveness of CLAIP-Emo, achieving state-of-the-art performance with a significantly reduced number of parameters compared to existing methods. The ablation studies provide valuable insights into the contributions of various components of the model, reinforcing the claims made about the advantages of the proposed architecture. However, the paper could benefit from a more extensive comparison with a broader range of existing methods to further validate its claims.
Reproducibility
The authors have committed to making their code and models available on GitHub, which is a positive step towards reproducibility. However, the paper lacks detailed implementation specifics, such as hyperparameter settings and training protocols, which are crucial for other researchers to replicate the results accurately. Providing these details would enhance the reproducibility of the findings.
Limitations
One limitation of the study is the reliance on specific datasets (DFEW and MAFW), which may not fully represent the diversity of real-world scenarios in audiovisual emotion recognition. Additionally, while the parameter-efficient approach is commendable, it remains to be seen how well the model generalizes to other datasets or tasks outside the scope of the current evaluation. The potential for catastrophic forgetting, even with the proposed methods, should also be addressed more thoroughly.
Broader impact
The implications of this research are significant, as it provides a scalable and efficient alternative to traditional domain-specific pre-training methods in the field of affective computing. The ability to recognize emotions from audiovisual cues has applications in various domains, including human-computer interaction, mental health monitoring, and entertainment. By improving the efficiency and effectiveness of emotion recognition systems, this work could contribute to more responsive and empathetic AI systems. The main contribution of this paper is the introduction of CLAIP-Emo, a novel framework that efficiently adapts language-supervised foundation models for in-the-wild audiovisual emotion recognition, achieving state-of-the-art results with minimal parameters. This work represents a meaningful advancement in the field, addressing key challenges in emotion recognition and offering a pathway for future research in scalable affective computing solutions.
Author of this paper? Send a correction.