Do Compact SSL Backbones Matter for Audio Deepfake Detection? A Controlled Study with RAPTOR
Abstract
Self-supervised learning (SSL) underpins modern audio deepfake detection, yet most prior work centers on a single large wav2vec2-XLSR backbone, leaving compact under studied. We present RAPTOR, Representation Aware Pairwise-gated Transformer for Out-of-domain Recognition a controlled study of compact SSL backbones from the HuBERT and WavLM within a unified pairwise-gated fusion detector, evaluated across 14 cross-domain benchmarks. We show that multilingual HuBERT pre-training is the primary driver of cross-domain robustness, enabling 100M models to match larger and commercial systems. Beyond EER, we introduce a test-time augmentation protocol with perturbation-based aleatoric uncertainty to expose calibration differences invisible to standard metrics: WavLM variants exhibit overconfident miscalibration under perturbation, whereas iterative mHuBERT remains stable. These findings indicate that SSL pre-training trajectory, not model scale, drives reliable audio deepfake detection.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the demonstration that compact SSL backbones can achieve competitive performance in audio deepfake detection through careful pre-training strategies, while also introducing a novel method for assessing model calibration under distributional shifts. This work significantly advances the understanding of how SSL pre-training affects model robustness and reliability in practical applications.
Methodology
The paper introduces RAPTOR, a pairwise-gated hierarchical layer-fusion architecture, to evaluate the performance of compact self-supervised learning (SSL) backbones for audio deepfake detection. The methodology is robust, employing a controlled experimental setup where only the SSL encoder is varied while keeping the downstream detection framework constant. This approach allows for a clear analysis of the impact of different pre-training strategies on model performance. The introduction of test-time augmentation (TTA) for uncertainty estimation is particularly noteworthy, as it provides a novel way to assess model calibration beyond traditional metrics.
Results & evaluation
The authors conduct extensive experiments across 14 cross-domain benchmarks, which is a significant contribution to the field as it highlights the robustness of the proposed models under varying conditions. The results demonstrate that the compact models can achieve competitive performance compared to larger models, which is an important finding for practical applications. The use of multiple evaluation metrics, including EER and pooled EER, adds depth to the analysis and provides a comprehensive view of model performance.
Reproducibility
The paper provides sufficient implementation details, including training protocols, datasets, and evaluation metrics, which enhances reproducibility. However, the absence of a publicly accessible code repository limits the ease with which other researchers can replicate the findings.
Limitations
One limitation of the study is the reliance on specific datasets for training and evaluation, which may not fully capture the diversity of real-world audio deepfake scenarios. Additionally, the paper acknowledges the need for further investigation into the sensitivity-diversity trade-off observed in the final mHuBERT checkpoint.
Broader impact
The findings of this research have significant implications for the field of audio deepfake detection, particularly in enhancing the reliability of detection systems in real-world applications. The emphasis on model calibration and the effectiveness of compact models could lead to more accessible and efficient solutions for combating audio deepfakes. The main contribution of this paper is the demonstration that compact SSL backbones can achieve competitive performance in audio deepfake detection through careful pre-training strategies, while also introducing a novel method for assessing model calibration under distributional shifts. This work significantly advances the understanding of how SSL pre-training affects model robustness and reliability in practical applications.
Author of this paper? Send a correction.