Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection
Abstract
Existing audio deepfake detection (ADD) methods frequently exhibit limited generalization, as fixed bottleneck architectures fail to adapt to the heterogeneous information densities inherent in diverse spoofing artifacts. Moreover, standard augmentation often leads to a pathological robustness-accuracy trade-off. In this paper, we propose a unified ADD framework to address these challenges. Our approach integrates: (1) an Adaptive Audio Feature Integration (AAFI) module that dynamically selects optimal latent dimensions via a compression pool and a stochastic exploration mechanism, ensuring flexible representations across multiple types of dynamic attacks; and (2) a Mel-Band Adversarial Perturbation (Mel-BAP) strategy that applies targeted regularization in the mel-spectrogram domain to encourage the model to learn invariant decision boundaries. Extensive evaluations on benchmarks including ADD 2022, FakeOrReal, and In-the-Wild demonstrate that our model based on Whisper-small achieves a state-of-the-art average Equal Error Rate (EER) of 1.51%. Compared to existing SOTA models such as the 3-billion-parameter Resemble-Detect-3B-Omni, our framework reduces the parameter count by up to 91.6% while achieving a 60.7% improvement in average EER. In a separate cross-lingual evaluation, the Whisper-base configuration achieves EERs of 3.91-5.65% on five unseen languages. The framework also maintains strong resilience to real-world acoustic distortions. This work provides a scalable and efficient "plug-in" solution for enhancing trust and security in the audio Web ecosystem.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel adaptive compression and targeted perturbation framework for audio deepfake detection, achieving state-of-the-art performance with significantly reduced model size. The integration of dynamic feature selection and frequency-domain adversarial training offers a robust solution to the generalization challenges inherent in current ADD methods, providing a scalable and efficient tool for enhancing trust in digital audio content.
Methodology
The paper proposes a unified framework for Audio Deepfake Detection (ADD) built upon the Whisper-small backbone. The core contribution is the Adaptive Audio Feature Integration (AAFI) module, which employs a heterogeneous compression pool and a stochastic router to dynamically select latent dimensions based on input characteristics. This addresses the limitation of fixed bottleneck architectures by allowing the model to adapt its information density to different types of spoofing artifacts. The second component, Mel-Band Adversarial Perturbation (Mel-BAP), applies targeted adversarial training in the mel-spectrogram domain, specifically focusing on frequency bands where neural vocoder artifacts are prevalent. The methodology is sound, leveraging the strong pre-trained representations of Whisper while introducing adaptive mechanisms to enhance generalization. The use of the Whisper decoder for classification rather than a simple MLP head is a creative choice, justified by the need for temporal granularity and semantic-acoustic alignment.
Results & evaluation
The experiments are extensive, covering ADD 2022, FakeOrReal, and In-the-Wild benchmarks. The reported state-of-the-art average EER of 1.51% is impressive, especially given the significant reduction in parameter count compared to larger models like Resemble-Detect-3B-Omni. The cross-lingual evaluation on five unseen languages demonstrates strong zero-shot transferability. The ablation studies (referenced in the text) appear to validate the individual contributions of AAFI and Mel-BAP. The comparison against SOTA models is rigorous, highlighting both accuracy and efficiency gains.
Reproducibility
The paper provides detailed architectural descriptions, including the formulation of the routing mechanism and the loss functions. However, specific hyperparameters (e.g., noise scale, top-k value, exact compression dimensions) are deferred to the appendix. The reliance on the Whisper backbone ensures that the base model is accessible, but the specific implementation details of the AAFI module and Mel-BAP strategy would require careful replication. The code availability is not explicitly mentioned in the provided text, which is a minor drawback for immediate reproducibility.
Limitations
The framework relies heavily on the Whisper encoder, which, while powerful, may still struggle with extremely low-quality or heavily distorted audio where the pre-trained features are less reliable. The stochastic exploration mechanism in AAFI adds computational overhead during training, although inference remains efficient. The paper focuses on binary classification (genuine vs. spoofed), and extending this to multi-class or open-set detection scenarios is not explored. Additionally, the performance on real-world, non-benchmarked audio sources is only partially addressed through the In-the-Wild dataset.
Broader impact
This work has significant implications for the security of the audio web ecosystem. By providing a scalable and efficient solution for detecting audio deepfakes, it can be integrated into content moderation systems, verification platforms, and forensic tools. The focus on generalization and robustness makes it particularly relevant for real-world deployment where spoofing techniques evolve rapidly. The efficiency gains also make it suitable for edge devices or large-scale processing, broadening its applicability. The paper presents a novel adaptive compression and targeted perturbation framework for audio deepfake detection, achieving state-of-the-art performance with significantly reduced model size. The integration of dynamic feature selection and frequency-domain adversarial training offers a robust solution to the generalization challenges inherent in current ADD methods, providing a scalable and efficient tool for enhancing trust in digital audio content.
Author of this paper? Send a correction.