A SUPERB-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection
Abstract
Self-supervised learning (SSL) has transformed speech processing, with benchmarks such as SUPERB establishing fair comparisons across diverse downstream tasks. Despite it's security-critical importance, Audio deepfake detection has remained outside these efforts. In this work, we introduce Spoof-SUPERB, a benchmark for audio deepfake detection that systematically evaluates 20 SSL models spanning generative, discriminative, and spectrogram-based architectures. We evaluated these models on multiple in-domain and out-of-domain datasets. Our results reveal that large-scale discriminative models such as XLS-R, UniSpeech-SAT, and WavLM Large consistently outperform other models, benefiting from multilingual pretraining, speaker-aware objectives, and model scale. We further analyze the robustness of these models under acoustic degradations, showing that generative approaches degrade sharply, while discriminative models remain resilient. This benchmark establishes a reproducible baseline and provides practical insights into which SSL representations are most reliable for securing speech systems against audio deepfakes.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Spoof-SUPERB, a benchmark for evaluating SSL models in audio deepfake detection, filling a critical gap in the literature. The technical contributions are significant, providing a systematic framework for assessing model performance and robustness, which is essential for advancing the field of speech processing in the context of security.
Methodology
The paper introduces a novel benchmarking framework, Spoof-SUPERB, specifically designed for evaluating self-supervised learning (SSL) models in the context of audio deepfake detection. The methodology is well-structured, utilizing a unified protocol for training and evaluation across multiple datasets, which enhances comparability. The choice of models and the systematic evaluation of their performance under various conditions, including acoustic degradations, is a significant strength. However, the paper could benefit from a more detailed discussion of the specific training and evaluation protocols used, as well as the rationale behind the selection of datasets.
Results & evaluation
The experiments are comprehensive, involving 20 different SSL models evaluated on multiple datasets, which provides a robust analysis of model performance. The results clearly demonstrate the superiority of large-scale discriminative models over generative ones, particularly in terms of resilience to noise and other acoustic degradations. The use of Equal Error Rate (EER) as a performance metric is appropriate for the task, although additional metrics could provide a more nuanced view of model performance.
Reproducibility
The paper emphasizes reproducibility by establishing a fixed training setup and evaluation protocol, which is crucial for benchmarking in machine learning. However, the absence of a publicly accessible code repository or detailed implementation guidelines limits the ability of other researchers to reproduce the results. Providing such resources would significantly enhance the paper's impact.
Limitations
One limitation is the potential overlap between the pretraining data of some models and the evaluation datasets, which could bias the results. Additionally, while the paper addresses robustness under various acoustic conditions, it does not explore the implications of different synthesis methods for audio deepfakes, which could be a critical area for future research.
Broader impact
The introduction of a standardized benchmark for audio deepfake detection is a timely contribution, given the increasing prevalence of deepfake technologies and their implications for security and trust in audio communications. This work could pave the way for further advancements in antispoofing techniques and the development of more secure speech processing systems. The paper presents Spoof-SUPERB, a benchmark for evaluating SSL models in audio deepfake detection, filling a critical gap in the literature. The technical contributions are significant, providing a systematic framework for assessing model performance and robustness, which is essential for advancing the field of speech processing in the context of security.
Author of this paper? Send a correction.