Echoes: A semantically-aligned music deepfake detection dataset
Abstract
We introduce Echoes, a new dataset for music deepfake detection designed for training and benchmarking detectors under realistic and provider-diverse conditions. Echoes comprises 3,577 tracks (110 hours of audio) spanning multiple genres (pop, rock, electronic), and includes content generated by ten popular AI music generation systems. To prevent shortcut learning and promote robust generalization, the dataset is deliberately constructed to be challenging, enforcing semantic-level alignment between spoofed audio and bona fide references. This alignment is achieved by conditioning generated audio samples directly on bona-fide waveforms or song descriptors. We evaluate Echoes in a cross-dataset setting against three existing AI-generated music datasets using state-of-the-art Wav2Vec2 XLS-R 2B representations. Results show that (i) Echoes is the hardest in-domain dataset; (ii) detectors trained on existing datasets transfer poorly to Echoes; (iii) training on Echoes yields the strongest generalization performance. These findings suggest that provider diversity and semantic alignment help learn more transferable detection cues.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Echoes, a semantically-aligned dataset for AI-generated music detection, which significantly enhances the benchmarking landscape for music deepfake detection by addressing key challenges in data diversity and shortcut learning.
Methodology
The methodology is robust, focusing on generating a diverse dataset that emphasizes semantic alignment between real and AI-generated music. The use of LLMs to derive song descriptors for conditioning the generation process is innovative and addresses the challenges of shortcut learning in deepfake detection. The dataset's design, including the variety of music genres and the inclusion of multiple AI music generation systems, enhances its applicability and relevance.
Results & evaluation
The experimental evaluation is thorough, demonstrating the dataset's effectiveness through cross-dataset testing. The results highlight the difficulty of the Echoes dataset and its ability to promote generalization in detection models. The use of Wav2Vec2 XLS-R 2B for feature extraction and the evaluation metrics employed (EER) are appropriate for the task at hand.
Reproducibility
The paper provides sufficient details about the dataset generation process, including the selection of bona fide tracks and the conditioning methods used for AI-generated samples. However, the lack of a direct link to the code or model training details limits full reproducibility.
Limitations
One limitation is the reliance on a specific set of AI music generation systems, which may not encompass the full spectrum of current technologies. Additionally, the dataset may not cover all possible music genres or styles, potentially limiting its generalizability. The paper also mentions that future work will explore more complex scenarios, indicating that the current evaluation may not fully capture real-world conditions.
Broader impact
The dataset has significant implications for the music industry, particularly in addressing the challenges posed by AI-generated music. By providing a benchmark for deepfake detection, it can help improve the integrity of music platforms and support the development of more reliable detection systems. This work also opens avenues for further research in audio forensics and the ethical implications of AI in creative fields. The paper presents Echoes, a semantically-aligned dataset for AI-generated music detection, which significantly enhances the benchmarking landscape for music deepfake detection by addressing key challenges in data diversity and shortcut learning.
Author of this paper? Send a correction.