Melody or Machine: Detecting Synthetic Music with Dual-Stream Contrastive Learning
Abstract
The rapid evolution of end-to-end AI music generation poses an escalating threat to artistic authenticity and copyright, demanding detection methods that can keep pace. While foundational, existing models like SpecTTTra falter when faced with the diverse and rapidly advancing ecosystem of new generators, exhibiting significant performance drops on out-of-distribution (OOD) content. This generalization failure highlights a critical gap: the need for more challenging benchmarks and more robust detection architectures. To address this, we first introduce Melody or Machine (MoM), a new large-scale benchmark of over 130,000 songs (6,665 hours). MoM is the most diverse dataset to date, built with a mix of open and closed-source models and a curated OOD test set designed specifically to foster the development of truly generalizable detectors. Alongside this benchmark, we introduce CLAM, a novel dual-stream detection architecture. We hypothesize that subtle, machine-induced inconsistencies between vocal and instrumental elements, often imperceptible in a mixed signal, offer a powerful tell-tale sign of synthesis. CLAM is designed to test this hypothesis by employing two distinct pre-trained audio encoders (MERT and Wave2Vec2) to create parallel representations of the audio. These representations are fused by a learnable cross-aggregation module that models their inter-dependencies. The model is trained with a dual-loss objective: a standard binary cross-entropy loss for classification, complemented by a contrastive triplet loss which trains the model to distinguish between coherent and artificially mismatched stream pairings, enhancing its sensitivity to synthetic artifacts without presuming a simple feature alignment. CLAM establishes a new state-of-the-art in synthetic music forensics. It achieves an F1 score of 0.925 on our challenging MoM benchmark.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a robust framework for detecting AI-generated music through a novel dual-stream architecture and a comprehensive benchmark dataset. This work significantly advances the field of audio forensics by addressing critical challenges in generalization and robustness against evolving generative models.
Methodology
The paper introduces a novel dual-stream architecture, CLAM, which leverages two distinct pre-trained audio encoders to capture the nuances of vocal and instrumental elements in music. The methodology is well-structured, focusing on the contrastive learning approach to enhance the model's sensitivity to synthetic artifacts. The introduction of the Melody or Machine (MoM) benchmark is a significant advancement, addressing the limitations of existing datasets by providing a more diverse and challenging evaluation framework. The dual-loss objective, combining binary cross-entropy with a contrastive triplet loss, is a thoughtful design choice that enhances the model's robustness against out-of-distribution samples.
Results & evaluation
The experiments are comprehensive, demonstrating the efficacy of the proposed model against existing state-of-the-art methods. The results on the MoM benchmark, achieving an F1 score of 0.925, significantly outperforming previous models, underscore the technical impact of the research. The ablation studies provide a solid foundation for understanding the contributions of various components of the model, validating the architectural choices made.
Reproducibility
The paper provides sufficient implementation details, including the training setup, model architecture, and evaluation metrics, which facilitates reproducibility. However, the absence of a publicly accessible code repository or demo URL limits the ease with which other researchers can replicate the results.
Limitations
The primary limitation noted is the rapid pace of innovation in AI music generation, which may render the proposed detection methods obsolete as new models emerge. Additionally, the dataset's predominant focus on English songs may limit its applicability across diverse linguistic and cultural contexts.
Broader impact
The research has significant implications for the music industry, particularly in protecting intellectual property rights and maintaining artistic authenticity in the face of advancing AI technologies. The MoM dataset and CLAM model can aid content platforms and rights holders in identifying synthetic music, fostering trust in music distribution. However, there are ethical considerations regarding the potential misuse of the dataset for training more effective generative models. The main contribution of this paper is the introduction of a robust framework for detecting AI-generated music through a novel dual-stream architecture and a comprehensive benchmark dataset. This work significantly advances the field of audio forensics by addressing critical challenges in generalization and robustness against evolving generative models.
Author of this paper? Send a correction.