Robust Neural Audio Fingerprinting using Music Foundation Models
Abstract
The proliferation of distorted, compressed, and manipulated music on modern media platforms like TikTok motivates the development of more robust audio fingerprinting techniques to identify the sources of musical recordings. In this paper, we develop and evaluate new neural audio fingerprinting techniques with the aim of improving their robustness. We make two contributions to neural fingerprinting methodology: (1) we use a pretrained music foundation model as the backbone of the neural architecture and (2) we expand the use of data augmentation to train fingerprinting models under a wide variety of audio manipulations, including time streching, pitch modulation, compression, and filtering. We systematically evaluate our methods in comparison to two state-of-the-art neural fingerprinting models: NAFP and GraFPrint. Results show that fingerprints extracted with music foundation models (e.g., MuQ, MERT) consistently outperform models trained from scratch or pretrained on non-musical audio. Segment-level evaluation further reveals their capability to accurately localize fingerprint matches, an important practical feature for catalog management.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the introduction of pretrained music foundation models for robust audio fingerprinting, demonstrating superior performance over traditional methods and establishing a new benchmark for future research in the field. The comprehensive evaluation and innovative methodology position this paper as a significant advancement in audio processing and machine learning applications.
Methodology
The methodology is robust, leveraging pretrained music foundation models (MuQ and MERT) for audio fingerprinting, which is a significant advancement over traditional methods. The systematic evaluation of data augmentations and the architecture design (including a two-layer MLP projection head) are well-justified and contribute to the overall effectiveness of the approach. The choice of datasets and the detailed description of augmentations provide a solid foundation for the experiments.
Results & evaluation
The experiments are comprehensive, comparing against established models (NAFP and GraFPrint) and including both track-level and segment-level evaluations. The results demonstrate a clear performance advantage of the proposed models, particularly in challenging conditions, which is crucial for real-world applications. However, the paper could benefit from more detailed statistical analyses to quantify the significance of the results.
Reproducibility
The paper provides sufficient details about the datasets, model architectures, and training procedures, which enhances reproducibility. However, the lack of publicly accessible code or demo URLs limits the ability for others to replicate the findings easily.
Limitations
The paper acknowledges weaknesses in handling certain transformations, such as spectral filtering, which could be a significant limitation in practical applications. Additionally, the focus on specific datasets may not fully represent the diversity of audio encountered in real-world scenarios.
Broader impact
The advancements in audio fingerprinting have significant implications for music identification, copyright enforcement, and catalog management in the age of digital media. The robustness of the proposed methods against various audio manipulations is particularly relevant for platforms like TikTok, where music is frequently altered. The main contribution of this work is the introduction of pretrained music foundation models for robust audio fingerprinting, demonstrating superior performance over traditional methods and establishing a new benchmark for future research in the field. The comprehensive evaluation and innovative methodology position this paper as a significant advancement in audio processing and machine learning applications.
Author of this paper? Send a correction.