AudioFM: Towards General Purpose Audio Representation Learning
Abstract
Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive objectives encourage agreement between pooled audio representations, but do not explicitly supervise local time-frequency representations. We introduce AudioFM, a self-supervised framework that combines recording-level contrast with patch-wise masked latent discrimination. Using visible acoustic context, a student predicts representations of masked patches and identifies their corresponding full-view teacher targets among candidates from other recordings. We further incorporate weaker supervision from eligible temporal neighbors to encourage local temporal consistency. Extensive evaluations show that AudioFM achieves state-of-the-art performance across diverse audio domains in linear probing, fine-tuning, and zero-shot transfer, demonstrating superior event detection, robust scene-level understanding, and strong domain generalizability.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces AudioFM, a self-supervised audio foundation model that effectively bridges global and local representation learning through a novel masked latent discrimination objective. By combining recording-level contrast with patch-wise discrimination against cross-recording negatives and incorporating temporal neighborhood consistency, AudioFM achieves state-of-the-art performance across diverse domains, including general audio, bioacoustics, and medical acoustics, demonstrating superior transferability and robustness in both fine-tuning and zero-shot settings.
Methodology
The paper proposes AudioFM, a self-supervised framework that integrates recording-level contrastive learning with a novel local objective called Masked Latent Discrimination (MLD). The core innovation lies in the local branch, where a student model predicts representations for masked time-frequency patches and discriminates the correct full-view teacher target from negative candidates drawn from *other* recordings at the same spatial coordinate. This design avoids the common pitfall of forcing the model to distinguish acoustically similar regions within the same recording solely based on position. Additionally, the method incorporates a temporal neighborhood extension, providing weaker supervision from adjacent masked patches to enforce local temporal consistency. The use of a 2D Transformer predictor with axis-separated positional encodings is a solid architectural choice for handling the grid structure of spectrograms. The joint optimization of global and local losses is well-motivated, addressing the gap between holistic scene understanding and fine-grained event detection.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors compare AudioFM against strong baselines (AudioMAE, BEATs, EAT, SSLAM, BAT, AudioMosaic) across multiple settings: supervised fine-tuning, linear probing (frozen encoder), and zero-shot transfer. A standout feature is the systematic cross-domain evaluation spanning general audio, bioacoustics, medical acoustics, and speech emotion recognition. The inclusion of medical acoustics (PSG-Audio, CirCor) and bioacoustics (BEANS, BirdSet) is particularly valuable, as these domains are often underrepresented in standard audio SSL benchmarks. The results show consistent state-of-the-art performance, especially in linear probing where the quality of the frozen representation is isolated. The scaling analysis with model size and data volume further strengthens the contribution by demonstrating empirical scaling laws for audio pretraining.
Reproducibility
The paper provides detailed hyperparameters, architecture specifications, and training protocols in the appendix. The use of standard datasets (AudioSet, ESC-50, FSD50K, etc.) and clear evaluation metrics (mAP, Macro-F1, PSDS) enhances reproducibility. However, the specific implementation details of the "mask-aware target selection" and the exact handling of the EMA teacher updates could benefit from code release to ensure full reproducibility. The paper mentions that code is not yet available or provides no link, which is a minor drawback for immediate replication.
Limitations
The pretraining corpus is limited to AudioSet (approx. 1.68M clips), which is smaller than some recent large-scale efforts. The model is trained on single-channel audio, ignoring spatial cues that could be beneficial for certain tasks. The evaluation, while broad, does not include generative tasks or audio-text alignment, limiting the scope of "foundation model" claims. The medical acoustics evaluation, while impressive, uses relatively small datasets, and the generalizability to larger clinical datasets remains to be seen.
Broader impact
This work has significant potential for applications in environmental monitoring, healthcare (respiratory and cardiac sound analysis), and assistive technologies. By demonstrating strong transferability to specialized domains like bioacoustics and medical acoustics, the paper highlights the utility of general-purpose audio representations in niche but critical fields. The systematic evaluation framework provided could serve as a benchmark for future audio SSL research. The paper introduces AudioFM, a self-supervised audio foundation model that effectively bridges global and local representation learning through a novel masked latent discrimination objective. By combining recording-level contrast with patch-wise discrimination against cross-recording negatives and incorporating temporal neighborhood consistency, AudioFM achieves state-of-the-art performance across diverse domains, including general audio, bioacoustics, and medical acoustics, demonstrating superior transferability and robustness in both fine-tuning and zero-shot settings.
Author of this paper? Send a correction.