AudioMosaic: Contrastive Masked Audio Representation Learning
Abstract
Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce \textbf{AudioMosaic}, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-language models improves performance on audio-language tasks. The code is publicly available in our \href{https://github.com/HanxunH/AudioMosaic}{GitHub repository}.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the introduction of AudioMosaic, a contrastive audio encoder that utilizes structured time-frequency masking to enhance representation learning in audio self-supervised learning. This paper makes a meaningful contribution to the field by providing a novel approach that not only improves performance across standard benchmarks but also offers insights into the importance of masking strategies in contrastive learning for audio applications.
Methodology
The proposed methodology of AudioMosaic introduces a novel approach to contrastive audio representation learning by leveraging structured time-frequency masking to create complementary views for contrastive learning. This method addresses the challenges of effective augmentation in audio SSL, particularly in the context of spectrograms. By focusing on utterance-level representations and reducing computational costs through efficient masking, the methodology is well-justified and innovative, providing a clear advancement over previous methods that primarily relied on generative reconstruction objectives.
Results & evaluation
The experimental evaluation is comprehensive, with extensive benchmarks including AudioSet, ESC-50, and Speech Commands. The results demonstrate state-of-the-art performance across multiple tasks, validating the effectiveness of the AudioMosaic encoder in both linear probing and fine-tuning scenarios. The inclusion of ablation studies further strengthens the findings by elucidating the impact of different masking strategies and batch sizes on performance.
Reproducibility
The paper provides sufficient implementation details, including architecture specifications, training protocols, and hyperparameter settings, which enhance reproducibility. The availability of the code on GitHub supports this aspect, allowing other researchers to replicate the experiments and results.
Limitations
While the paper presents a strong methodology and results, it does not extensively discuss potential limitations or challenges in applying the proposed approach to diverse audio tasks beyond those tested. Additionally, the reliance on large batch sizes may limit accessibility for researchers with less computational resources.
Broader impact
The advancements in audio representation learning have significant implications for various applications, including audio classification, deepfake detection, and multimodal audio-language tasks. The potential for improved audio understanding systems can enhance technologies in fields such as media production, surveillance, and accessibility for the hearing impaired. The main contribution of this work is the introduction of AudioMosaic, a contrastive audio encoder that utilizes structured time-frequency masking to enhance representation learning in audio self-supervised learning. This paper makes a meaningful contribution to the field by providing a novel approach that not only improves performance across standard benchmarks but also offers insights into the importance of masking strategies in contrastive learning for audio applications.
Author of this paper? Send a correction.