NABEATs: Noise-Aware Audio Representation Learning
Abstract
We propose the concept of noise-aware audio self-supervised learning (SSL), whose goal is to encode audio mixtures while suppressing undesired noise, and present Noise-Aware BEATs (NABEATs) as a BEATs-based realization of this framework. Audio SSL models are designed to handle a wide range of audio signals. Consequently, under noisy conditions, they cannot effectively focus on the target sounds relevant to a downstream task, resulting in degraded performance. To address this issue, NABEATs is trained to estimate clean BEATs representations from a noisy audio signal with an auxiliary reference noise input. This reference noise enables the model to account for specific noise characteristics at inference time, thereby achieving better generalization across operating environments. Our experimental evaluations demonstrate that NABEATs significantly improves performance of various downstream tasks under noisy conditions and also generalizes well to unseen noise types.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
Methodology
The paper proposes NABEATs, a noise-aware adaptation of the BEATs self-supervised learning (SSL) model. The core innovation lies in conditioning the representation learning process on a reference noise signal. The authors introduce two mechanisms to integrate this reference noise: Cross-Attention (CA) and Feature-wise Linear Modulation (FiLM). The model is trained via distillation from a frozen, pre-trained BEATs model, aiming to estimate clean representations from noisy inputs. This approach is theoretically sound and addresses a specific gap in general-purpose audio SSL, which typically lacks robustness to environmental noise. The distinction between using a reference noise (NABEATs) versus unconditional denoising (DBEATs) is clearly defined. However, the method is essentially an architectural modification to an existing SSL framework rather than a fundamentally new learning paradigm. The use of reference noise is analogous to techniques in speech enhancement and speaker verification, making the novelty incremental rather than transformative.
Results & evaluation
The experimental setup is rigorous and well-structured. The authors evaluate NABEATs on six diverse downstream tasks (environmental sound, speech, music) under various noisy conditions using multiple noise datasets (WHAM!, CHiME-3, MUSDB18). The inclusion of an "unseen noise" generalization test (MUSDB18) is a strong point, demonstrating the practical utility of the reference noise mechanism. The results show consistent improvements over the baseline BEATs and the unconditional denoising baseline (DBEATs), particularly in unseen noise conditions. The use of t-SNE visualizations and SNR analysis of representations provides good qualitative and quantitative support for the claims. The evaluation on the DCASE 2025 Task 2 benchmark adds real-world relevance, although the performance gain is modest. The ablation between CA and FiLM variants provides useful insight into how reference information is utilized.
Reproducibility
The paper provides sufficient detail for reproduction. It specifies the base model (BEATs_iter3.pt), the additional layer architectures (Transformer layers with MHSA/MHCA, RMSNorm, SwiGLU), training hyperparameters (AdamW, learning rate, batch size, epochs), and dataset sources (FSD50K, WHAM!, DEMAND, QUT-NOISE). The training procedure for distillation is clearly described. However, the code is not publicly linked in the provided text, and some details regarding the exact mixing ratios and SNR distributions for the downstream evaluation splits could be more precise. The use of standard datasets and a widely available base model enhances reproducibility.
Limitations
The primary limitation is the dependency on a reference noise signal. While the authors argue this is practical in many scenarios (e.g., pre-recorded noise), it is not universally available. The method does not address the case where no reference noise is present. Furthermore, the performance gains, while statistically significant, are not massive, and the model introduces additional complexity and computational overhead compared to the base BEATs. The reliance on MSE loss for distillation may not perfectly align with downstream classification objectives, as noted in the discussion of the CRM-D task. The evaluation is limited to 16kHz audio, which may not reflect high-fidelity audio applications.
Broader impact
This work contributes to the development of robust audio AI systems that can operate effectively in real-world, noisy environments. By improving the generalization of SSL models, it facilitates the deployment of audio understanding systems in diverse settings, from smart homes to industrial monitoring. The concept of noise-aware SSL could inspire similar approaches in other modalities where background interference is a major challenge. However, the potential for misuse in creating robust surveillance or eavesdropping systems is a minor concern, though mitigated by the requirement for reference noise. This paper presents a practical and effective method for enhancing the noise robustness of audio self-supervised learning models by conditioning representation learning on reference noise, demonstrating significant improvements in downstream task performance under both seen and unseen noisy conditions.
Author of this paper? Send a correction.