PACE: Pretrained Audio Continual Learning
Abstract
Audio is a fundamental modality for analyzing speech, music, and environmental sounds. Although pretrained audio models have significantly advanced audio understanding, they remain fragile in real-world settings where data distributions shift over time. In this work, we present the first systematic benchmark for audio continual learning (CL) with pretrained models (PTMs), together with a comprehensive analysis of its unique challenges. Unlike in vision, where parameter-efficient fine-tuning (PEFT) has proven effective for CL, directly transferring such strategies to audio leads to poor performance. This stems from a fundamental property of audio backbones: they focus on low-level spectral details rather than structured semantics, causing severe upstream-downstream misalignment. Through extensive empirical study, we identify analytic classifiers with first-session adaptation (FSA) as a promising direction, but also reveal two major limitations: representation saturation in coarse-grained scenarios and representation drift in fine-grained scenarios. To address these challenges, we propose PACE, a novel method that enhances FSA via a regularized analytic classifier and enables multi-session adaptation through adaptive subspace-orthogonal PEFT for improved semantic alignment. In addition, we introduce spectrogram-based boundary-aware perturbations to mitigate representation overlap and improve stability. Experiments on six diverse audio CL benchmarks demonstrate that PACE substantially outperforms state-of-the-art baselines, marking an important step toward robust and scalable audio continual learning with PTMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of PACE, a novel framework for pretrained audio continual learning that effectively addresses the unique challenges posed by audio data distributions. This work significantly advances the field by providing a systematic approach to continual learning in audio, demonstrating state-of-the-art performance across multiple benchmarks while offering a foundation for future research in this area.
Methodology
The methodology presented in this paper is robust and innovative, addressing the unique challenges of continual learning (CL) in audio contexts, particularly the upstream-downstream misalignment that has hindered previous approaches. The introduction of PACE, which combines improved first-session adaptation (FSA) with multi-session adaptation (MSA) and boundary-aware regularization, is a significant advancement. The paper meticulously details the design choices behind each component, demonstrating a clear understanding of the audio domain's intricacies. The use of analytic classifiers and adaptive subspace-orthogonal PEFT is particularly noteworthy, as it showcases a tailored approach to audio CL that diverges from traditional vision-based methods.
Results & evaluation
The experimental evaluation is thorough, employing six diverse audio CL benchmarks that effectively highlight the strengths and weaknesses of the proposed method. The results consistently demonstrate that PACE outperforms state-of-the-art methods, providing strong empirical evidence for its effectiveness. The ablation studies further reinforce the validity of the proposed components, illustrating how each contributes to the overall performance. However, the paper could benefit from additional comparisons with more recent methods in the audio domain, if available.
Reproducibility
The authors commit to releasing their code and benchmarks, which is a positive aspect for reproducibility. The detailed descriptions of the experimental setup, including hyperparameters and dataset configurations, enhance the likelihood that other researchers can replicate the results. However, the absence of a demo or interactive component limits immediate accessibility for broader audiences.
Limitations
One limitation is the potential for overfitting in fine-grained tasks, as indicated by the authors. The paper also acknowledges that while PACE narrows the gap to joint training, it does not completely eliminate it, suggesting that further improvements could be made. Additionally, the reliance on specific pretrained models may limit the generalizability of the findings across different audio tasks.
Broader impact
The implications of this work are significant, particularly for applications in speech recognition, audio event detection, and environmental sound understanding. By addressing the challenges of continual learning in audio, the proposed methods could enhance the robustness and adaptability of audio models in real-world scenarios, leading to more effective and reliable systems. The main contribution of this paper is the introduction of PACE, a novel framework for pretrained audio continual learning that effectively addresses the unique challenges posed by audio data distributions. This work significantly advances the field by providing a systematic approach to continual learning in audio, demonstrating state-of-the-art performance across multiple benchmarks while offering a foundation for future research in this area.
Author of this paper? Send a correction.