USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
Abstract
Audio encoders are critical to modern audio applications as large language models (LLMs) increasingly rely on a single encoder for diverse inputs. While self-supervised learning (SSL) has yielded strong domain-specific encoders like speech or music experts, multi-domain approaches like USAD and SPEAR remain limited in coverage and evaluation. Recent studies also suggest supervised encoders align better with audio LLMs. We present USAD 2.0, a universal encoder integrating knowledge from both SSL and supervised foundation models. USAD 2.0 introduces domain-aware distillation to address teacher mismatch, extends coverage to the music domain, and adds second-stage supervised distillation for downstream use. We further scale the model to one billion parameters via depth scaling. Experiments show USAD 2.0 achieves strong or state-of-the-art performance across probing and LLM-based evaluations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of USAD 2.0, a scalable universal audio encoder that effectively integrates knowledge from self-supervised and supervised models, achieving state-of-the-art performance across multiple audio domains. This work represents a significant step forward in the field of audio understanding, particularly in the context of large language models.
Methodology
The methodology presented in USAD 2.0 is robust, integrating domain-aware distillation and second-stage supervised distillation to enhance audio representation across multiple domains. The introduction of a music-domain expert and the scaling of model parameters to one billion through depth scaling are significant advancements. The paper effectively combines insights from self-supervised and supervised learning, addressing teacher mismatch and enhancing performance across diverse audio tasks.
Results & evaluation
The experimental setup is comprehensive, utilizing a large multi-domain audio dataset and rigorous evaluation protocols, including probing tasks and LLM-based evaluations. The results demonstrate that USAD 2.0 outperforms previous state-of-the-art models in various benchmarks, indicating strong empirical validation of the proposed methods. The ablation studies further substantiate the effectiveness of the domain-aware distillation and scaling techniques.
Reproducibility
The paper provides detailed implementation details, including hyperparameters and training setups, which enhances reproducibility. However, the absence of a publicly available code repository or demo limits the ease of reproduction for external researchers.
Limitations
One limitation is the reliance on the quality of the training datasets, which could affect the model's performance in real-world applications. Additionally, while the model shows strong performance across domains, further exploration of its generalization capabilities in unseen audio contexts is necessary.
Broader impact
The advancements in universal audio understanding have significant implications for various applications, including automatic speech recognition, music classification, and multimodal LLMs. The integration of diverse audio domains into a single encoder can facilitate more efficient and effective audio processing systems, potentially transforming how audio data is utilized in AI applications. The main contribution of this paper is the development of USAD 2.0, a scalable universal audio encoder that effectively integrates knowledge from self-supervised and supervised models, achieving state-of-the-art performance across multiple audio domains. This work represents a significant step forward in the field of audio understanding, particularly in the context of large language models.
Author of this paper? Send a correction.