S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
Abstract
General audio foundation models have recently achieved remarkable progress, enabling strong performance across diverse tasks. However, state-of-the-art models remain extremely large, often with hundreds of millions of parameters, leading to high inference costs and limited deployability on edge devices. Knowledge distillation is a proven strategy for model compression, but prior work in audio has mostly focused on supervised settings, relying on class logits, intermediate features, or architecture-specific techniques. Such assumptions exclude models that output only embeddings, such as self-supervised or metric-learning models. We introduce S-SONDO (Self-Supervised KnOwledge DistillatioN for General AuDio FOundation Models), the first framework to distill general audio models using only their output embeddings. By avoiding the need for logits or layer-level alignment, S-SONDO is architecture-agnostic and broadly applicable to embedding-based teachers. We demonstrate its effectiveness by distilling two audio foundation models into three efficient students that are up to 61 times smaller while retaining up to 96% of teacher performance. We also provide practical insights on loss choice and clustering-based balanced data sampling. Code is available here: https://github.com/MedAliAdlouni/ssondo.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of S-SONDO, a self-supervised knowledge distillation framework that effectively compresses large audio foundation models by utilizing only their output embeddings. This innovative approach significantly enhances the deployability of audio models on edge devices while maintaining high performance levels.
Methodology
The proposed S-SONDO framework introduces a novel approach to self-supervised knowledge distillation by focusing solely on the output embeddings of teacher and student models, rather than relying on class logits or intermediate features. This architecture-agnostic method is particularly significant as it opens up new avenues for distilling models that do not produce class-based outputs, thus broadening the applicability of knowledge distillation in audio processing. The methodology is well-structured, with clear definitions of the distillation process and the loss functions used, including a thorough exploration of balanced data sampling strategies.
Results & evaluation
The experiments conducted are robust, demonstrating the effectiveness of S-SONDO across multiple audio tasks and architectures. The authors provide comprehensive results that show distilled models achieving competitive performance compared to their supervised counterparts, with significant reductions in model size. The use of diverse datasets and the evaluation of multiple student-teacher combinations strengthen the validity of the findings. However, the paper could benefit from a more detailed analysis of the results, particularly in terms of statistical significance.
Reproducibility
The paper includes sufficient implementation details, such as the training setup, hyperparameters, and the architecture of the models used. The availability of code on GitHub enhances reproducibility, allowing other researchers to validate the findings and potentially build upon this work. However, the paper could improve by providing more detailed instructions on how to replicate the experiments, especially concerning the data preprocessing steps.
Limitations
One limitation of the study is the reliance on clustering for balanced data sampling, which may not always yield optimal results, particularly in complex multi-label scenarios. Additionally, while the results are promising, the paper does not explore the scalability of the method across significantly larger models or different types of audio tasks beyond those tested. The authors also acknowledge that the performance of the method may vary depending on the specific architectures of the teacher and student models.
Broader impact
The implications of this work are significant for the deployment of audio models in resource-constrained environments, such as mobile devices. By enabling effective knowledge distillation without architectural constraints, S-SONDO paves the way for more efficient audio processing applications, including real-time audio analysis and classification in various domains. The approach could also inspire future research into self-supervised learning techniques across other modalities, potentially leading to advancements in areas such as speech recognition, music generation, and environmental sound classification. The main contribution of this paper is the introduction of S-SONDO, a self-supervised knowledge distillation framework that effectively compresses large audio foundation models by utilizing only their output embeddings. This innovative approach significantly enhances the deployability of audio models on edge devices while maintaining high performance levels.
Author of this paper? Send a correction.