UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

Yuxuan Chen, Peize He, Haoyuan Xu ...Full author list in the original paper.

Jilin University +3 · arXiv

Affiliations: Hunan University, Jilin University, Shandong University, University of Electronic Science and Technology of China

Paper

Abstract

A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

UniWhisper presents a novel approach to continual multi-task training for universal audio representation, achieving competitive performance across diverse audio tasks while maintaining strong speech capabilities. This work significantly contributes to the field by addressing the limitations of existing models and offering a streamlined methodology that enhances both efficiency and effectiveness in audio representation learning.

Methodology

The methodology presented in UniWhisper is innovative, focusing on a unified instruction and answer format for continual multi-task training. This approach effectively addresses the limitations of existing audio encoders that excel in specific domains but struggle with others. By leveraging a single encoder and a compact pretrained language model as the decoder, the authors streamline the training process and reduce redundancy in audio token representation. The decision to utilize shallow MLP probes and kNN for evaluation is appropriate, as it allows for a clear assessment of the model's performance across diverse tasks.

Results & evaluation

The experimental setup is robust, utilizing a substantial training dataset of 38k hours of public audio, which enhances the generalizability of the results. The evaluation across 20 tasks provides a comprehensive view of the model's capabilities. The results indicate that UniWhisper outperforms existing models like Whisper, HuBERT, and others, particularly in non-speech tasks, which is a significant achievement. The use of normalized weighted averages for performance metrics is a strong point, ensuring comparability across different tasks.

Reproducibility

The paper mentions that code and pretrained weights will be released upon acceptance, which is crucial for reproducibility. However, the details provided in the methodology, such as specific hyperparameters and training configurations, are adequately described, allowing other researchers to replicate the experiments. The use of a compact pretrained language model as a decoder is also a noteworthy detail that aids in understanding the architecture.

Limitations

One limitation is the reliance on a single encoder, which may not capture all nuances across diverse audio tasks as effectively as dual-encoder systems. Additionally, while the results are promising, the paper does not extensively discuss the potential trade-offs in performance when adapting UniWhisper to other audio domains not covered in the evaluation.

Broader impact

The implications of this work are significant, as it proposes a more efficient method for training universal audio representations that can be applied in various applications, including speech recognition, environmental sound classification, and music analysis. The potential for reduced training costs and improved performance across multiple tasks could lead to advancements in audio processing technologies, making them more accessible and effective in real-world applications. UniWhisper presents a novel approach to continual multi-task training for universal audio representation, achieving competitive performance across diverse audio tasks while maintaining strong speech capabilities. This work significantly contributes to the field by addressing the limitations of existing models and offering a streamlined methodology that enhances both efficiency and effectiveness in audio representation learning.

Author of this paper? Send a correction.