Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Abstract
Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Qwen-Audio, a large-scale audio-language model that significantly advances universal audio understanding capabilities. The innovative multi-task training framework and the integration of diverse audio types and tasks position this work as a meaningful contribution to the field of machine learning, particularly in audio processing and multimodal interaction.
Methodology
The paper presents a novel multi-task training framework that effectively addresses the one-to-many interference problem in audio-language models by conditioning on hierarchical tags. This approach allows for the integration of diverse audio types and tasks, showcasing a significant advancement in the field of audio understanding. The architecture leverages a single audio encoder initialized from a robust Whisper model, which is a strategic choice that enhances the model's ability to generalize across various audio tasks. The incorporation of the SRWT task is particularly innovative, as it provides fine-grained timestamp predictions that improve the model's grounding capabilities.
Results & evaluation
The experimental evaluation is comprehensive, covering a wide range of tasks across multiple datasets. The results demonstrate that Qwen-Audio outperforms existing models without requiring task-specific fine-tuning, which is a significant achievement. The benchmarks used are relevant and diverse, including ASR, AAC, and AQA, which contribute to a robust assessment of the model's capabilities. However, the paper could benefit from more detailed comparisons against a broader array of contemporary models to further validate its claims.
Reproducibility
The paper provides a clear description of the model architecture and training methodology, which aids in reproducibility. However, specific hyperparameters and dataset details are not fully disclosed, which may hinder complete replication of the results by other researchers. The availability of the models as open-source is a positive aspect that supports reproducibility.
Limitations
One limitation is the potential overfitting to the specific datasets used during training, as the model's performance on unseen data is not thoroughly discussed. Additionally, the paper does not address the computational cost associated with training such large models, which could be a barrier for some researchers. The reliance on a single audio encoder may also limit the model's performance on highly specialized audio tasks.
Broader impact
The development of Qwen-Audio has significant implications for the audio-text multimodal community, as it enables more versatile audio interaction capabilities. Its open-source nature promotes further research and development in the field, potentially leading to advancements in applications such as virtual assistants, automated transcription services, and interactive audio systems. The model's ability to handle multiple audio types and tasks could also enhance accessibility for users with diverse needs. The paper introduces Qwen-Audio, a large-scale audio-language model that significantly advances universal audio understanding capabilities. The innovative multi-task training framework and the integration of diverse audio types and tasks position this work as a meaningful contribution to the field of machine learning, particularly in audio processing and multimodal interaction.
Curator note
Chu et al., Alibaba; universal audio LLM with 30+ tasks; strong multilingual speech + sound understanding
Author of this paper? Send a correction.