Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Abstract
Audio-language pretraining holds promise for general-purpose audio understanding, yet remains underexplored compared to its vision counterpart. While vision-language models like CLIP serve as widely adopted foundations, existing audio-language models primarily excel at retrieval tasks with limited adoption as general-purpose encoders. We identify three key barriers: limited large-scale audio-text corpora, insufficient caption diversity, and lack of systematic exploration and evaluation. To this end, we introduce CaptionStew, a 10.7M caption dataset aggregating diverse open-source audio-text corpora across multiple domains and captioning styles. Using this resource, we conduct the first comprehensive evaluation comparing contrastive and captioning objectives for audio representation learning across speech, music, and environmental sound tasks. Our results demonstrate that audio-language pretraining yields competitive, transferable representations. Through systematic data-scaling experiments, we reveal complementary objective strengths: contrastive learning achieves superior data efficiency at smaller scales, while captioning demonstrates better scalability on language-involved audio understanding tasks. We also find that common supervised initialization practices provide diminishing returns at scale, challenging current approaches. These findings establish audio-language pretraining as a viable pathway toward general-purpose audio representations, guiding future research. To accelerate progress, we release data preparation recipes, training protocols, and pretrained models, paving the way toward universal audio understanding.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of CaptionStew, a large and diverse audio-text dataset, and a systematic evaluation of audio-language pretraining methods, which collectively advance the understanding of audio representation learning. This work is significant as it addresses key barriers in the field and provides a pathway for future research towards universal audio understanding.
Methodology
The paper introduces a novel dataset, CaptionStew, which aggregates a large and diverse collection of audio-text pairs, addressing a significant gap in the audio-language pretraining landscape. The methodology is well-structured, comparing contrastive and captioning objectives in a systematic manner, which is a crucial contribution to the understanding of audio representation learning. The evaluation of these objectives across different audio domains (speech, music, environmental sounds) adds depth to the analysis and enhances the robustness of the findings.
Results & evaluation
The experiments are comprehensive, utilizing the newly introduced dataset to conduct a thorough evaluation of the proposed methods. The results indicate that audio-language pretraining can yield competitive representations, which is a promising finding for future research. The systematic data-scaling experiments reveal important insights into the strengths of different learning objectives, although the paper could benefit from more extensive quantitative results to support its claims.
Reproducibility
The authors have made a commendable effort to ensure reproducibility by providing comprehensive source code and detailed experimental setups. This transparency is crucial for the research community and will facilitate further exploration of the proposed methods.
Limitations
One limitation of the study is the potential bias in the dataset, as it aggregates open-source audio-text pairs, which may not cover all relevant audio domains or may have varying quality. Additionally, while the findings challenge common initialization practices, the paper does not provide extensive discussion on alternative approaches that could be explored.
Broader impact
The implications of this research are significant, as it lays the groundwork for developing general-purpose audio representations that could enhance various applications, including audio retrieval, classification, and understanding tasks. The release of the dataset and pretrained models will likely accelerate advancements in the field of audio processing and machine learning. The main contribution of this paper is the introduction of CaptionStew, a large and diverse audio-text dataset, and a systematic evaluation of audio-language pretraining methods, which collectively advance the understanding of audio representation learning. This work is significant as it addresses key barriers in the field and provides a pathway for future research towards universal audio understanding.
Author of this paper? Send a correction.