Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
Abstract
We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reasoning over speech, environmental sounds and music. Compared to Audio Flamingo 3, AF-Next introduces: (i) a stronger foundational audio-language model that significantly improves accuracy across diverse audio understanding tasks; (ii) scalable strategies for constructing large-scale audio understanding and reasoning data beyond existing academic benchmarks; (iii) support for long and complex audio inputs up to 30 minutes; and (iv) Temporal Audio Chain-of-Thought, a new reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio, enabling fine-grained temporal alignment and improved interpretability. To enable these capabilities, we first conduct a systematic analysis of Audio Flamingo 3 to identify key gaps in audio understanding and reasoning. We then curate and scale new large-scale datasets totaling over 1 million hours to address these limitations and expand the existing AudioSkills-XL, LongAudio-XL, AF-Think and AF-Chat datasets. AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training and post-training stages. Extensive experiments across 20 audio understanding and reasoning benchmarks, including challenging long-audio tasks, show that AF-Next outperforms similarly sized open models by large margins and remains highly competitive with and sometimes surpasses, much larger open-weight and closed models. Beyond benchmark performance, AF-Next exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. In addition to all data, code and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think and AF-Next-Captioner.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Audio Flamingo Next (AF-Next), a state-of-the-art open audio-language model that significantly advances audio understanding and reasoning capabilities, particularly for long and complex audio inputs. The comprehensive methodology, extensive experimental validation, and commitment to open science position this work as a significant milestone in the development of large audio-language models.
Methodology
The methodology presented in the paper is robust, featuring a systematic analysis of previous models to identify gaps in audio understanding and reasoning. The introduction of the Temporal Audio Chain-of-Thought paradigm is particularly noteworthy, as it enhances the model's ability to handle long audio inputs by grounding reasoning steps to timestamps. The training strategy, which includes a four-stage curriculum and the curation of a large-scale dataset of over 1 million hours, demonstrates a comprehensive approach to improving model performance across various audio tasks. The use of diverse data sources and the focus on real-world applicability are commendable.
Results & evaluation
The experiments conducted across 20 audio understanding and reasoning benchmarks are extensive and well-structured. The results show that AF-Next consistently outperforms previous models, including both open-weight and closed models, particularly in long-audio tasks. The paper provides a thorough comparison with state-of-the-art models, showcasing significant improvements in accuracy and robustness. The inclusion of qualitative examples further strengthens the evaluation of the model's capabilities.
Reproducibility
The authors have committed to open-sourcing the model weights, training data, and code, which is a significant step towards ensuring reproducibility. However, the paper could benefit from more detailed descriptions of the training configurations and hyperparameters used in each stage, as well as clearer guidelines for replicating the experiments.
Limitations
The paper acknowledges several limitations, including the challenges posed by noisy and unevenly distributed training data, particularly for low-resource languages and rare sound events. Additionally, while the model improves long-audio understanding, it still faces difficulties with temporally distant evidence. The evaluation focuses primarily on established benchmarks, which may not fully capture the model's capabilities in more complex scenarios.
Broader impact
The advancements presented in AF-Next have the potential to significantly enhance audio understanding applications, including automatic speech recognition, audio captioning, and music information retrieval. The model's ability to handle long-form audio and its open-source nature could foster further research and development in the field, promoting transparency and collaboration among researchers. The main contribution of this paper is the introduction of Audio Flamingo Next (AF-Next), a state-of-the-art open audio-language model that significantly advances audio understanding and reasoning capabilities, particularly for long and complex audio inputs. The comprehensive methodology, extensive experimental validation, and commitment to open science position this work as a significant milestone in the development of large audio-language models.
Author of this paper? Send a correction.