Audio Interaction Model
Abstract
Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of a unified streaming audio interaction model that effectively integrates offline and online capabilities, paving the way for more interactive and responsive audio processing systems. The methodology is innovative, addressing critical challenges in real-time audio interaction, and the experimental results demonstrate its effectiveness across a range of tasks, marking a significant advancement in the field of audio language models.
Methodology
The paper introduces the Audio Interaction Model, which innovatively combines offline and online capabilities in audio processing through a unified framework called SoundFlow. The methodology is robust, employing a perceive-decide-respond loop that allows real-time interaction with audio streams. The hierarchical event curation pipeline for data synthesis and the dual-loss training approach are particularly noteworthy, addressing challenges in context retention and false triggering effectively. The asynchronous FIFO scheduling for inference further enhances the model's responsiveness and efficiency.
Results & evaluation
The experimental validation is thorough, with extensive benchmarking across eight different tasks, demonstrating competitive performance against state-of-the-art models. The introduction of the StreamAudio-2M dataset and Proactive-Sound-Bench provides a solid foundation for evaluating proactive audio intervention capabilities. The results indicate that the proposed model not only retains performance on traditional tasks but also excels in new interactive capabilities, showcasing its versatility.
Reproducibility
The paper provides sufficient details on the training pipeline, data construction, and evaluation metrics, which supports reproducibility. However, the absence of a publicly available code repository or demo limits the ease with which others can replicate the results. The detailed methodology and algorithms presented are useful for understanding the implementation but would benefit from accompanying code.
Limitations
One limitation is the reliance on synthetic data for training, which may not fully capture the complexities of real-world audio interactions. Additionally, while the model shows promising results in controlled environments, its performance in highly variable real-world conditions remains to be fully validated. The paper also does not discuss potential biases in the training data or the implications of deploying such models in sensitive applications.
Broader impact
The Audio Interaction Model has significant potential applications in various domains, including virtual assistants, interactive entertainment, and accessibility technologies. By enabling real-time audio understanding and proactive assistance, this work could enhance user experiences in smart environments. The implications of deploying such models responsibly, particularly in contexts where audio interactions can impact safety or privacy, should be considered. The main contribution of this paper is the development of a unified streaming audio interaction model that effectively integrates offline and online capabilities, paving the way for more interactive and responsive audio processing systems. The methodology is innovative, addressing critical challenges in real-time audio interaction, and the experimental results demonstrate its effectiveness across a range of tasks, marking a significant advancement in the field of audio language models.
Author of this paper? Send a correction.