I'll Keep an Ear Out: Teaching AudioLLMs Proactive Audio Assistance
Abstract
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: \texttt{ } and \texttt{ }, capturing four states: onset detection, sustained-relevance triggering, irrelevance suppression, and de-duplication. Applied to Qwen2-Audio-7B, ISM achieves 99.6\% interrupt F1 and perfect de-duplication recall on ESC-50. On noisy Epic-Sounds kitchen audio, ISM achieves the highest interrupt F1 without domain-specific training, the only method maintaining strong onset detection without over-triggering or over-suppression. Streaming evaluation confirms real-time viability with 3.5-second average latency.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
Methodology
The paper proposes Interrupt and Silent Modeling (ISM), a model-agnostic framework that enables proactive audio assistance in AudioLLMs. The core method involves extending the vocabulary of a standard AudioLLM (Qwen2-Audio-7B) with two special tokens, and , and fine-tuning the model to predict these tokens based on a user's natural language intent and the current audio stream. The methodology is clever in its simplicity, leveraging the autoregressive nature of LLMs to handle temporal reasoning and de-duplication (via conversation history) without architectural changes. The four-state decision framework (onset, sustained, irrelevant, de-duplicated) is a well-defined formalization of the problem. However, the reliance on a 7B parameter model for a binary classification-like task (interrupt/silent) raises questions about computational efficiency for wearable deployment, despite the reported latency.
Results & evaluation
The experiments are conducted on ESC-50 (training and evaluation) and Epic-Sounds (zero-shot evaluation). The results on ESC-50 are exceptionally high (99.6% F1), which is somewhat suspiciously perfect for a real-world audio task, suggesting potential overfitting or a very easy evaluation setup (e.g., clean, isolated clips). The zero-shot transfer to Epic-Sounds is a strong point, showing the model generalizes to noisy, egocentric kitchen audio without domain-specific training. The streaming evaluation confirms real-time viability with 3.5s latency. However, the lack of comparison with other proactive or streaming audio models (if any exist) and the limited number of baselines (Zero-Shot, Reactive SFT) weakens the comparative analysis. The "perfect" de-duplication recall is also a red flag that warrants scrutiny of the evaluation protocol.
Reproducibility
The paper provides sufficient details on the architecture (Qwen2-Audio-7B, LoRA rank 8), training procedure (two-stage SFT), and data construction for the four states. However, the exact prompts used for the "watch-out intent" and the specific construction of the streaming evaluation set are not fully detailed in the provided text. The code and data are not explicitly linked, which hinders immediate reproducibility.
Limitations
The primary limitation is the computational cost of using a 7B LLM for a binary decision task, which may be prohibitive for battery-powered wearable devices despite the latency claim. The near-perfect scores on ESC-50 may not reflect real-world performance in highly complex, overlapping acoustic scenes. The paper focuses only on explicit intents, leaving implicit and semantic assistance for future work. The lack of user studies with DHH participants limits the validation of the system's practical utility.
Broader impact
This work has significant potential for assistive technology, particularly for Deaf and Hard of Hearing (DHH) users who rely on sound awareness. By enabling proactive, intent-based alerts with de-duplication, it addresses key pain points in current wearable sound awareness systems (notification fatigue, lack of personalization). The model-agnostic nature of ISM makes it a portable capability that can be integrated into future AudioLLMs, potentially driving a shift from reactive to proactive audio AI. The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
Author of this paper? Send a correction.