From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models
Abstract
Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant and timely contribution to the field of audio AI by introducing CASU, a benchmark that rigorously evaluates the ability of Large Audio Language Models to perform context-aware reasoning over complex, multi-layered auditory scenes. By shifting the focus from isolated perception to holistic scene understanding, the authors identify a critical limitation in current state-of-the-art models and provide a scalable, semi-synthetic pipeline to address it, thereby establishing a new standard for evaluating auditory intelligence.
Methodology
The paper introduces Context-Aware Auditory Scene Understanding (CASU), a novel benchmark and evaluation paradigm designed to assess Large Audio Language Models (LALMs) on their ability to integrate multiple acoustic layers (speech, events, background) for scene-level reasoning. The core methodological contribution is a semi-synthetic data generation pipeline that combines real-world environmental sounds and discrete events with synthetic speech, controlled via structured JSON scripts generated by LLMs. This approach allows for precise manipulation of cross-layer contextual relationships, which is difficult to achieve with naturalistic, unannotated audio. The benchmark defines four specific tasks: Contextual Reasoning, Entity Extraction, Role Inference, and Counterfactual Reasoning. The methodology is sound in its intent to move beyond isolated perception tasks, addressing a genuine gap in current LALM evaluations where models often treat non-speech audio as mere noise or background rather than semantic anchors. The use of an agent-based question generation framework adds a layer of scalability to dataset creation.
Results & evaluation
The experimental evaluation is comprehensive, benchmarking a wide range of state-of-the-art LALMs, including open-source models (Qwen series, Audio Flamingo, Voxtral, SALMONN, LTU) and closed-source giants (GPT-4o Audio, Gemini 2.0 Flash). The results clearly demonstrate a "Perception-Understanding Gap," where models with high transcription accuracy (low WER) and event detection performance still struggle with tasks requiring logical integration of context. Key findings include the superiority of joint processing (omni-modal models) over cascaded pipelines (transcription + text reasoning) due to information loss in textual descriptions. The ablation studies effectively isolate the contribution of different audio layers, confirming that removing any single layer (speech, event, or background) significantly degrades performance. The error analysis provides valuable insights into whether failures stem from perceptual errors or reasoning flaws.
Reproducibility
The paper provides detailed descriptions of the data generation pipeline, including the use of specific TTS tools (Zonos), retrieval datasets (Clotho, ARCA23K), and the matching score formula. The structured JSON script format for ground truth is a strong point for reproducibility, as it allows other researchers to regenerate similar scenes. However, the reliance on proprietary models for question generation and human curation steps introduces some opacity. The code and dataset are not explicitly linked in the provided text (URLs are "none"), which hinders immediate reproducibility, though the methodology is described sufficiently for replication.
Limitations
The primary limitation is the synthetic nature of the speech component, which, while using high-fidelity TTS, may not fully capture the nuances of natural human speech (prosody, disfluencies, emotional variance) present in real-world recordings. The constraint of audio clips to under 30 seconds limits the complexity of scenes that can be modeled, potentially missing long-range dependencies. Additionally, the current scope is limited to one-person monologues and two-person conversations, excluding more complex multi-party interactions. The reliance on LLMs for script generation and question creation may introduce biases or logical inconsistencies that require significant human filtering, as acknowledged by the authors.
Broader impact
This work has significant implications for the development of more robust and context-aware audio AI systems. By highlighting the "Perception-Understanding Gap," it directs future research towards architectures that can better integrate multimodal signals for reasoning. This is crucial for applications such as autonomous driving (interpreting sirens vs. speech), smart home assistants, and accessibility tools for the hearing impaired. The benchmark provides a standardized way to evaluate progress in this under-explored area, fostering competition and improvement in holistic audio understanding. This paper presents a significant and timely contribution to the field of audio AI by introducing CASU, a benchmark that rigorously evaluates the ability of Large Audio Language Models to perform context-aware reasoning over complex, multi-layered auditory scenes. By shifting the focus from isolated perception to holistic scene understanding, the authors identify a critical limitation in current state-of-the-art models and provide a scalable, semi-synthetic pipeline to address it, thereby establishing a new standard for evaluating auditory intelligence.
Author of this paper? Send a correction.