EvA: An Evidence-First Audio Understanding Paradigm for LALMs
Abstract
Large Audio Language Models (LALMs) still struggle in complex acoustic scenes because they often fail to preserve task-relevant acoustic evidence before reasoning begins. We call this failure the evidence bottleneck: state-of-the-art systems show larger deficits in evidence extraction than in downstream reasoning, suggesting that the main limitation lies in upstream perception rather than reasoning policy. To address this problem, we propose EvA (Evidence-First Audio), a dual-path architecture that combines Whisper and CED-Base through non-compressive, time-aligned fusion. EvA first aggregates intermediate CED layers to preserve multi-scale acoustic cues, then aligns the aggregated CED features to the Whisper timeline and adds the two streams without changing sequence length. We also build EvA-Perception, a large-scale open-source training set with about 54K event-ordered captions (150 h) and about 500K QA pairs. Under a unified zero-shot protocol, EvA achieves the best open-source Perception scores on MMAU, MMAR, and MMSU, and improves over Kimi-Audio-7B on all reported metrics, with the largest gains on perception-heavy splits. These results support the evidence-first hypothesis: stronger audio understanding depends on preserving acoustic evidence before reasoning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the EvA architecture, which effectively addresses the evidence bottleneck in large audio language models through a dual-path approach that preserves acoustic evidence before reasoning. This work significantly advances the state of the art in audio understanding, demonstrating the importance of upstream perception in achieving better performance in complex acoustic scenes.
Methodology
The proposed EvA architecture introduces a dual-path system that effectively combines speech and non-speech audio processing through hierarchical evidence aggregation and non-compressive, time-aligned fusion. This innovative approach addresses the identified evidence bottleneck in existing LALMs, allowing for improved acoustic evidence retention before reasoning. The methodology is well-structured, with clear explanations of the dual-path architecture and the training process, including the creation of the EvA-Perception dataset. However, while the architecture is novel, it builds on existing frameworks such as Whisper and CED-Base, which may limit the perceived originality of the approach.
Results & evaluation
The experiments conducted demonstrate the effectiveness of the EvA architecture across multiple benchmarks (MMAU, MMAR, MMSU, and CochlScene), with significant improvements in perception-heavy tasks. The results are compelling, showcasing the model's ability to outperform existing systems, particularly in preserving acoustic evidence. The use of a unified zero-shot protocol for evaluation adds rigor to the experimental design. However, the paper could benefit from more detailed comparisons with a broader range of existing models to contextualize the improvements more effectively.
Reproducibility
The paper provides a comprehensive overview of the training strategy, including hyperparameters and the two-stage training process. However, the lack of specific implementation details and code availability may hinder full reproducibility. The authors mention that the EvA model is open-source, which is a positive aspect, but the absence of a direct link to the repository limits accessibility for other researchers.
Limitations
The paper acknowledges several limitations, including the focus on English-only captions in the EvA-Perception dataset and the need for more systematic multilingual evaluation. Additionally, the temporal reasoning capabilities are constrained by the soft event boundaries in the training data, and the model's performance on music analysis is limited by the lack of expert-level concepts. These limitations suggest areas for future work and improvement.
Broader impact
The advancements made by the EvA architecture have significant implications for audio understanding applications, particularly in complex acoustic environments. By improving the retention of acoustic evidence, the model can enhance various tasks, including audio captioning, event detection, and question answering. The open-source nature of the dataset and model also encourages further research and development in the field, potentially leading to more robust audio processing systems in diverse applications. The main contribution of this paper is the introduction of the EvA architecture, which effectively addresses the evidence bottleneck in large audio language models through a dual-path approach that preserves acoustic evidence before reasoning. This work significantly advances the state of the art in audio understanding, demonstrating the importance of upstream perception in achieving better performance in complex acoustic scenes.
Author of this paper? Send a correction.