AudioSpan: Spanning the Duration and Depth of Audio Comprehension
Abstract
General audio comprehension now covers speech, sound, and music over durations from seconds to hours, driven by large audio-language models (LALMs) that are increasingly omni-modal. Yet the benchmarks that test them still rely on clips of seconds, where scores saturate and models converge; recent long-form efforts extend duration but evaluate long audio much as short clips are. We introduce AudioSpan, a benchmark that spans both duration and depth: it pairs audio from 10 minutes to over 2 hours with 3,240 questions across three cognitive levels, namely perception, understanding, and reasoning. Two paths supply the questions, differing in how question content is sourced and how ground truth is obtained. Native QA extracts questions from the audio's content, posing each as a multiple-choice item and an open-ended one graded by detailed rubrics. Anchor QA instead injects ground truth, planting acoustic anchors into the audio and building a perception-to-reasoning chain scored only to the first error. A fully automated pipeline constructs every item through structured captioning, QA generation, and adversarial critic feedback. Evaluating 12 LALMs on AudioSpan, we find the hard part comes before reasoning: distilling a few relevant facts from a long, redundant signal. This difficulty grows with audio length and falls hardest on perception, especially temporal grounding. AudioSpan is available at https://huggingface.co/datasets/holvan/AudioSpan.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
Methodology
AudioSpan introduces a meticulously designed benchmark for long-form audio comprehension, addressing critical gaps in existing LALM evaluation. The core methodology revolves around two complementary question paths: Native QA and Anchor QA, both built upon a three-level cognitive taxonomy (Perception, Understanding, Reasoning). Native QA derives questions from the audio's inherent content, offering both multiple-choice (MCQ) and open-ended (OEQ) formats, with OEQs graded by an LLM ensemble using detailed rubrics that include semantic and temporal scores. Anchor QA is particularly innovative, injecting acoustic anchors into the audio to create verifiable ground truth and constructing a perception-to-reasoning chain of MCQs, scored only up to the first error to penalize shortcutting. This dual-path approach effectively combines broad coverage of natural content with precisely verifiable ground truth. The fully automated construction pipeline is a significant strength. Phase 1 generates a structured caption from the audio, involving ASR, semantic segmentation, acoustic captioning, dependency analysis, and caption assembly into evidence records. Phase 2 then generates QA pairs, with a sophisticated multi-level quality assurance system. This system includes evidence verification against the audio, a three-level solver (blind, text-only, full access) to filter out guessable or text-only answerable items, and a novel adversarial critic feedback loop that refines question generation by learning from past failure modes. This adaptive refinement mechanism is a key technical contribution, ensuring high-quality, audio-dependent questions at scale. The detailed rubric for OEQs, incorporating temporal grounding as a critical component, is also well-conceived and crucial for evaluating true comprehension.
Results & evaluation
The experimental evaluation is comprehensive and insightful. The paper evaluates 12 LALMs (7 open-source, 5 proprietary) on AudioSpan, using a native-audio-only protocol, which is appropriate for assessing true audio understanding. The inclusion of three text-only baselines (Question-only, Transcript, Caption) provides essential context, highlighting the inherent difficulty of processing audio compared to text. The results reveal a consistent and significant finding: current LALMs struggle most with "distilling a few relevant facts from a long, redundant signal" before any reasoning can begin. This difficulty grows with audio length and disproportionately affects perception, especially temporal grounding. The analysis across cognitive levels clearly shows Perception as the weakest link, particularly when temporal localization and counting are required, and this weakness is exacerbated in open-ended formats where models must generate rather than select answers. The comparison between MCQ and OEQ performance, showing that MCQ success is a weak proxy for
Author of this paper? Send a correction.