ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding
Abstract
Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
Methodology
The paper proposes ATTEST, a reinforcement learning framework based on Group Relative Policy Optimization (GRPO) specifically tailored for long-audio temporal grounding. The core novelty lies in the design of three distinct reward signals: (1) a Counterfactual Audio-Grounding (CAG) reward that penalizes predictions surviving the removal of acoustic evidence (via silence or corruption), effectively acting as a hallucination penalty; (2) an Attention Density Grounding reward that encourages the model to attend to the specific audio tokens corresponding to the time spans it claims in its output; and (3) a token-wise fine-grained advantage mechanism that assigns credit at the segment level rather than the sequence level, which is crucial for long, dense captions. This approach avoids modifying the base model's architecture or vocabulary, making it a plug-and-play improvement for existing Large Audio Language Models (LALMs). The methodology is sound, addressing the specific failure modes of LALMs on long audio (hallucination and temporal drift) with targeted RL interventions.
Results & evaluation
The authors introduce EgoSCAPE, a new benchmark derived from Ego4D with 700 manually verified audio clips up to 5 minutes long, featuring dense captions, sound sources, and temporal grounding queries. This is a significant contribution as existing benchmarks often lack dense annotations for long-form audio. Experiments show that ATTEST outperforms strong baselines (including Gemini 3.1 Pro, GPT-4o Audio, and specialized models like TimeAudio and AudioFlamingo 3) on both the new EgoSCAPE benchmark and existing datasets (Clotho-Moment, CASTELLA). The ablation studies convincingly demonstrate the individual contribution of each reward component, particularly the CAG reward in reducing hallucinations. The results on general audio understanding benchmarks (MMAR, LongAudioBench) confirm that the method does not cause catastrophic forgetting, which is a common issue with temporal grounding fine-tuning.
Reproducibility
The paper provides a detailed description of the benchmark creation pipeline, including the use of BEATs for initial detection and LLMs for verification, followed by human verification. The RL training details (LoRA, GRPO parameters, reward weights) are specified. However, the code and the EgoSCAPE benchmark itself are not explicitly linked in the provided text (though a reproducibility statement is made). The reliance on specific proprietary models (Gemini, GPT) for baselines limits full reproducibility of those comparisons, but the open-source base models (Qwen, AudioFlamingo) allow for partial reproduction.
Limitations
The benchmark is limited to 5-minute clips, which is still shorter than some real-world long-form audio scenarios (e.g., hours-long podcasts). The method relies on the availability of attention weights, which may not be accessible in all proprietary or optimized inference engines. The counterfactual reward requires generating a reference model output on corrupted audio, adding computational overhead during training. The paper acknowledges that the method is currently evaluated on audio-only tasks and does not extend to video or omni-modal temporal grounding in the main experiments.
Broader impact
This work addresses a critical bottleneck in deploying LALMs for real-world applications where audio is long and unstructured. By providing a rigorous benchmark (EgoSCAPE) and a training framework (ATTEST) that reduces hallucinations without architectural changes, it enables more reliable temporal reasoning in audio AI. This has implications for accessibility tools, video indexing, and autonomous agents that need to understand and locate specific events in long audio streams. The paper introduces EgoSCAPE, a manually verified long-audio benchmark, and ATTEST, a GRPO-based reinforcement learning framework that significantly improves temporal grounding and reduces hallucinations in Large Audio Language Models without altering their architecture. The combination of a novel counterfactual reward for evidence grounding, an attention-density reward, and fine-grained token-wise advantages represents a substantial methodological advance in audio temporal understanding, supported by rigorous experiments demonstrating state-of-the-art performance on both new and existing benchmarks.
Author of this paper? Send a correction.