Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding
Abstract
Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. While this task is crucial for real-world audio understanding and LALM adaptation, it is bottlenecked by data scarcity. Few large-scale resources provide open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields promising performance gains on both the DESED SED benchmark and AEGBench, an independent difficulty-stratified benchmark we release. Our results show that automatically constructed data, coupled with interval-aware reward function design, is an effective data-side route to expanding the temporal localization capability of LALMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel and effective pipeline for automatically constructing open-vocabulary audio event grounding data, combining synthetic data for cold-start SFT with real-world pseudo-labels for RL fine-tuning, thereby significantly advancing the temporal localization capabilities of Large Audio-Language Models.
Methodology
The paper proposes Auto-AEG, a pipeline for constructing open-vocabulary audio event grounding (AEG) data without manual annotation. The methodology is two-fold: Stage 1 uses programmatic synthesis of synthetic audio clips with exact ground-truth intervals to perform Supervised Fine-Tuning (SFT) on Large Audio-Language Models (LALMs). Stage 2 uses a multi-model pipeline (Gemini for labeling, PE A-Frame for localization, CLAP for label cleaning) to generate pseudo-labels on real-world audio, which are then used to fine-tune the model via Group Relative Policy Optimization (GRPO). The GRPO reward function is specifically designed to be interval-aware, combining F1-IoU, format rewards, and penalties for over-prediction. The approach is technically sound, leveraging the noise tolerance of RL to correct imperfect pseudo-labels, a strategy increasingly common in LLM alignment but novel in this specific audio grounding context. The separation of semantic identification (LALM) and temporal localization (frame-level model) in the annotation pipeline is a clever engineering choice to mitigate individual model weaknesses.
Results & evaluation
The authors evaluate their method on a newly introduced benchmark, AEGBench, which is difficulty-stratified and independently annotated. They also report results on the standard DESED SED benchmark. The results show significant improvements in mIoU and Event F1 over zero-shot baselines. The ablation studies effectively demonstrate the contribution of each stage: SFT provides a necessary cold-start, and GRPO further refines the temporal boundaries. The analysis of hard cases (e.g., long duration, polyphonic overlap) provides valuable insights into the current limitations of LALMs. The comparison with existing benchmarks highlights the unique contribution of AEGBench in terms of difficulty stratification and real-world relevance. The results on DESED confirm that the grounding capabilities transfer to closed-set SED tasks.
Reproducibility
The paper provides detailed descriptions of the data construction pipeline, including the synthesis parameters, the multi-model annotation steps, and the GRPO training hyperparameters. The code for the synthesis and annotation pipeline is not explicitly linked in the text provided, but the methodology is described with sufficient detail for reproduction. The release of AEGBench is a significant contribution to reproducibility in this area. The use of open-source models (Qwen-Omni, PE A-Frame, CLAP) facilitates replication.
Limitations
The synthetic data in Stage 1 suffers from a domain gap compared to real-world audio (room acoustics, source interaction). The pseudo-labels in Stage 2 are inherently noisy, although the RL component is designed to be robust to this. The performance on long-duration events is limited by the encoder window size of the underlying LALMs. The reliance on external models (Gemini, PE A-Frame) for annotation introduces potential failure modes if these models make errors, although the paper argues that the RL process mitigates this. The benchmark, while high-quality, is still limited in size (3,427 items) compared to some large-scale audio datasets.
Broader impact
This work addresses a critical bottleneck in audio understanding: the lack of large-scale, open-vocabulary temporal grounding data. By providing a scalable data construction pipeline and a new benchmark, it enables further research into LALMs for precise audio localization. This has implications for applications such as audio search, video editing, and assistive technologies for the visually impaired. The emphasis on open-vocabulary capabilities aligns with the broader trend towards more flexible and generalizable AI systems. This paper presents a novel and effective pipeline for automatically constructing open-vocabulary audio event grounding data, combining synthetic data for cold-start SFT with real-world pseudo-labels for RL fine-tuning, thereby significantly advancing the temporal localization capabilities of Large Audio-Language Models.
Author of this paper? Send a correction.