ACE-Cap: Active Evidence Acquisition via Agentic Co-Evolution for Long-Paragraph Fine-Grained Audio Captioning
Abstract
Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners remain passive one-shot generators: once a detail is overlooked, they cannot identify the evidence gap, query the audio for targeted information, or decide when sufficient evidence has been collected. We formulate this task as active evidence acquisition and introduce Agentic Co-Evolution for Captioning (ACE-Cap). The framework uses multi-turn interaction between a Composer and an Instruct model to form a closed evidence-acquisition loop. A Captioner first produces an initial description. Conditioned on this description and the interaction history, a text-only Composer asks targeted questions about unresolved acoustic attributes, while an audio-conditioned Instruct model provides grounded answers. The Composer then decides when to terminate and synthesizes the accumulated evidence into a final caption. ACE-Cap trains these roles through a unified gold-to-prediction reward derived from fixed, gold-grounded multiple-choice questions and a frozen caption-only judge. For credit assignment in variable-length interactions, LOOP-GRPO replaces the trajectory-wide scalar advantage with span-aligned signals: leave-one-out contributions of individual questions to the accumulated evidence, a quality-cost utility for stopping, and an evidence-preservation utility for final synthesis. Role-wise warm-up followed by alternating Composer and Instruct optimization keeps each update a well-defined single-policy problem while allowing the roles to co-evolve. ACE-Cap thus turns captioning from passive one-shot generation into an adaptive process that learns what evidence to acquire, when to stop, and how to preserve it in a long-paragraph caption.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ACE-Cap introduces a novel agentic framework for fine-grained audio captioning that leverages active evidence acquisition and a specialized reinforcement learning algorithm (LOOP-GRPO) to improve factual accuracy and detail recovery, representing a significant step forward in adaptive audio understanding systems.
Methodology
The paper proposes ACE-Cap, a framework that reformulates long-paragraph fine-grained audio captioning as an active evidence acquisition problem. The core innovation lies in the agentic loop involving a text-only Composer and an audio-conditioned Instruct model. The Composer identifies missing acoustic details by querying the Instruct model, which provides grounded answers. This addresses the limitation of passive one-shot captioners. The training methodology introduces LOOP-GRPO (Leave-One-Out Per-turn GRPO), which assigns credit to individual questions based on their counterfactual contribution to the final evidence state, rather than using a trajectory-wide scalar advantage. This is a significant methodological contribution to reinforcement learning for agentic systems, specifically addressing the credit assignment problem in variable-length interaction trajectories. The use of a frozen caption-only judge and gold-grounded multiple-choice questions for reward modeling is a practical approach to aligning the agent with fine-grained factual accuracy.
Results & evaluation
The experimental evaluation is comprehensive, covering multiple benchmarks (Omni-Cloze, MMAR, MMAU, MMSU). The paper demonstrates that ACE-Cap outperforms strong open-source baselines and remains competitive with proprietary models like GPT-4o Audio and Gemini 2.5 Pro. The ablation studies effectively isolate the contributions of the different training stages (Captioner GRPO, Instruct GRPO, Composer LOOP-GRPO, and alternating co-evolution), showing monotonic improvement. The analysis of the Composer's query behavior and the impact of query budget provides valuable insights into the agent's decision-making process. The results support the claim that active evidence acquisition improves fine-grained captioning accuracy.
Reproducibility
The paper provides detailed descriptions of the methodology, including the reward function, the LOOP-GRPO algorithm, and the training pipeline. The use of publicly available models (Qwen2.5-Omni-7B, Qwen3-8B) and datasets (ASID-1M) enhances reproducibility. However, the reliance on a proprietary model (Gemini 3.1 Pro) for generating gold captions and MCQs during the offline stage may introduce some variability, although the core training and evaluation are performed on open-source components. The code is not explicitly mentioned as open-sourced in the text, which is a minor drawback for immediate reproducibility.
Limitations
The paper acknowledges limitations such as the restriction to audio-only inputs and the fixed caption-as-evaluation protocol. The reliance on a fixed question bank derived from gold captions means the reward signal cannot detect unsupported claims outside the predefined set. Additionally, the alternating optimization strategy, while stabilizing training, may not fully capture the complex interdependencies between the Composer and Instruct models compared to joint optimization, although the authors argue it prevents instability. The performance gap with the strongest proprietary models (e.g., Gemini 3.1 Pro on MMAR) suggests there is still room for improvement in handling very complex audio scenes.
Broader impact
This work contributes to the field of audio-language models by introducing an agentic paradigm for fine-grained understanding. The techniques developed for active evidence acquisition and credit assignment in multi-turn interactions have broader applications in other domains requiring iterative reasoning and information gathering, such as visual question answering or document QA. The emphasis on factual accuracy and evidence preservation aligns with the growing need for reliable and interpretable AI systems. ACE-Cap introduces a novel agentic framework for fine-grained audio captioning that leverages active evidence acquisition and a specialized reinforcement learning algorithm (LOOP-GRPO) to improve factual accuracy and detail recovery, representing a significant step forward in adaptive audio understanding systems.
Author of this paper? Send a correction.