TAC: Timestamped Audio Captioning
Abstract
Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC$\rightarrow$LLM and TAC-V$\rightarrow$LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of TAC, a model that produces temporally grounded audio captions with low hallucination rates, significantly advancing the state of audio understanding. This work addresses critical shortcomings in existing models and presents a robust framework for future research in audio and audio-visual reasoning.
Methodology
The paper introduces the Timestamped Audio Captioner (TAC) and its extension TAC-V, which leverage a synthetic data pipeline to create temporally grounded audio descriptions. The methodology is innovative, utilizing a dynamic acoustic mixer to generate complex audio mixtures with precise temporal annotations, addressing the limitations of traditional audio captioning methods that often rely on sparse annotations. The approach of separating the audio captioning task from reasoning tasks through a cascade with a text-only LLM is particularly noteworthy, allowing for independent scaling and improved performance.
Results & evaluation
The experiments are comprehensive, comparing TAC against state-of-the-art models on multiple benchmarks, including MMAU-Pro, MMSU, and others. The results demonstrate significant improvements in temporal grounding and reduced hallucination rates, validating the effectiveness of the proposed methods. The ablation studies provide insights into the importance of various components of the model, further strengthening the findings.
Reproducibility
The paper provides sufficient detail regarding the implementation, including the use of specific architectures (Qwen2-Audio) and training procedures (LoRA). However, the reliance on synthetic data may introduce challenges in replicating results in real-world scenarios, which could limit reproducibility.
Limitations
The authors acknowledge limitations related to the synthetic data approach, including potential biases and a sim-to-real gap. Additionally, the model may struggle with fine-grained musical precision, which could affect its applicability in certain contexts.
Broader impact
The work has significant implications for improving the reliability of audio understanding systems, particularly in safety-critical applications and accessibility tools for the hearing impaired. However, the potential for misuse in surveillance contexts raises ethical considerations that must be addressed. The main contribution of this paper is the development of TAC, a model that produces temporally grounded audio captions with low hallucination rates, significantly advancing the state of audio understanding. This work addresses critical shortcomings in existing models and presents a robust framework for future research in audio and audio-visual reasoning.
Author of this paper? Send a correction.