TAC2: Timestamped Audio Captioning as composable functions
Abstract
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
TAC2 introduces a scalable framework for timestamped audio captioning by fusing weak teacher predictions and eliciting latent paralinguistic capabilities, resulting in a composable, temporally grounded representation that enhances both audio understanding and controllable generation.
Methodology
The paper proposes a two-stage framework, TAC2, for timestamped audio captioning. Stage 1 addresses the data scarcity problem for fine-grained temporal grounding by programmatically fusing predictions from weak teachers (sound event detectors, diarizers, captioners) using a time-registered IoU matching algorithm. This creates a structured, speaker-attributed timeline without expensive manual annotation. The model is trained using an Instruction-Input-Output (IIO) completion format, which allows for composable functions where outputs of one task can condition subsequent tasks. Stage 2 extends this by eliciting latent paralinguistic capabilities (stress, emotion, etc.) from a frozen backbone using lightweight LoRA specialists trained on small datasets, then distilling these into a unified student model. The methodology is sound, leveraging the strengths of weak supervision and parameter-efficient fine-tuning to build a complex, structured representation.
Results & evaluation
The authors validate the model in two settings: audio understanding (cascading with frozen LLMs) and audio generation (paired supervision). In understanding, TAC2 achieves strong results on MMSU and MSU-Bench, outperforming baselines like TAC and Qwen3Omni-Captioner, particularly in perception tasks. The generation experiments demonstrate zero-shot control over event timing and speaker attributes. The evaluation is comprehensive, covering both the utility of the structured representation for reasoning and its application in generative tasks.
Reproducibility
The paper provides detailed descriptions of the fusion algorithm, the IIO format, and the specialist training procedure. However, specific hyperparameters, dataset sizes for the weak supervision corpus, and the exact implementation details of the IoU matching thresholds are not fully specified in the provided text, which may hinder exact reproduction. The use of standard tools like LoRA and common benchmarks aids reproducibility.
Limitations
The reliance on weak teachers means the quality of the fused timeline is bounded by the accuracy of the individual detectors and diarizers. Errors in the weak supervision stage can propagate to the final model. Additionally, the "latent capability elicitation" approach, while efficient, may not capture all nuances of paralinguistic features if they are not strongly encoded in the backbone. The paper is truncated, so the full extent of the generation results and ablations is not visible.
Broader impact
This work has significant potential for applications requiring precise audio understanding, such as automated video captioning, audio search, and accessible media. By decoupling perception from reasoning, it enables more robust and controllable audio pipelines. The composable nature of the IIO format makes it adaptable to various downstream tasks. TAC2 introduces a scalable framework for timestamped audio captioning by fusing weak teacher predictions and eliciting latent paralinguistic capabilities, resulting in a composable, temporally grounded representation that enhances both audio understanding and controllable generation.
Author of this paper? Send a correction.