Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception
Abstract
Fine-grained perception of multimodal information is critical for advancing human-AI interaction. With recent progress in audio-visual technologies, Omni Language Models (OLMs), capable of processing audio and video signals in parallel, have emerged as a promising paradigm for achieving richer understanding and reasoning. However, their capacity to capture and describe fine-grained details remains limited explored. In this work, we present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark. We first identify an inherent "co-growth" between detail and hallucination in current OLMs. To address this, we propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data. Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception. Under the cascade evaluation protocol, Audio-Captioner achieves the best performance on MMAU and MMAR among all open-source models, surpassing Gemini 2.5 Flash and delivering performance comparable to Gemini 2.5 Pro. On existing detailed captioning benchmarks, Omni-Captioner sets a new state-of-the-art on VDC and achieves the best trade-off between detail and hallucination on the video-SALMONN 2 testset. Given the absence of a dedicated benchmark for omni detailed perception, we design Omni-Cloze, a novel cloze-style evaluation for detailed audio, visual, and audio-visual captioning that ensures stable, efficient, and reliable assessment. Experimental results and analysis demonstrate the effectiveness of Omni-Detective in generating high-quality detailed captions, as well as the superiority of Omni-Cloze in evaluating such detailed captions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive framework for omni detailed perception, significantly advancing the field of multimodal AI by introducing innovative methodologies and benchmarks that address critical challenges in fine-grained perception tasks.
Methodology
The paper introduces a novel data generation pipeline, Omni-Detective, which employs an agentic approach to generate detailed multimodal captions while minimizing hallucinations. This is a significant advancement as it addresses the critical challenge of balancing detail and hallucination in captioning tasks. The two-stage training process for the Audio-Captioner and Omni-Captioner models is well-structured, allowing for effective cross-modal integration and fine-grained perception. The introduction of the Omni-Cloze benchmark for evaluation further enhances the methodology by providing a reliable assessment framework for detailed captioning.
Results & evaluation
The experiments are comprehensive, showcasing the performance of the proposed models across various benchmarks. The results indicate that both models outperform existing state-of-the-art methods in terms of detail coverage and hallucination rates. The use of cascade evaluation protocols adds rigor to the assessment, demonstrating the models' capabilities in real-world applications. However, the paper could benefit from additional comparisons with a broader range of existing models to further validate its claims.
Reproducibility
The paper provides sufficient details regarding the training data, model architecture, and evaluation metrics, which aids reproducibility. The open-source nature of the project, with a GitHub repository available, enhances the potential for other researchers to replicate the findings and build upon the work.
Limitations
The paper acknowledges limitations in fully capturing all types of hallucinations, particularly those that may be contextually accurate but not aligned with the ground truth. This is a significant challenge in multimodal perception tasks and highlights the need for continued research in this area. Additionally, the reliance on specific datasets may limit the generalizability of the results.
Broader impact
The advancements presented in this paper have the potential to significantly enhance human-AI interaction by improving the accuracy and richness of multimodal understanding. The methodologies and benchmarks developed could pave the way for more sophisticated applications in assistive technologies, content generation, and interactive systems. The focus on reducing hallucination rates is particularly relevant for applications requiring high factual accuracy. The paper presents a comprehensive framework for omni detailed perception, significantly advancing the field of multimodal AI by introducing innovative methodologies and benchmarks that address critical challenges in fine-grained perception tasks.
Author of this paper? Send a correction.