VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categories
Abstract
Video-to-Audio (V2A) generation is essential for immersive multimedia experiences, yet its evaluation remains underexplored. Existing benchmarks typically assess diverse audio types under a unified protocol, overlooking the fine-grained requirements of distinct audio categories. To address this gap, we propose VidAudio-Bench, a multi-task benchmark for V2A evaluation with four key features: (1) Broad Coverage: It encompasses four representative audio categories - sound effects, music, speech, and singing - under both V2A and Video-Text-to-Audio (VT2A) settings. (2) Extensive Evaluation: It comprises 1,634 video-text pairs and benchmarks 11 state-of-the-art generation models. (3) Comprehensive Metrics: It introduces 13 task-specific, reference-free metrics to systematically assess audio quality, video-audio consistency, and text-audio consistency. (4) Human Alignment: It validates all metrics through subjective studies, demonstrating strong consistency with human preferences. Experimental results reveal that current V2A models perform poorly in speech and singing compared to sound effects. Our VT2A results further highlight a fundamental tension between instruction following and visually grounded generation: stronger visual conditioning improves video-audio alignment, but often at the cost of generating the intended audio category. These findings establish VidAudio-Bench as a comprehensive and scalable framework for diagnosing V2A systems and provide new insights into multimodal audio generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the establishment of VidAudio-Bench, a comprehensive benchmark for evaluating V2A and VT2A systems, which systematically addresses the limitations of existing evaluation methodologies and provides valuable insights into the performance of current models. The technical contributions, including the innovative evaluation metrics and the extensive dataset, position this work as a significant advancement in the field of audio generation.
Methodology
The paper introduces VidAudio-Bench, a novel benchmarking framework for Video-to-Audio (V2A) and Video-Text-to-Audio (VT2A) generation that addresses the limitations of existing evaluation methods by providing a multi-task benchmark with task-specific metrics. The methodology is robust, featuring a comprehensive dataset of 1,634 video-text pairs across four audio categories, and it employs both objective and subjective evaluation metrics, including human alignment studies to validate the proposed metrics. The introduction of a zero-information-leak design for VT2A evaluation is particularly innovative, allowing for a clearer assessment of visual understanding without relying on textual shortcuts.
Results & evaluation
The experimental evaluation is thorough, benchmarking 11 state-of-the-art models across various tasks and dimensions. The results reveal significant insights into the performance of current V2A models, particularly their struggles with speech and singing tasks. The paper effectively uses a variety of metrics to assess audio quality, video-audio consistency, and text-audio consistency, providing a comprehensive view of model performance. The correlation analysis with human evaluations further strengthens the credibility of the findings.
Reproducibility
The paper provides detailed descriptions of the dataset construction, evaluation metrics, and experimental setup, which are essential for reproducibility. However, the absence of publicly available code or datasets limits the ability for other researchers to replicate the results directly. The methodology is well-documented, but the lack of a project URL or demo limits broader accessibility.
Limitations
One limitation is the reliance on subjective human evaluations, which, while valuable, can introduce variability and bias. Additionally, the dataset may not cover all possible scenarios in V2A generation, potentially limiting the generalizability of the findings. The paper also notes a fundamental tension between instruction following and visually grounded generation, indicating that there are inherent challenges in achieving optimal performance across all tasks.
Broader impact
The development of VidAudio-Bench has significant implications for the field of multimodal audio generation, providing a structured framework that can guide future research and model development. By highlighting the challenges faced by current V2A models, the paper encourages further exploration into improving audio generation systems, which can enhance applications in entertainment, accessibility, and human-computer interaction. The main contribution of this paper is the establishment of VidAudio-Bench, a comprehensive benchmark for evaluating V2A and VT2A systems, which systematically addresses the limitations of existing evaluation methodologies and provides valuable insights into the performance of current models. The technical contributions, including the innovative evaluation metrics and the extensive dataset, position this work as a significant advancement in the field of audio generation.
Author of this paper? Send a correction.