AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Models
Abstract
Rapid advances in audio-video (AV) generation have enabled high-fidelity synthesis with synchronized sound, particularly for human-related scenarios involving speech and interactions. Yet evaluation for AV generation remains at an early stage, with only a few coarse-grained benchmarks for human-related scenarios and relying on limited preset evaluations with generic multimodal LLMs, leading to inaccurate assessments of model capabilities. To address these issues, we introduce AVBench, a fully automated benchmark tailored for human-centric AV generation. AVBench is built on two key designs for comprehensive and accurate evaluation: (i) Human-centric and fine-grained metrics. AVBench integrates ten evaluation dimensions designed for human-centered real-world scenarios, covering visual quality, audio quality, and multi-level consistency across modalities. These practical metrics capture human-related details that existing benchmarks often overlook. (ii) Specialized evaluators via preference learning. To address the lack of specialized training data, we construct large-scale supervision by transforming real-world videos into diverse training pairs with controlled perturbations. After fine-tuning on this high-quality dataset, the evaluators learn to reliably detect subtle cross-modal inconsistencies. Crucially, instead of producing discrete textual judgment, AVBench derives continuous evaluation scores from the model's prediction confidence on binary decisions. This probabilistic scoring mechanism enables a more reliable assessment than traditional VQA-style evaluation and aligns closely with human judgment. Taken together, AVBench offers automated evaluation for AV generation, demonstrates strong potential for data filtering, and serves as a differentiable reward signal for Reinforcement Learning from Human Feedback (RLHF).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AVBench, a novel automated evaluation benchmark for audio-video generative models that incorporates fine-grained human-centric metrics and specialized evaluators, significantly improving the assessment of model capabilities in generating coherent and realistic audio-video outputs. The comprehensive methodology and rigorous experimental validation position this work as a significant advancement in the field, addressing critical gaps in existing evaluation frameworks.
Methodology
The methodology presented in this paper is robust and innovative, focusing on the creation of AVBench, a specialized evaluation benchmark for audio-video generative models. The authors effectively address the limitations of existing benchmarks by introducing fine-grained, human-centric metrics and specialized evaluators trained on a large dataset with hard negatives. The use of supervised fine-tuning (SFT) on multimodal models to derive continuous evaluation scores is particularly noteworthy, as it enhances the precision of the evaluation process. Additionally, the hierarchical difficulty evaluation structure allows for a comprehensive assessment across varying levels of complexity, which is a significant advancement in the evaluation of generative models.
Results & evaluation
The experiments conducted are thorough and well-structured, demonstrating the effectiveness of the proposed evaluators against state-of-the-art models. The use of a diverse dataset and the clear distinction between normal and hard subsets provide a solid basis for evaluating model performance. The results indicate significant improvements in alignment and consistency detection, showcasing the practical applicability of AVBench in real-world scenarios. The correlation analysis with human judgments further validates the effectiveness of the proposed framework.
Reproducibility
The paper provides sufficient detail regarding the methodology and experimental setup, which aids in reproducibility. However, the absence of a publicly available code repository limits the ability for others to replicate the exact experiments. The authors do mention a project page, but without code or data access, full reproducibility remains a challenge.
Limitations
While the framework is comprehensive, it primarily focuses on short-form video clips (5-12 seconds), which may limit its applicability to longer video formats. Additionally, the reliance on human-centric scenarios may not fully encompass the broader range of audio-video generation tasks. Future work could explore extending the framework to longer videos and other domains beyond human-centric content.
Broader impact
The introduction of AVBench has the potential to significantly influence the field of audio-video generation by providing a standardized evaluation framework that aligns closely with human perception. This could lead to improved model development and optimization, ultimately enhancing the quality of generated content in various applications, including entertainment, education, and virtual reality. The main contribution of this paper is the introduction of AVBench, a novel automated evaluation benchmark for audio-video generative models that incorporates fine-grained human-centric metrics and specialized evaluators, significantly improving the assessment of model capabilities in generating coherent and realistic audio-video outputs. The comprehensive methodology and rigorous experimental validation position this work as a significant advancement in the field, addressing critical gaps in existing evaluation frameworks.
Author of this paper? Send a correction.