AVC-Reward: Event-Centric Audio-Video Consistency Evaluation
Abstract
As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel event-centric, three-level consistency metric and a corresponding trainable reward model (AVC-Reward) that effectively evaluates and improves the audio-video consistency of joint generation models. By disentangling events and using a structured reasoning chain, it overcomes the limitations of traditional clip-level metrics, achieving state-of-the-art human alignment and demonstrating significant utility as a reinforcement learning signal for generative model optimization.
Methodology
The paper proposes an event-centric evaluation framework for audio-video consistency, addressing the limitations of clip-level metrics in handling multi-event compositions, multi-layer audio mixtures, and asymmetric cross-modal relations. The core contribution is a three-level coarse-to-fine metric (Global Narrative Harmony, Event-level Temporal Synchronization, Fine-grained Attribute Alignment) implemented via a trainable reward model, AVC-Reward. The model is initialized from Qwen3-Omni-Thinking and trained using a two-stage process: Supervised Fine-Tuning (SFT) on structured reasoning data generated by a multi-LLM annotation pipeline, followed by Group Relative Policy Optimization (GRPO) to align scores with human preferences. The methodology is rigorous, introducing a 39-label taxonomy for human annotation to reduce subjectivity and a label-to-score calibration mechanism using LLMs to generate continuous targets for RL training. The five-stage reasoning protocol (blind audio analysis, blind video analysis, bidirectional verification, and final judgment) ensures that the model learns to disentangle events before scoring, which is a significant methodological improvement over black-box similarity metrics.
Results & evaluation
The authors construct AVC-Benchmark, the first benchmark primarily built from model-generated samples (A2V, V2A, JAVG) to evaluate human-alignment. AVC-Reward achieves state-of-the-art performance in pairwise accuracy, SRCC, and PLCC compared to baselines like SyncNet, ImageBind, and other learned evaluators. Crucially, the paper validates the utility of the reward model by using it in an online reinforcement learning setup (OmniNFT) for the LTX-2 generation model. This results in measurable improvements in external consistency metrics and human preference scores, demonstrating that the evaluator can effectively serve as a training signal to close the "generate-evaluate-optimize" loop. The inclusion of human evaluation with blinded randomized judgments further strengthens the experimental claims.
Reproducibility
The paper provides detailed descriptions of the data construction pipeline, annotation protocol, and training hyperparameters. It references a project page and GitHub repository for prompts and code. The use of specific open-source models (Qwen3-Omni, LTX-2, OmniNFT) and public benchmarks (JavisBench, AV-Phys-Bench) enhances reproducibility. However, the reliance on specific internal models for some data generation and the complexity of the multi-LLM annotation harness may pose challenges for exact replication without access to the specific model versions and internal datasets.
Limitations
The evaluation is heavily dependent on the quality of the initial LLM-based annotations for SFT, which may propagate biases from the annotator models (Gemini 2.5 Pro, Qwen3-VL-Max). The GRPO training relies on a relatively small set of human-calibrated samples (10k), which might limit the generalization of the reward model to unseen failure modes. Additionally, the computational cost of the five-stage reasoning process during inference could be high, potentially limiting its use in real-time applications or large-scale screening without distillation.
Broader impact
This work significantly advances the field of multimodal generation by providing a robust, interpretable, and trainable metric for audio-video consistency. It addresses a critical gap in evaluating modern joint generation models that produce complex scenes. The ability to use this metric as a reinforcement learning signal offers a practical pathway for improving generative models, making it highly impactful for both research and industrial applications in video and audio synthesis. The paper introduces a novel event-centric, three-level consistency metric and a corresponding trainable reward model (AVC-Reward) that effectively evaluates and improves the audio-video consistency of joint generation models. By disentangling events and using a structured reasoning chain, it overcomes the limitations of traditional clip-level metrics, achieving state-of-the-art human alignment and demonstrating significant utility as a reinforcement learning signal for generative model optimization.
Author of this paper? Send a correction.