VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
Abstract
Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
Methodology
The paper proposes VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. The methodology addresses the misalignment of existing metric-based rewards by introducing a human-preference dataset (VAPref-10K) and a three-stage training pipeline: easy cold-start SFT, hard preference alignment via rejection sampling, and dimension-wise reinforcement learning (GRPO). The approach is technically sound, leveraging recent advances in LLM-based reward modeling (CoT, GRPO) and adapting them to the multimodal video-audio domain. The introduction of dimension-wise rewards to provide denser signals than binary labels is a logical and valuable contribution to the field of preference optimization.
Results & evaluation
The experimental evaluation is comprehensive. The authors construct a new benchmark (VA-Judger-Bench) with in-domain and out-of-domain splits. They demonstrate that VA-Judger outperforms standard metrics (VideoAlign, AudioBox, CLIP, etc.) in predicting human preferences. Crucially, they show that using VA-Judger for post-training (RLHF-style) the LTX-2 model leads to significant improvements in both automatic metrics (JavisBench) and human preference scores (62.3% win rate). The ablation studies on the training stages are well-presented.
Reproducibility
The paper provides detailed implementation details, including hyperparameters, model architectures (Qwen3-Omni backbone), and training configurations (DeepSpeed ZeRO 3, BF16). The code repository is linked. The dataset construction process is described, though the specific human annotation protocol details are somewhat condensed. Overall, reproducibility is high.
Limitations
The reliance on Gemini 3.1 Pro for generating initial CoT responses and the rejection sampling step introduces a dependency on a specific, powerful proprietary model, which may limit the accessibility of the data generation pipeline for some researchers. Additionally, the dataset size (10.3K pairs) is relatively small compared to large-scale web datasets, though sufficient for preference modeling. The generalization to closed-source models (Sora, Veo) is promising but relies on the availability of their outputs for evaluation.
Broader impact
This work significantly advances the field of multimodal generation by providing a robust, human-aligned reward signal for joint video-audio generation. It addresses the critical "reward hacking" problem inherent in metric-based optimization. The dataset and benchmark will serve as valuable resources for future research in multimodal alignment and evaluation. VA-Judger introduces a novel chain-of-thought reward modeling framework for joint video-audio generation, effectively aligning generative models with human preferences through dimension-wise reinforcement learning and a newly constructed human-preference dataset.
Author of this paper? Send a correction.