Do Joint Audio-Video Generation Models Understand Physics?
Abstract
Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency? We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench tests models across three scene categories: Steady State, Event Transition, and Environment Transition. It covers physics-grounded subcategories drawn from real-world scenes, plus Anti-AV-Physics prompts that deliberately request physically inconsistent audio-video behavior. Each generation is evaluated along five dimensions: visual semantic adherence, audio semantic adherence, visual physical commonsense, audio physical commonsense, and cross-modal physical commonsense. Across three proprietary and four open-source models, we find that Seedance 2.0 performs best overall, but all models remain far from robust physical understanding. Performance drops sharply on event-driven and environment-driven transitions, and even strong proprietary systems collapse on Anti-AV-Physics prompts. We further introduce AV-Phys Agent, a ReAct-style evaluator that combines a multimodal language model with deterministic acoustic measurement tools, producing rankings that closely align with human ratings. Our results identify cross-modal physical consistency and transition-driven scene dynamics as key open challenges for joint audio-video generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the establishment of a comprehensive benchmark for evaluating physical commonsense in joint audio-video generation models, which addresses a critical gap in the current evaluation landscape. The innovative methodology and thorough experimental evaluation provide valuable insights into the limitations of existing models and pave the way for future advancements in the field.
Methodology
The paper introduces AV-Phys Bench, a novel benchmark for evaluating physical commonsense in joint audio-video generation models. The methodology is robust, employing a structured evaluation rubric that assesses five dimensions of performance across three scene categories. The use of both human evaluation and an automated evaluator (AV-Phys Agent) that integrates multimodal reasoning with deterministic audio measurement tools is particularly innovative. This dual approach enhances the reliability of the assessments and provides a comprehensive framework for understanding model performance beyond mere perceptual quality.
Results & evaluation
The experiments conducted are thorough, evaluating seven models across various categories and dimensions. The results reveal significant gaps in physical commonsense understanding among current models, particularly in transition scenarios. The performance metrics are well-defined, and the analysis is detailed, providing insights into the strengths and weaknesses of the evaluated models. The findings are significant, highlighting the challenges in achieving physical consistency in audio-video generation.
Reproducibility
The paper provides sufficient details regarding the evaluation setup, including the datasets and scoring mechanisms. The availability of the dataset and the code repository enhances reproducibility. However, the reliance on specific models for evaluation may limit the generalizability of the findings to other models not included in the study.
Limitations
The paper acknowledges limitations such as the focus on English prompts and the binary nature of the evaluation rubric, which may not capture the nuances of model performance. Additionally, the study is constrained to eight-second clips, which may not represent longer or more complex scenarios effectively.
Broader impact
The introduction of AV-Phys Bench has the potential to significantly influence the development of joint audio-video generation models by providing a clear framework for assessing physical commonsense. This could lead to improvements in model architectures and training methodologies, ultimately enhancing the applicability of these models in real-world scenarios, such as virtual environments and educational content. The main contribution of this paper is the establishment of a comprehensive benchmark for evaluating physical commonsense in joint audio-video generation models, which addresses a critical gap in the current evaluation landscape. The innovative methodology and thorough experimental evaluation provide valuable insights into the limitations of existing models and pave the way for future advancements in the field.
Author of this paper? Send a correction.