PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation
Abstract
Text-to-audio-video (T2AV) generation underpins a wide range of applications demanding realistic audio-visual content, including virtual reality, world modeling, gaming, and filmmaking. However, existing T2AV models remain incapable of generating physically plausible sounds, primarily due to their limited understanding of physical principles. To situate current research progress, we present PhyAVBench, a challenging audio physics-sensitivity benchmark designed to systematically evaluate the audio physics grounding capabilities of existing T2AV models. PhyAVBench comprises 1,000 groups of paired text prompts with controlled physical variables that implicitly induce sound variations, enabling a fine-grained assessment of models' sensitivity to changes in underlying acoustic conditions. We term this evaluation paradigm the Audio-Physics Sensitivity Test (APST). Unlike prior benchmarks that primarily focus on audio-video synchronization, PhyAVBench explicitly evaluates models' understanding of the physical mechanisms underlying sound generation, covering 6 major audio physics dimensions, 4 daily scenarios (music, sound effects, speech, and their mix), and 50 fine-grained test points, ranging from fundamental aspects such as sound diffraction to more complex phenomena, e.g., Helmholtz resonance. Each test point consists of multiple groups of paired prompts, where each prompt is grounded by at least 20 newly recorded or collected real-world videos, thereby minimizing the risk of data leakage during model pre-training. Both prompts and videos are iteratively refined through rigorous human-involved error correction and quality control to ensure high quality. We argue that only models with a genuine grasp of audio-related physical principles can generate physically consistent audio-visual content. We hope PhyAVBench will stimulate future progress in this critical yet largely unexplored domain.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of PhyAVBench, a novel benchmark designed to evaluate the audio physics grounding capabilities of T2AV models through a systematic and controlled approach. This work significantly advances the field by addressing a critical gap in existing evaluation frameworks and promoting the development of more realistic audio-visual generation systems.
Methodology
The methodology presented in PhyAVBench is robust and innovative, employing a systematic approach to benchmark T2AV models against a comprehensive set of audio-physical dimensions. The use of controlled variables to assess models' sensitivity to physical changes is a significant advancement over existing benchmarks, which often overlook the underlying physical principles. The integration of human refinement and quality control in prompt generation and video collection enhances the reliability of the benchmark. The introduction of the Audio-Physics Sensitivity Test (APST) and the Contrastive Physical Response Score (CPRS) as evaluation metrics is particularly noteworthy, as it allows for a nuanced understanding of models' performance in relation to physical realism.
Results & evaluation
While the paper primarily focuses on the benchmark design and data curation, the proposed framework is well-structured to facilitate future experimental evaluations of various T2AV models. The comprehensive coverage of audio-physical dimensions and scenarios ensures that the benchmark can be applied to a wide range of models, from commercial to academic. However, the paper lacks detailed results from model evaluations, which would have provided concrete evidence of the benchmark's effectiveness and the models' capabilities.
Reproducibility
The paper outlines a clear data curation pipeline and methodology for creating the benchmark, which enhances reproducibility. However, the absence of specific implementation details or code availability limits the ability to fully reproduce the results. Future work should consider providing access to the benchmark dataset and evaluation scripts to foster reproducibility within the research community.
Limitations
One limitation of the study is the focus on the benchmark's design without presenting empirical results from model evaluations. This leaves questions about the practical applicability of the benchmark unanswered. Additionally, while the benchmark covers a wide range of physical phenomena, it may not encompass all possible scenarios encountered in real-world audio-visual generation, which could limit its generalizability.
Broader impact
PhyAVBench has the potential to significantly influence the development of T2AV models by providing a rigorous framework for evaluating their understanding of audio-physical principles. This can lead to advancements in applications such as virtual reality, gaming, and filmmaking, where realistic audio-visual content is crucial. By emphasizing the importance of physical realism in audio generation, the benchmark encourages researchers to develop models that are not only perceptually convincing but also physically grounded. The main contribution of this paper is the introduction of PhyAVBench, a novel benchmark designed to evaluate the audio physics grounding capabilities of T2AV models through a systematic and controlled approach. This work significantly advances the field by addressing a critical gap in existing evaluation frameworks and promoting the development of more realistic audio-visual generation systems.
Author of this paper? Send a correction.