FoleyBench: A Benchmark For Video-to-Audio Models
Abstract
Video-to-audio generation (V2A) is of increasing importance in domains such as film post-production, AR/VR, and sound design, particularly for the creation of Foley sound effects synchronized with on-screen actions. Foley requires generating audio that is both semantically aligned with visible events and temporally aligned with their timing. Yet, there is a mismatch between evaluation and downstream applications due to the absence of a benchmark tailored to Foley-style scenarios. We find that 74% of videos from past evaluation datasets have poor audio-visual correspondence. Moreover, they are dominated by speech and music, domains that lie outside the use case for Foley. To address this gap, we introduce FoleyBench, the first large-scale benchmark explicitly designed for Foley-style V2A evaluation. FoleyBench contains 5,000 (video, ground-truth audio, text caption) triplets, each featuring visible sound sources with audio causally tied to on-screen events. The dataset is built using an automated, scalable pipeline applied to in-the-wild internet videos from YouTube-based and Vimeo-based sources. Compared to past datasets, we show that videos from FoleyBench have stronger coverage of sound categories from a taxonomy specifically designed for Foley sound. Each clip is further labeled with metadata capturing source complexity, UCS/AudioSet category, and video length, enabling fine-grained analysis of model performance and failure modes. We benchmark several state-of-the-art V2A models, evaluating them on audio quality, audio-video alignment, temporal synchronization, and audio-text consistency. Samples are available at: https://gclef-cmu.org/foleybench
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents FoleyBench, a novel benchmark for evaluating video-to-audio generation models specifically for Foley sound effects. It addresses critical gaps in existing datasets and provides valuable insights into model performance, paving the way for future advancements in the field.
Methodology
The methodology employed in constructing FoleyBench is robust, utilizing an automated multi-stage pipeline that effectively filters and curates video clips to ensure high-quality audio-visual correspondence. The approach includes scene detection, audio filtering using YAMNet, and audiovisual filtering with Gemini 2.5 Pro, which collectively ensure that the dataset is tailored for Foley sound effects. The systematic categorization of clips with metadata enhances the dataset's usability for fine-grained analysis of model performance. However, the reliance on automated processes may introduce biases if the filtering models are not perfectly calibrated.
Results & evaluation
The experiments conducted on various state-of-the-art V2A models provide a comprehensive evaluation of their performance on the newly established benchmark. The paper presents a clear comparison between FoleyBench and existing datasets like VGGSound, highlighting the inadequacies of the latter for Foley-style evaluations. The inclusion of multiple metrics (audio quality, audio-video alignment, temporal synchronization, and audio-text consistency) allows for a nuanced understanding of model capabilities. The results reveal significant insights into the strengths and weaknesses of different models, particularly in relation to specific sound categories and conditions.
Reproducibility
The paper provides sufficient details regarding the dataset construction and evaluation metrics, which aids in reproducibility. However, the exact implementation details of the models evaluated are not fully disclosed, which may hinder complete reproducibility of the results. The availability of the dataset and benchmark scores is a positive aspect, as it allows other researchers to validate findings and build upon this work.
Limitations
One limitation is the potential bias introduced by the automated filtering process, which may not capture all relevant audio-visual relationships accurately. Additionally, while the dataset is extensive, the reliance on internet-sourced videos may introduce variability in audio quality and content diversity. The paper also notes that current models struggle with high-fidelity audio generation for discrete sounds and long-form videos, indicating areas for further research.
Broader impact
The introduction of FoleyBench has significant implications for the fields of audio generation and sound design, particularly in creative industries such as film and gaming. By providing a dedicated benchmark for Foley sound effects, it encourages the development of more sophisticated V2A models that can produce high-quality, contextually appropriate audio. This work could lead to advancements in AR/VR applications and enhance user experiences in multimedia content creation. The paper presents FoleyBench, a novel benchmark for evaluating video-to-audio generation models specifically for Foley sound effects. It addresses critical gaps in existing datasets and provides valuable insights into model performance, paving the way for future advancements in the field.
Author of this paper? Send a correction.