Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generation
Abstract
Video-to-Audio generation has made remarkable strides in automatically synthesizing sound for video. However, existing evaluation metrics, which focus on semantic and temporal alignment, overlook a critical failure mode: models often generate acoustic events, particularly speech and music, that have no corresponding visual source. We term this phenomenon Insertion Hallucination and identify it as a systemic risk driven by dataset biases, such as the prevalence of off-screen sounds, that remains completely undetected by current metrics. To address this challenge, we first develop a systematic evaluation framework that employs a majority-voting ensemble of multiple audio event detectors. We also introduce two novel metrics to quantify the prevalence and severity of this issue: IH@vid (the fraction of videos with hallucinations) and IH@dur (the fraction of hallucinated duration). Building on this, we propose Posterior Feature Correction, a novel training-free inference-time method that mitigates IH. PFC operates in a two-pass process: it first generates an initial audio output to detect hallucinated segments, and then regenerates the audio after masking the corresponding video features at those timestamps. Experiments on several mainstream V2A benchmarks first reveal that state-of-the-art models suffer from severe IH. In contrast, our PFC method reduces both the prevalence and duration of hallucinations by over 50\% on average, without degrading, and in some cases even improving, conventional metrics for audio quality and temporal synchronization. Our work is the first to formally define, systematically measure, and effectively mitigate Insertion Hallucination, paving the way for more reliable and faithful V2A models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper meaningfully contributes to machine learning research by defining and addressing Insertion Hallucination in Video-to-Audio generation, proposing a novel evaluation framework and mitigation strategy that enhances the fidelity of generated audio in multimedia contexts.
Methodology
The paper introduces a novel concept of Insertion Hallucination (IH) in Video-to-Audio generation, which is a significant contribution to the field. The methodology includes a systematic evaluation framework that utilizes a majority-voting ensemble of audio event detectors, along with the introduction of two new metrics (IH@vid and IH@dur) to quantify hallucinations. The proposed Posterior Feature Correction (PFC) method is innovative and effectively addresses the identified problem without requiring model retraining, showcasing a practical approach to mitigating hallucinations during inference.
Results & evaluation
The experiments are comprehensive, validating the IH detection pipeline against human annotations and assessing the effectiveness of the PFC method across multiple benchmarks. The results demonstrate a significant reduction in hallucination prevalence and duration, while maintaining or improving conventional audio quality metrics. The use of diverse datasets strengthens the findings, although the paper could benefit from more extensive ablation studies and comparisons with a wider range of existing methods.
Reproducibility
The paper provides detailed descriptions of datasets, models, and evaluation metrics, which enhances reproducibility. However, the lack of a publicly available code repository limits the ability for others to fully replicate the experiments. The authors mention that they will release code and configuration files, which is a positive step towards reproducibility.
Limitations
One limitation is the potential subjectivity in human annotations for hallucination detection, which may introduce variability in the evaluation. Additionally, while PFC shows promise, its effectiveness may vary across different types of videos and audio events, suggesting that further research is needed to generalize the approach. The paper could also expand on the implications of hallucinations in practical applications of V2A systems.
Broader impact
The work has significant implications for improving the reliability of automatic sound generation systems, which are increasingly used in various multimedia applications. By addressing a critical failure mode, the research paves the way for more trustworthy and immersive audio-visual experiences, potentially influencing future developments in the field. This paper meaningfully contributes to machine learning research by defining and addressing Insertion Hallucination in Video-to-Audio generation, proposing a novel evaluation framework and mitigation strategy that enhances the fidelity of generated audio in multimedia contexts.
Author of this paper? Send a correction.