When Vision Speaks for Sound
Abstract
Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper meaningfully contributes to the understanding of audio-visual grounding in multimodal models, providing a systematic approach to diagnose and improve model performance through controlled interventions. The methodology and experimental results present a significant advancement in the field, addressing a critical gap in current multimodal understanding.
Methodology
The paper introduces a novel probing framework, Thud, which systematically diagnoses the reliance of multimodal models on visual cues for audio understanding. The methodology is robust, employing counterfactual audio edits to expose the Clever Hans effect, thus providing a clear mechanism to evaluate audio-visual grounding. The two-stage alignment recipe is innovative, combining intervention-derived preferences with general video data to enhance model performance without over-specialization.
Results & evaluation
The experiments are well-structured, utilizing a variety of models and intervention conditions to assess audio-visual grounding. The results demonstrate significant improvements in model performance across multiple dimensions, validating the effectiveness of the proposed methods. The use of controlled interventions allows for a clear assessment of model weaknesses and strengths.
Reproducibility
The paper provides detailed descriptions of the experimental setup, data sourcing, and annotation processes, which enhances reproducibility. However, the reliance on specific models and datasets may limit broader applicability unless similar setups are replicated.
Limitations
One limitation is the potential overfitting to the specific intervention cases, which could affect generalization to real-world scenarios. Additionally, the paper does not extensively discuss the computational resources required for the proposed methods, which may pose challenges for wider adoption.
Broader impact
The findings have significant implications for the development of future multimodal models, particularly in improving audio-visual grounding. By exposing and addressing the Clever Hans effect, this work can lead to more reliable and accurate models in applications such as video analysis, content generation, and interactive AI systems. This paper meaningfully contributes to the understanding of audio-visual grounding in multimodal models, providing a systematic approach to diagnose and improve model performance through controlled interventions. The methodology and experimental results present a significant advancement in the field, addressing a critical gap in current multimodal understanding.
Author of this paper? Send a correction.