The Attention Triangle in Audio-Video Models
Abstract
Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage. We study these models by probing and analyzing the ``attention triangle,'' comprising the three cross-attention edges connecting the text, audio, and video streams, and examine how semantic information is routed across modalities during generation. Our analysis reveals that routing along the audio-video edge is bidirectional: audio can influence video generation, while video can influence audio generation. This edge is shaped by biases encoded in the model's parameters and emerges as a major contributor to leakage: when prompts are in tension with learned priors, cross-modal interactions may override the intended conditioning and reroute semantics toward visually canonical but incorrect outcomes. These effects suggest that semantic artifacts arise not merely from attention spreading beyond its intended target, but from structured, bias-driven interactions along specific pathways. Building on this perspective, we extract attention-derived signals that expose how semantics are distributed and grounded across modalities, and use them as a diagnostic tool to both analyze and deliberately incur leakage under controlled conditions. This enables us to probe the internal dynamics of cross-modal routing and isolate the role of individual interactions. We further leverage these signals to guide inference-time interventions that encourage more consistent cross-modal alignment. Extensive experiments support our analysis and demonstrate improved semantic grounding while preserving generation quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies the "Attention Triangle" as a structural cause of semantic leakage in audio-video diffusion models and proposes inference-time interventions to correct source attribution and appearance errors. By analyzing the bidirectional flow of information between text, audio, and video streams, the authors demonstrate that leakage is not random but driven by specific biases in the audio-video attention edge, enabling targeted fixes that improve cross-modal consistency without retraining.
Methodology
The paper introduces the "Attention Triangle" framework to analyze cross-modal semantic leakage in audio-video diffusion models. The core contribution is a diagnostic and intervention method that decomposes the cross-attention mechanisms into three edges: Text-Video, Text-Audio, and Audio-Video. The authors identify the Audio-Video edge as a critical source of bias, where learned priors (e.g., speech associated with human faces) override explicit text prompts. They propose extracting attention-derived signals to map these biases and using them for inference-time steering. The methodology is sound, leveraging the internal structure of Diffusion Transformers (DiT) to manipulate attention maps without retraining. The formalization of the "bias matrices" and the distinction between direct and indirect routing paths provides a rigorous theoretical basis for the observed failures.
Results & evaluation
The experiments focus on the LTX-2 model, a state-of-the-art open-source audio-video generator. The evaluation demonstrates the ability to isolate specific leakage types (appearance vs. source attribution) and correct them via targeted steering. The qualitative results (e.g., the pirate/parrot example) are compelling and clearly illustrate the decoupling of visual and audio errors. However, the paper relies heavily on qualitative visual/audio examples and ablation studies on specific prompts. There is a lack of large-scale quantitative benchmarks (e.g., FID, CLAP scores, or human preference studies) to generalize the effectiveness of the intervention across diverse datasets. The scope is limited to one specific model architecture (LTX-2), which may limit the generalizability of the findings to other audio-video models.
Reproducibility
The paper provides sufficient detail on the attention manipulation techniques, describing how to extract and modify attention maps. However, specific hyperparameters for the steering strength, layer selection, and the exact implementation of the "bias matrix" extraction are not fully detailed in the provided text. The reliance on the specific LTX-2 model makes reproduction dependent on access to that specific codebase and weights.
Limitations
The primary limitation is the lack of quantitative evaluation metrics. The improvements are demonstrated qualitatively, which is subjective. Additionally, the method is tested on a single model (LTX-2), so it is unclear if the "Attention Triangle" dynamics and the proposed interventions generalize to other architectures like Sora or Veo. The computational overhead of inference-time steering is not explicitly discussed.
Broader impact
This work contributes to the interpretability of multimodal generative models, offering a new lens (the attention triangle) to understand cross-modal interactions. The proposed inference-time interventions could be adopted by developers of audio-video models to improve alignment without retraining. It highlights a fundamental issue in multimodal generation: the conflict between learned statistical priors and explicit user intent, providing a pathway to mitigate such conflicts. The paper identifies the "Attention Triangle" as a structural cause of semantic leakage in audio-video diffusion models and proposes inference-time interventions to correct source attribution and appearance errors. By analyzing the bidirectional flow of information between text, audio, and video streams, the authors demonstrate that leakage is not random but driven by specific biases in the audio-video attention edge, enabling targeted fixes that improve cross-modal consistency without retraining.
Author of this paper? Send a correction.