Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Abstract
Natively training joint video-audio generation models at higher resolutions empowers them to learn richer visual details and sharper motion dynamics. However, full attention incurs quadratic cost and, as resolution increases, spreads attention over increasingly redundant tokens, diluting learning signals for informative content and disrupting pretrained priors. Existing sparse attention methods either target training-free acceleration or overlook the unique structure of joint video-audio data, where cross-modal interactions are inherently concentrated around sound-producing regions. To address this, we propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K. In particular, Prism organizes the token sequence into spatiotemporal macro-zones, enabling the attention structure to adapt to local content. For each zone, it estimates local information structure via video feature variance along the channel and feature norms from the audio-to-video cross-attention, jointly capturing how visual content varies directionally and how strongly audio influences each visual region. Based on these signals, Prism dynamically assigns a tailored block shape to each zone, applying finer partitioning along axes of rapid visual content variation and strong audio-visual coupling. This encourages tokens within each block to remain semantically coherent, allowing block-level features to capture both visual content and joint video-audio interaction patterns. Prism further adopts a hybrid block selection strategy to dynamically determine per-query sparsity. Experiments show that Prism achieves 2.5times training speedup compared to full attention, while surpassing it in generation quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Prism introduces a dynamic sparse attention mechanism that adapts block shapes based on visual variance and audio-visual coupling to enable efficient native 2K joint video-audio generation. The technical contribution is significant as it bridges the gap between efficient sparse attention and the complex, heterogeneous structure of high-resolution multimodal data, offering a 2.5x speedup with improved quality over full attention baselines.
Methodology
The paper proposes Prism, a dynamic sparse attention framework specifically designed for native 2K resolution joint video-audio generation. The core innovation lies in moving away from fixed-shape block sparsity (common in prior works like VMoBA or LongCat) to a content-adaptive approach. It partitions the token sequence into spatiotemporal macro-zones and dynamically assigns 3D block shapes based on two signals: video channel-wise variance (capturing visual complexity) and audio-to-video cross-attention norms (capturing audio-visual coupling). This allows the model to use finer blocks in regions with high motion or strong audio influence (like lips or hands) and coarser blocks in static backgrounds. Additionally, it employs a hybrid Top-k/Top-p selection strategy to adapt sparsity per query. The theoretical derivation for block shape assignment via Lagrange multipliers is sound, providing a principled way to minimize intra-block information loss.
Results & evaluation
The paper claims a 2.5x training speedup compared to full attention while surpassing it in generation quality. The comparison against baselines like LTX-2.3 (with super-resolution) and MOVA (native 2K with full attention) is relevant. The qualitative results described (stable 2K video-audio, complex human-object interactions) suggest significant practical utility. However, the provided text is truncated, so specific quantitative metrics (FID, CLAP score, human preference scores) are not fully visible, though the abstract asserts superiority. The focus on "native" training rather than post-hoc upscaling is a strong experimental design choice.
Reproducibility
The paper provides detailed mathematical formulations for the variance calculation, audio coupling strength, and block shape selection. It specifies the use of a Triton kernel with fixed 64-token tiles and the specific candidate set of block shapes. The reliance on specific hardware constraints (Triton) and the complexity of the dynamic shape assignment logic may pose challenges for reproduction without access to the specific codebase, but the algorithmic description is sufficiently detailed for implementation.
Limitations
The method is specifically tailored for the MOVA DiT architecture and 2K resolution; generalization to other resolutions or architectures may require re-tuning the macro-zone sizes and thresholds. The dynamic shape assignment adds computational overhead (calculating variances and norms), which must be carefully managed to ensure the net speedup is maintained. The paper focuses on video self-attention; the impact on cross-attention or audio branch efficiency is noted as negligible but not deeply explored.
Broader impact
This work addresses a critical bottleneck in scaling generative models to high resolutions. By enabling efficient native high-resolution training for multimodal (video-audio) models, it paves the way for more realistic and detailed generative AI applications. The integration of audio-visual coupling into the sparsity decision is a novel insight that could influence future multimodal attention mechanisms beyond just video generation. Prism introduces a dynamic sparse attention mechanism that adapts block shapes based on visual variance and audio-visual coupling to enable efficient native 2K joint video-audio generation. The technical contribution is significant as it bridges the gap between efficient sparse attention and the complex, heterogeneous structure of high-resolution multimodal data, offering a 2.5x speedup with improved quality over full attention baselines.
Author of this paper? Send a correction.