Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
Abstract
We present Kandinsky 6.0 Video, a family of foundation diffusion models for synchronized text-to-audio-video generation, comprising Kandinsky 6.0 Video Lite (3B parameters) and Kandinsky 6.0 Video Pro (29B parameters). Both models generate 5-second video clips with synchronized 44 kHz audio, including lip-sync, in text-to-audio-video (T2AV) and image-to-audio-video (I2AV) modes; a built-in super-resolution model raises the output resolution to Full-HD (1920times1080). Building on the video generation capabilities of Kandinsky 5.0, Kandinsky 6.0 Video employs a dual-stream CrossDiT architecture that connects a pretrained video stream and a newly trained audio stream through bidirectional cross-attention for temporal and semantic alignment. Our continuous pretraining strategy first trains the audio stream from scratch on large-scale audio corpora and then trains both streams jointly on paired audio-video data while preserving unimodal fidelity; pretraining is followed by supervised fine-tuning, reinforcement-learning-based post-training, and distillation. In side-by-side human evaluation, Kandinsky 6.0 Video Pro clearly outperforms its predecessor, Kandinsky 5.0 Video Pro, and remains competitive with leading audio-video generation models, particularly in speech quality. To accelerate open research and deployment in multimedia generation, we release the code, model checkpoints, and diffusers integration under the MIT license.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Kandinsky 6.0 Video, an open-source foundation model for synchronized text-to-audio-video generation that employs a dual-stream CrossDiT architecture and a continuous pretraining strategy to achieve competitive performance with proprietary systems while releasing all assets under an MIT license.
Methodology
The paper proposes a dual-stream CrossDiT architecture extending the Kandinsky 5.0 video model with a newly trained audio stream. The core methodological contribution is the "continuous pretraining" strategy: the audio stream is first trained independently on large-scale audio corpora, then connected to the pretrained video stream via bidirectional cross-attention, and finally fine-tuned jointly on paired audio-video data. This approach aims to preserve unimodal fidelity while achieving temporal and semantic alignment. The pipeline includes supervised fine-tuning, reinforcement learning (adapted from OmniNFT), and distillation (D-Flow and adversarial refinement) to reduce inference steps to 10 NFEs. The architecture leverages existing components (Hunyuan VAE for video, MMAudio for audio) and standard text encoders (Qwen2.5-VL, CLIP). While the modular design is practical, the architectural novelty is moderate, as it follows established patterns in recent joint audio-video generation models (e.g., LTX-2, Ovi). The primary innovation lies in the specific training schedule and the integration of RL for post-training in this multimodal context.
Results & evaluation
The evaluation relies on VABench metrics and side-by-side human evaluations. The paper claims that Kandinsky 6.0 Video Pro outperforms its predecessor (Kandinsky 5.0) and open-source competitors like LTX 2.5, and remains competitive with proprietary systems (Veo, Sora) particularly in speech quality. However, the provided text lacks detailed quantitative tables comparing specific metrics (e.g., FID, FVD, Sync Confidence, Lip-Sync Score) against baselines. The reliance on "side-by-side human evaluation" without providing the full statistical breakdown or sample size details in the excerpt limits the rigor of the experimental assessment. The claim of "competitive with leading audio-video generation models" is strong but requires the missing quantitative data to be fully verified.
Reproducibility
The paper explicitly states that code, model checkpoints, and diffusers integration are released under the MIT license. This is a significant positive factor for reproducibility. The detailed description of the data processing pipeline, including specific filtering metrics (LSA, Desync, DOVER, Q-Align) and captioning models used, provides a clear roadmap for data preparation. However, the exact hyperparameters for the RL stage and the specific details of the "model soup" combination are not fully detailed in the excerpt, which may hinder exact reproduction of the post-training phase.
Limitations
The generated clips are limited to 5 seconds, which is shorter than some proprietary competitors. The base resolution is SD, requiring a separate super-resolution step for Full-HD, which adds computational overhead and potential artifact risk. The model is heavily optimized for Russian and English captions, which may limit its generalizability to other languages. The reliance on a specific VAE (Hunyuan) and audio autoencoder (MMAudio) ties the model to these specific latent spaces, potentially limiting architectural flexibility.
Broader impact
The release of a 29B parameter open-source model for synchronized audio-video generation is highly impactful for the research community, enabling reproducible studies on multimodal alignment, lip-sync, and audio-visual consistency. It bridges the gap between closed-source commercial systems and open-source research models. The inclusion of a Russian Cultural Code dataset highlights a specific niche application for culturally aware generation, which is a unique aspect of this work. The paper presents Kandinsky 6.0 Video, an open-source foundation model for synchronized text-to-audio-video generation that employs a dual-stream CrossDiT architecture and a continuous pretraining strategy to achieve competitive performance with proprietary systems while releasing all assets under an MIT license.
Author of this paper? Send a correction.