Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
Abstract
Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a practical and effective method for adding voice cloning to T2AV models with minimal architectural changes, achieving state-of-the-art speaker similarity at the cost of some transcription accuracy.
Methodology
The paper proposes a method to add voice cloning capabilities to existing Text-to-Audio-Video (T2AV) diffusion models. The core innovation is architectural minimalism: adding a single zero-initialized linear layer to inject a global speaker embedding (via FiLM) and prepending reference audio latents to the input sequence. This allows the model to condition on a reference voice without retraining the entire backbone from scratch. The approach leverages the existing self-attention mechanisms for latent prepending and introduces a new modulation path for the speaker embedding. The methodology is sound and builds logically on existing practices in TTS (speaker embeddings) and diffusion (latent conditioning), but the combination within a T2AV context is a novel application. The zero-initialization ensures stability during the fine-tuning phase.
Results & evaluation
The evaluation is conducted on a benchmark of 674 speaker-text pairs across 30 speakers from the VCTK corpus. The authors compare their method against five strong baselines, including dedicated TTS models (Qwen3-TTS, XTTS-v2, IndexTTS2) and a joint audio-video model (NAVA). Results show that the proposed method achieves higher speaker-encoder cosine similarity (SECS) across three verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer) compared to baselines. However, this comes at the cost of higher Word Error Rates (WER), indicating a trade-off between speaker fidelity and transcription accuracy. The paper also demonstrates that the audio-only path can be run independently for a ~30x speed-up with minimal loss in speaker similarity. The "no regression" analysis shows that fine-tuning does not degrade the base model's performance on text-to-audio generation.
Reproducibility
The paper provides detailed implementation details, including hyperparameters (learning rates, optimizer settings, diffusion steps), architecture dimensions, and training schedules. The use of open-source components (VCTK, Qwen3-TTS encoder, ECAPA-TDNN, etc.) aids reproducibility. However, the base T2AV model (Kandinsky 5.0) is described as having an "internal corpus" and the specific checkpoints are not publicly linked in the text provided, which may hinder exact replication. The code is not explicitly linked in the provided text.
Limitations
The primary limitation is the trade-off between speaker similarity and speech intelligibility (WER). The model tends to preserve acoustic quirks of the reference, which can lead to hallucinations or errors in the generated text, especially on short or difficult prompts. The performance is also sensitive to reference length and language matching. The method relies on the availability of a pre-trained T2AV model, limiting its applicability to models that have not been trained on such data.
Broader impact
This work enables more personalized and controllable audio-visual generation, which has applications in content creation, dubbing, and virtual avatars. However, the ability to clone voices raises significant ethical concerns regarding misuse for deepfakes or non-consensual voice synthesis. The authors should have included a more robust discussion on these risks and potential mitigation strategies. The paper presents a practical and effective method for adding voice cloning to T2AV models with minimal architectural changes, achieving state-of-the-art speaker similarity at the cost of some transcription accuracy.
Author of this paper? Send a correction.