ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA
Abstract
Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of ID-LoRA, a unified framework for audio-video personalization that allows for joint generation of visual and auditory content based on text prompts, significantly advancing the capabilities of generative models in the multimedia domain. The technical contributions, particularly in methodology and experimental validation, position this work as a notable advancement in the field of machine learning and generative media.
Methodology
The proposed ID-LoRA framework innovatively integrates audio and video generation through a unified model, addressing the limitations of existing cascaded approaches. The introduction of negative temporal positions and identity guidance are significant methodological advancements that enhance the model's ability to preserve speaker identity while allowing for flexible text-based control over both audio and visual outputs. The use of a joint audio-video diffusion backbone (LTX-2) is a strong choice, leveraging the latest advancements in diffusion models to achieve high-quality generative results.
Results & evaluation
The experiments are robust, featuring a comprehensive evaluation protocol that includes both automatic metrics and human preference studies. The paper demonstrates significant improvements over state-of-the-art models, including commercial solutions, in speaker similarity and lip synchronization. The use of multiple datasets (CelebV-HQ and TalkVid) and the careful construction of evaluation splits (easy and hard) provide a thorough assessment of the model's performance across different conditions. The human evaluation metrics, including A/B preference tests and Mean Opinion Scores, add credibility to the findings.
Reproducibility
The paper provides detailed implementation information, including training parameters, dataset preprocessing, and evaluation metrics, which enhances reproducibility. However, the reliance on specific datasets and the mention of proprietary models (like Kling 2.6 Pro) could pose challenges for complete reproducibility in diverse contexts.
Limitations
While the model shows strong performance, it may still struggle with extreme variations in acoustic environments or highly dynamic visual scenes that were not extensively tested. The model's reliance on a relatively small training dataset (~3K pairs) raises questions about its generalization capabilities in broader applications. Additionally, the ethical implications of generating realistic audio-visual content without consent are not fully addressed.
Broader impact
The potential applications of ID-LoRA are significant, including personalized content creation, multilingual dubbing, and accessibility tools. However, the technology also poses risks related to misuse, such as deepfakes and unauthorized impersonation. The authors acknowledge these risks and suggest mitigations, emphasizing the need for ethical considerations in deployment. The main contribution of this paper is the introduction of ID-LoRA, a unified framework for audio-video personalization that allows for joint generation of visual and auditory content based on text prompts, significantly advancing the capabilities of generative models in the multimedia domain. The technical contributions, particularly in methodology and experimental validation, position this work as a notable advancement in the field of machine learning and generative media.
Author of this paper? Send a correction.