Source-linked AI summary
ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA
Aviad Dahan, Moran Yanuka, Noa Kraicer, Lior Wolf, Raja Giryes
TL;DR
Existing personalization methods separate visual identity from audio, limiting unified control over appearance, voice, scene acoustics, and speaking style. ID-LoRA jointly personalizes both modalities with reference identity inputs and text conditioning, achieving stronger audio-visual coherence and identity preservation than cascaded baselines.
Problem
Existing visual personalization methods preserve appearance but treat video and audio separately, while unified generation aims to preserve visual and vocal identity in novel contexts.
Method
ID-LoRA adapts a unified audio-video diffusion backbone with In-Context LoRA, using negative temporal positions and identity guidance to preserve reference identity during joint generation.
Results
ID-LoRA outperforms cascaded baselines and competes with closed-source models in speaker similarity and lip synchronization, while achieving stronger audio prompt adherence on difficult cross-environment splits.
Takeaways & Limitations
Joint audio-visual generation supports more coherent outputs by allowing text conditioning and cross-modal attention to govern appearance, voice, environment sounds, and speaking style together.
Takeaways & Limitations
Face similarity and lip-synchronization metrics can be confounded by reduced lip motion, so synchronization should be evaluated alongside intelligibility and identity.
Abstract
from arXiv · showhide
Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.
1 Introduction
ID-LoRA addresses unified audio-visual personalization by jointly generating a subject’s appearance and voice in novel scenes. It combines a shared audio-video backbone with identity-transfer components and evaluates synchronization, identity preservation, and prompt adherence.
- Motivation: Visual personalization methods preserve appearance but commonly produce silent content or separate audio that cannot fully represent multimodal identity.Human identity includes both visual appearance and vocal characteristics such as timbre, cadence, and prosody.
- Problem: ID-LoRA jointly synthesizes a specific subject’s appearance and voice in a shared latent space for entirely new scenes.A text prompt can simultaneously control visual content, environmental acoustics, and speaking style.
- Approach: ID-LoRA adapts the LTX-2 unified audio-video diffusion backbone with In-Context LoRA for zero-shot identity-preserving generation.The model attends to a first-frame image and short audio reference while generating coherent outputs in novel settings.
- Approach: Negative temporal positions separate reference audio tokens from target tokens in RoPE space while preserving their internal temporal structure.This architectural component addresses confusion between reference and generation tokens.
- Approach: Identity guidance amplifies speaker-specific features during inference by contrasting predictions with and without the reference signal.The method applies a classifier-free-guidance variant to the audio stream.
- Evaluation: ID-LoRA is evaluated with automatic metrics and human studies covering speaker similarity, lip synchronization, and audio prompt adherence.The evaluation includes A/B preference and MOS studies across diverse environments.
2 Related Work
Related work spans joint audio-video generation, visual identity personalization, voice and scene synthesis, talking-head animation, and audio-visual editing. ID-LoRA differs by enabling fully generative cross-video personalization with controllable voice, appearance, and environment.
- Joint Audio-Video Generation: Shared audio-video backbones model video and audio latents jointly for synchronized synthesis and related generation, editing, and inpainting tasks.Examples include AV-DiT, JavisDiT, Apollo, and Ovi.
- Visual Identity Personalization: Visual identity personalization evolved from optimization-based methods to encoder-based diffusion conditioning and parameter-efficient LoRA adaptation.In-Context LoRA transfers identity by concatenating reference and target latents for self-attention without per-subject optimization.
- Audio Identity and Scene Control: Voice-cloning systems preserve speaker identity, while text-to-audio models generate environmental soundscapes but do not preserve a specific speaker’s identity.These capabilities remain separated across the cited audio-identity and scene-control approaches.
- Audio-Driven Talking Heads: Audio-driven talking-head methods synchronize lips to existing speech but generally do not generate audio or model broader environmental sound.They typically animate tightly cropped portraits from pre-existing driving audio.
- Editing and Dubbing: Audio-visual editing and dubbing methods modify existing videos, inheriting their speaker configuration and acoustic environment rather than synthesizing new contexts.ID-LoRA instead performs fully generative personalization from a reference audio clip, first-frame image, and text prompt.
- Identity-Aware Synthesis: ID-LoRA uses approximately 3K training pairs, contrasting with related identity-aware systems trained on approximately 1M or 100M pairs.Its emphasis is text control over environment sounds and speaking style.
3 Method
ID-LoRA adapts a joint audio-video diffusion backbone for zero-shot identity transfer, conditioning generation on a first-frame image, reference audio, and text. Its design separates reference from target audio temporally and strengthens speaker identity during denoising.
- Core Architecture: ID-LoRA adapts a pretrained joint audio-video diffusion backbone with In-Context LoRA for identity-preserving generation in novel contexts.The shared backbone processes audio and video latents together, enabling joint transfer of appearance and acoustic identity.
- Core Architecture: Reference audio and target latents are concatenated along the sequence dimension while first-frame conditioning anchors visual identity and supports synchronized audio.The model remains free to generate prompt-guided visuals while maintaining coherence with the audio.
- Negative Temporal Positions: Negative temporal positions place reference audio tokens in a separate RoPE region from target tokens without disrupting the reference’s internal temporal structure.Reference positions occupy [−T_ref, 0), whereas target positions occupy [0, T_target].
- Training Objective: ID-LoRA trains with a standard diffusion denoising objective on target latents conditioned on a same-speaker reference audio clip and text describing the target scene.The denoising network includes LoRA parameters and attends to the reference during training.
- Identity-Guided Inference: Identity guidance contrasts reference-conditioned and unconditioned predictions to amplify speaker-specific features during inference.The method applies classifier-free guidance to reference audio, strengthening vocal timbre, speaking rhythm, and pronunciation while text controls scene and environment sounds.
4 Experimental Settings
The experiments train and evaluate ID-LoRA on CelebV-HQ and TalkVid, using curated held-out speakers and comparisons against cascaded voice-video pipelines and Kling 2.6 Pro. Evaluation covers identity, synchronization, prompt adherence, and transcription quality across same-video and cross-video settings.
- Datasets and Splits: ID-LoRA is trained on CelebV-HQ and TalkVid with separate checkpoints and evaluated on 120 curated video pairs from 63 held-out speakers.CelebV-HQ includes easy same-video and hard cross-video splits; TalkVid provides an additional evaluation dataset.
- Baselines: The comparison includes three cascaded voice-cloning and WAN2.2 pipelines plus the closed-source Kling 2.6 Pro model.The cascaded systems pair CosyVoice 3.0, VoiceCraft, or ElevenLabs with WAN2.2; ElevenLabs provides built-in emotion and style control.
- Evaluation Metrics: Evaluation measures speaker similarity, face similarity, lip synchronization, audio prompt adherence, and WER against ground-truth target videos.Speaker and face similarity use embedding cosine similarity; synchronization uses LSE-D and LSE-C, while CLAP measures audio prompt adherence.
- Implementation: ID-LoRA uses rank-128 LoRA trained for 6,000 steps on a single NVIDIA H100 GPU with batch size 4 and learning rate 2×10^-4.First-frame conditioning is applied with probability 0.9 during training.
- Implementation: Inference uses 30 denoising steps with video CFG 3.0, audio CFG 7.0, identity guidance 4.0, and AV-bimodal CFG 3.0.Generated videos have 1024×1024 resolution, 121 frames, and 25fps.
5 Results
ID-LoRA consistently improves audio-visual personalization across automatic, ablation, and human evaluations, particularly when reference and target environments differ. Its unified generation preserves speaker identity and synchronization while enabling prompt-controlled environmental audio and physically grounded sound interaction.
- Main comparisons: ID-LoRA outperforms cascaded baselines on speaker similarity and lip synchronization across evaluation settings.On the hard split, the speaker-similarity gap over the best cascaded baseline widens from +0.063 on easy to +0.086 on hard.
- Cross-dataset generalization: ID-LoRA maintains strong cross-dataset transfer, with the CelebV-HQ checkpoint reaching 0.595 speaker similarity on TalkVid without fine-tuning.This score is only 0.004 below the in-domain checkpoint and surpasses other baselines.
- Metric interpretation: Face similarity alone is an incomplete measure because reduced lip motion can inflate face embeddings and low lip-sync distance can coincide with unintelligible speech.LTX-Zeroshot has the lowest LSE-D but also low LSE-C confidence and WER above 2.3 on CelebV-HQ.
- Design ablations: The full design improves synchronization, intelligibility, and identity over ablated variants on the CelebV-HQ hard split.Removing identity guidance lowers speaker similarity from 0.477 to 0.438, while standard positional encoding changes LSE-D from 8.50 to 9.02 and WER from 0.113 to 0.252.
- Human preference: 73.1% of annotators preferred ID-LoRA to Kling 2.6 Pro for voice similarity, while 65.2% preferred it for speech manners.ID-LoRA was also preferred for environment sounds, at 54.8% versus 20.7%.
- Human evaluation: ID-LoRA achieves higher interaction MOS than Kling 2.6 Pro, winning 8 of 10 physical-interaction scenarios.Overall MOS is 3.05 versus 2.90, with fewer “Bad” ratings: 15.3% versus 23.3%.
6 Conclusions
ID-LoRA jointly personalizes appearance and vocal identity in a unified audio-visual model, using cross-modal attention and text conditioning across both modalities. Experiments report stronger identity preservation, prompt adherence, and audio-visual coherence than cascaded pipelines and competitive commercial models.
- ID-LoRA jointly synthesizes a subject’s appearance and vocal identity through a unified audio-visual personalization method.
- Negative temporal positions separate reference tokens from target tokens, while identity guidance enhances speaker preservation during generation.
- ID-LoRA outperforms cascaded baselines and competes with state-of-the-art closed-source models in speaker similarity and lip synchronization.
- Human evaluations favor ID-LoRA for speech-manner matching, appropriate environment sounds, and physically grounded audio integration.
A.1 Dataset Details
The study uses CelebV-HQ and TalkVid, preprocesses clips into speaker- and caption-aware training data, and constructs reference-target pairs with identity correspondence controls. Evaluation includes held-out speaker splits spanning same-video, cross-video, and quiet-to-environment conditions, alongside cascaded and unified baselines.
- ID-LoRA is trained on CelebV-HQ and TalkVid using separate checkpoints.
- CelebV-HQ contains diverse talking-head videos with varied acoustic environments, whereas TalkVid spans 7,729 speakers across 15+ languages with less acoustic diversity.
- The preprocessing pipeline filters unsuitable clips, trims silence, generates visual and audio captions, retains English samples, and clusters speakers using face embeddings.
- Training pairs balance same-video and cross-video examples for CelebV-HQ and filter pairs below 0.45 speaker similarity, while reference audio is source-separated into clean speech.
- Evaluation uses held-out speakers in easy same-video, hard cross-video, and quiet-to-environment settings, with 35 hard pairs, 36 easy pairs, and 49 TalkVid pairs.
- Comparisons include three two-stage voice-cloning and video-generation pipelines plus Kling 2.6 Pro as a unified commercial baseline.
A.3 Evaluation Metric Details
The evaluation measures visual and vocal identity, audio-visual alignment, prompt adherence, and speech intelligibility using embedding similarities, SyncNet metrics, CLAP, and WER. Lip-motion analysis additionally examines a systematic relationship between facial similarity scores and speech animation.
- Speaker similarity is measured as cosine similarity between generated and target audio embeddings using WavLM Large and ECAPA-TDNN.
- Face similarity averages ArcFace embeddings from five uniformly sampled frames and compares them with the target face.
- Lip synchronization uses SyncNet LSE-D, where lower is better, and LSE-C, where higher is better, to quantify audio-visual alignment.
- Lip-motion analysis reports the standard deviation of normalized inner-mouth opening over time and finds that lower lip movement is associated with higher FaceSim.
- Audio prompt adherence uses CLAP similarity between generated audio and combined environment-sound and speaking-style descriptions.
- Speech intelligibility is evaluated with Whisper-large-v3 transcripts and word error rate against ground-truth transcripts, with lower WER indicating clearer speech.
B.1 Face Similarity and Lip Motion Analysis
Face similarity and lip-motion analysis reveal a metric tension: methods with less mouth movement can receive higher FaceSim despite less active speech articulation. Identity-guidance scaling improves speaker similarity but introduces trade-offs in synchronization and intelligibility.
- Identity Guidance Scale: At identity-guidance scale sid = 2, speaker similarity falls to 0.459 versus 0.477 at sid = 4, confirming the importance of identity guidance for voice preservation.
- Identity Guidance Scale: Increasing sid beyond 4 raises speaker similarity to 0.485 at sid = 10 but degrades lip synchronization and speech intelligibility.
- Identity Guidance Scale: The default identity-guidance scale is sid = 4 because larger values trade stronger speaker similarity for weaker lip synchronization and ASR metrics.
C Additional Qualitative Results
The studies use controlled human evaluations to compare ID-LoRA with baselines across voice similarity, environment sounds, and speech manners. Annotation procedures randomize presentation and incorporate safeguards intended to reduce bias and enforce complete viewing.
- Human evaluation design: Two A/B preference tests compare ID-LoRA against Kling 2.6 Pro and ElevenLabs + WAN2.2 on a hard cross-video split.The split contains 35 video pairs across 8 held-out CelebV-HQ speakers, with 9 annotators per pair.
- Human evaluation design: Annotators rate voice similarity, environment sounds, and speech manners using randomized A/B comparisons.Each question presents a reference or description appropriate to the evaluated axis.
- Bias prevention: A/B assignment is randomized independently by pair and question, while a cyclic Latin square prevents annotators from seeing the same pair across question types.Watch-gating also requires complete video viewing before ratings are enabled.
- Study materials: The evaluation instructions and example rating cards are provided for the human studies.Figures 6 and 7 document the annotator-facing materials.
D.2 Environment Sound Interaction MOS Study
The MOS study examines whether unified audio-visual generation can produce sounds matching physical interactions depicted in novel scenes. ID-LoRA shows stronger aggregate ratings and consistency than Kling 2.6 Pro, although the overall difference is not statistically significant.
- Motivation: Unified generation is needed because cascaded voice-cloning systems produce clean speech without non-speech sounds or visual timing cues for acoustic events.The study therefore targets physically grounded sounds such as thuds, claps, and breaking glass.
- Study design: The MOS study tests ten interaction scenarios, including object impacts, music, animal sounds, and environmental actions.Scenes are constructed by inserting scenario-appropriate objects into speakers’ first frames before generation.
- Results: 3.05 vs. 2.90 overall MOS favors ID-LoRA over Kling 2.6 Pro, with ID-LoRA winning on 8 of 10 scenarios.The largest gains occur for bird chirping in a park (+1.51), clapping (+0.42), and box dropping (+0.38); Kling leads on guitar (−1.40) and drums (−0.69).
- Results: 15.3% vs. 23.3% “Bad” ratings indicate fewer poor outputs for ID-LoRA than Kling 2.6 Pro.ID-LoRA also exhibits lower variance, suggesting more consistent generation quality.
- Caveat: The overall difference is not statistically significant (p = 0.093, Cohen’s d = 0.115).The authors frame the study as an exploratory first step toward evaluating diegetic consistency.
- Broader impact: The capability carries risks of non-consensual impersonation, misleading media, and unauthorized likeness cloning.Proposed safeguards include invisible watermarks or provenance metadata and explicit consent.