Source-linked AI summary
UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos
Yuxuan Zhang, Haozhong Xiong, Jiayi Song, Jinpeng Yu, Yang Shi, Jiaming Liu, Ruihua Huang, Liwei Wang
TL;DR
Existing talking-video identity replacement lacks a unified streaming approach that coordinates appearance and voice while preserving source content and dynamics. UniSwap addresses this with joint audio-visual diffusion and aligned swap-and-reconstruct training, achieving stronger synchronization, competitive identity preservation, and efficient streaming generation.
Problem
Existing methods do not jointly replace visual and vocal identity in streaming talking-video generation, despite the need to preserve motion, scene content, speech, and timing.
Method
UniSwap uses a single audio-video diffusion transformer with swap-and-reconstruct supervision to transfer reference appearance and vocal timbre while preserving source content and dynamics.
Results
UniSwap achieves stronger audio-visual synchronization than evaluated cascaded systems, competitive identity preservation, 13.6 FPS streaming generation, and stable long-form inference.
Takeaways & Limitations
UniSwap demonstrates that joint audio-visual identity replacement can combine synchronized appearance and voice transfer with blockwise streaming and long-form generation.
Takeaways & Limitations
UniSwap targets single-speaker talking videos and does not support independent or arbitrary user-specified facial-expression control.
Abstract
from arXiv · showhide
Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.
1 The Chinese University of Hong Kong 2 Qwen Applications Business Group of Alibaba · 1. Introduction
UniSwap performs streaming joint audio-visual identity replacement in a single diffusion transformer, transferring reference appearance and vocal timbre while preserving source motion, background, linguistic content, and timing. Its blockwise autoregressive generator achieves 13.6 FPS on one H100 with 3-step sampling and supports stable long-form generation.
- 1 The Chinese University of Hong Kong 2 Qwen Applications Business Group of Alibaba: 13.6 FPS is achieved on 1 H100 with 3-step sampling for real-time blockwise streaming generation.The conditioned 22B model supports real-time streaming operation.
- 1. Introduction: Audio-visual identity replacement changes both a talking-video character’s appearance and voice while preserving source motion, background, linguistic content, and timing.The task supports applications including film post-production, content localization, and personalized media.
- 1. Introduction: Existing video-replacement and voice-conversion systems handle only one modality, while cascades lack a joint objective enforcing converted-speech and lip-motion consistency.The independent optimization of cascaded modules makes cross-modal consistency difficult to enforce.
- 1. Introduction: UniSwap formulates character replacement as joint audio-video generation, using cross-modal interactions to coordinate generated speech and lip motion.Training requires aligned source–target pairs differing in visual and vocal identity while matching motion, scene content, speech, and timing.
- 1. Introduction: UniSwap transfers reference appearance and vocal timbre within one audio-video diffusion transformer while autoregressively generating synchronized output in blocks.The source motion, background, and linguistic content are preserved, unlike cascaded systems.
- 1. Introduction: The swap-and-reconstruct pipeline creates temporally aligned supervision from ordinary talking videos by identity-reducing motion and converting speaker timbre without changing speech content or timing.Original appearance and voice serve as references, while each real clip remains the reconstruction target.
- 1. Introduction: UniSwap progressively adapts a bidirectional backbone into a causal three-step generator through In-context Pretraining, Conditional Streaming Adaptation, and Efficient Self-forcing DMD.Feature-RoPE Decomposition bounds positional indices while preserving cross-modal temporal alignment during long-form inference.
2. Related Work · 3. Method
UniSwap extends joint audio-visual generation to streaming character replacement, transferring reference appearance and vocal timbre while preserving source motion, speech content, and timing. Its method combines self-reconstructed training pairs, in-context adaptation, block-causal streaming, three-step self-forcing distillation, shared adapters, and bounded positional caching.
- 2.1. Video Character Replacement: Video replacement methods preserve motion and expression but frame-based approaches can be temporally inconsistent, motivating diffusion-based alternatives such as MoCha, Wan-Animate, and VACE.
- 2.2. Voice Conversion: Voice conversion preserves linguistic content and prosody while changing timbre, with recent zero-shot systems using large-scale pretraining and self-supervised representations.
- 2.3. Audio-Visual Generation: Joint audio-visual generators use cross-modal attention or interacting modality streams for synchronized synthesis, and UniSwap applies this capability to coordinated visual and vocal identity transfer.
- 3. Method: UniSwap formulates replacement from source video, audio, reference image, and reference voice, targeting transferred appearance and timbre while preserving motion, lip movements, background, and linguistic content.
- 3.1. Swap-and-Reconstruct Paired Data Synthesis: Real talking videos become reconstruction targets, while pose-based visual replacement and voice conversion create identity-altered sources with separately sampled visual and vocal references.A random 30% segment of the target audio serves as the reference audio, and all three training stages use these synthesized tuples.
- 3.2. Stage 1: In-context Pretraining: In-context pretraining concatenates source, reference, and target audio-video latents, denoising only target portions while positional offsets preserve consistent semantics across variable-length inputs.Source and target share temporal positions because they represent the same time span; reference image and audio use fixed offsets.
- 3.3. Stage 2: Conditional Streaming Adaptation: Stage 2 converts the model into blockwise autoregressive streaming with K=3 video latent frames per block, aligned audio intervals, causal histories, and KV-cached inference.An initial 4-frame block is required by the causal video VAE, and per-block cost is independent of generated duration.
4. Experiments
Experiments show that UniSwap jointly transfers appearance and voice while preserving source motion and composition, with synchronized lip motion and stable long-duration identity. Quantitative results indicate leading synchronization, competitive identity preservation, efficient streaming generation, and stage-specific quality–efficiency trade-offs.
- Qualitative comparison: UniSwap jointly replaces appearance and voice while preserving the source composition, motion, pose, background, and lip synchronization.Video-only baselines require separate voice conversion, whereas voice-conversion baselines cannot change appearance.
- Long-video comparison: Across one-minute generations, UniSwap remains visually stable and achieves the highest identity similarity in all three 20-second segments.Its IQA ranges from 3.966 to 4.032 and DINO-S from 0.590 to 0.596, while baseline identity and quality metrics degrade over time.
- Short-video comparison: 3.633 Sync-C and 10.304 Sync-D make UniSwap the best evaluated replacement pipeline for audio-visual synchronization.Its DINO-S is 0.629 versus 0.630 for the strongest baseline, while aesthetic and image-quality scores remain below MoCha and SCAIL-2.
- Efficiency comparison: 13.6 wall-clock FPS results from generating each 24-frame block in 1.76 seconds, approximately 10× faster than Wan-Animate at 1.367 FPS.UniSwap processes only the current block against cached context; SCAIL-2 uses its accelerated 8-step LoRA configuration.
- Ablation studies: Stage 3 reduces denoising from 30 to 3 steps per block and improves DINO-S, SIG, BAK, OVRL, and SECS over Stage 2, while reducing synchronization and several quality scores.Stage 1 has the strongest synchronization, video-quality, and speaker-similarity scores; Stage 2 retains comparable visual quality while enabling block-causal generation.
5. Conclusion
UniSwap presents streaming joint audio-visual identity replacement for talking videos as a unified conditional generation task. Its training and adaptation pipeline constructs aligned supervision and reduces streaming diffusion sampling from 30 to 3 denoising steps per block.
- UniSwap is presented as the first framework for streaming joint audio-visual identity replacement in talking videos.
- UniSwap unifies appearance and voice transfer as a conditional generation task and creates aligned supervision through swap-and-reconstruct data synthesis.
- 30 to 3 denoising steps per block: Efficient Self-forcing DMD reduces sampling for streaming generation.
- In-context Pretraining learns joint visual and vocal identity replacement, while Conditional Streaming Adaptation introduces the Decoupled Streaming Conditioning Mask for block-causal generation.
Supplementary Material · 6. Qualitative Ablation
Qualitative ablations over one-minute generations show that Feature-RoPE Decomposition improves long-horizon consistency. Removing its components causes increasing identity drift and visual artifacts, while the full model remains more consistent.
- 6. Qualitative Ablation: Over one-minute generations, ablated Feature-RoPE Decomposition variants develop increasing identity drift and visual artifacts in later segments.Figure 6 visualizes the qualitative effect of each component.
- 6. Qualitative Ablation: The full model remains more consistent across later segments than the ablated variants.This complements the quantitative ablation reported in the main paper.
- 6. Qualitative Ablation: The results support bounded coordinates, reference re-anchoring, and the adaptive sink block as mechanisms for reducing long-horizon drift.Figure 6 is interpreted together with Table 5.
7. KV-Cached Streaming Inference
UniSwap performs blockwise streaming inference with persistent reference and clean-history caches, while source context remains temporary. Each target block is denoised using cached context, committed as clean history, and managed through bounded rolling-cache updates.
- Cache organization: Blockwise inference persists the reference cache throughout generation, temporarily stores source keys and values, and commits completed target blocks to the clean-history cache.The procedure is used by Stages 2 and 3.
- Cache organization: Reference prefill forwards the reference region once and writes its key/value tensors into a never-evicted cache.The reference region is denoted R.
- Blockwise procedure: For each target block, the method prefills the current source slot, denoises while attending to reference, history, and source context, then commits clean context.The loop processes blocks from i = 0 to N − 1.
- Cache maintenance: When the rolling cache is full, UniSwap evicts its oldest block, shifts retained blocks, reapplies RoPE at bounded positions, and discards temporary source tensors after commitment.The procedure returns the generated blocks {B_i} from i = 0 to N − 1.
8. User Study
A blinded study with 30 participants compared UniSwap against four video-replacement baselines paired with Seed-VC. UniSwap achieved the highest ratings for appearance identity, lip synchronization, and naturalness.
- Study design: 30 participants evaluated anonymized outputs from UniSwap and four video-replacement baselines paired with Seed-VC.Each participant assessed four source clips, with method order randomized independently.
- Evaluation criteria: Participants rated appearance identity, voice identity, lip synchronization, and naturalness on a five-point Likert scale.The reported ratings were mean scores on the five-point scale.
- Results: UniSwap received the highest ratings for appearance identity, lip synchronization, and naturalness.Voice identity was also among the evaluated criteria, but no comparative ranking is provided in the passage.
9. Additional Qualitative Results
Additional qualitative results show that UniSwap transfers reference appearance and voice while preserving source motion, composition, and recording context across short and long videos. One-minute generations maintain visual identity over continuous autoregressive streaming, and Feature-RoPE Decomposition improves consistency by preventing later identity drift and artifacts.
- Evaluation coverage: The qualitative evaluation covers diverse identities, poses, gestures, clothing, backgrounds, and recording conditions in joint audio-video outputs.Examples include reference image and voice inputs, source video and audio, and generated audio waveforms alongside visual outputs.
- Short-video results: Figures 7–8 show 16 short-video examples where UniSwap transfers reference appearance while retaining source composition, body motion, and facial activity.The examples span cross-gender replacement, varied framing, substantial hand and upper-body motion, and simple or cluttered backgrounds.
- Long-video results: Figure 9 presents three one-minute generations sampled every 10 seconds whose identities remain visually consistent while preserving background, framing, and time-varying motion.The results provide qualitative evidence that bounded cache coordinates and persistent identity context support stable long-form autoregressive generation.
- Feature-RoPE ablation: Removing any Feature-RoPE Decomposition component causes visible identity drift and artifacts in later segments, whereas the full model remains more consistent.The ablation samples frames every 10 seconds from one-minute generations.
10. Limitations
UniSwap is currently limited to single-speaker talking videos and lacks explicit, independently controllable facial-expression editing.
- UniSwap targets single-speaker talking videos, while multi-speaker scenes, occlusions, and complex interactions remain challenging.
- Audio-driven facial expressions cannot be independently edited or set to arbitrary user-specified expressions.
11. Broader Impact
UniSwap’s audio-video character replacement may support filmmaking, localization, and accessibility, while also creating risks of impersonation, non-consensual media, and misinformation. Responsible deployment should require consent, provenance, disclosure, access controls, and compatibility with forensic detection tools.
- Audio-video character replacement can support filmmaking, localization, and accessibility.
- The technology increases risks of impersonation, non-consensual media, and misinformation.
- Deployment should require consent, provenance mechanisms, visible disclosure where appropriate, access controls, and compatibility with forensic detection tools.