PaperlayerView exact revision on arXiv

Source-linked AI summary

Preserving Source Video Realism: High-Fidelity Face Swapping for Cinematic Quality

Zekai Luo, Zongze Du, Zhouhang Zhu, Hao Zhong, Muzhi Zhu, Wen Wang, Yuling Xi, Chenchen Jing, Hao Chen, Chunhua Shen

arXiv:2512.07951v2cs.CV

TL;DR

Long video face swapping must preserve cinematic source detail while maintaining identity and temporal consistency, challenges that prior approaches do not fully address. LIVINGSWAP uses keyframes, source-video references, and temporal stitching, and reports detailed identity integration with reduced frame-by-frame editing effort.

  • Problem

    Cinematic video face swapping must maintain realism and temporal consistency over long sequences while preserving source expressions, lighting, and facial detail.

    Source: §1 · pp. 1–2

  • Method

    LIVINGSWAP combines keyframe identity injection, source-video reference guidance, and temporal stitching, with Face2Face providing paired training data and reversed pairs for ground-truth supervision.

    Source: §1 · p. 2, §4.1 · p. 4, §4.3 · p. 5

  • Results

    40-fold reduction in frame-by-frame human editing is reported, alongside integration of target identity with source-video expressions, lighting, and facial details.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors position LIVINGSWAP as particularly suited to professional film and television workflows and introduce CineFaceBench for industrial-scenario comparisons.

    Source: §1 · p. 2

    Outputs can be biased by poor-quality selected keyframes, including those with identity or expression drift or image distortion.

    Source: K. Limitations · p. 19

Abstract

from arXiv · show

Video face swapping is crucial in film and entertainment production, where achieving high fidelity and temporal consistency over long and complex video sequences remains a significant challenge. Inspired by recent advances in reference-guided image editing, we explore whether rich visual attributes from source videos can be similarly leveraged to enhance both fidelity and temporal coherence in video face swapping. Building on this insight, this work presents LivingSwap, the first video reference guided face swapping model. Our approach employs keyframes as conditioning signals to inject the target identity, enabling flexible and controllable editing. By combining keyframe conditioning with video reference guidance, the model performs temporal stitching to ensure stable identity preservation and high-fidelity reconstruction across long video sequences. To address the scarcity of data for reference-guided training, we construct a paired face-swapping dataset, Face2Face, and further reverse the data pairs to ensure reliable ground-truth supervision. Extensive experiments demonstrate that our method achieves state-of-the-art results, seamlessly integrating the target identity with the source video's expressions, lighting, and motion, while significantly reducing manual effort in production workflows. Project webpage: https://aim-uofa.github.io/LivingSwap

1. Introduction

Cinematic face swapping must preserve source-video detail and remain realistic and temporally stable over long sequences, but existing approaches have fidelity or consistency limitations. LIVINGSWAP combines keyframe identity injection, source-video reference guidance, and temporal stitching, alongside the Face2Face dataset and CineFaceBench.

Motivation

  • Frame-by-frame GAN approaches can flicker or jitter in long sequences, while inpainting can lose facial detail; reference-guided methods use rich reference attributes.The introduction describes these as obstacles to meeting cinematic-production quality demands.Source: §1 · pp. 1–2, Figure 2 · p. 2
  • Reference-guided video swapping faces two challenges: maintaining a stable identity over long, complex sequences and obtaining paired training data.Reference-guided image editing has shown flexible, high-fidelity reconstruction, motivating its adaptation to video.Source: §1 · p. 2

Contributions

  • LIVINGSWAP decomposes long-video swapping into keyframe identity injection, video reference completion, and temporal stitching to guide identity and mitigate accumulated errors.Face2Face is a paired dataset for reference-guided swapping; reversing each pair makes generated results inputs and original data ground truth.Source: §1 · p. 2
  • LIVINGSWAP integrates target identity with high-definition source-video details, preserving expressions, lighting, and other facial attributes in cinematic clips.The authors also introduce CineFaceBench to evaluate challenging cinematic conditions.Source: §1 · p. 2
  • Its controllable pipeline reduces the need for frame-by-frame human editing by a factor of 40, targeting professional film and television workflows.The work also introduces CineFaceBench for comparisons tailored to industrial scenarios.Source: §1 · p. 2

2. Related Work

Prior video face-swapping approaches include framewise GAN pipelines and diffusion-based inpainting, while broader video-editing work also includes inversion and reference-guided methods. The paper positions direct source-video references as a way to preserve detail while maintaining identity and attributes across long sequences.

Video face swapping

  • GAN-based video swapping injects target identity with encoder–decoder pipelines but can flicker or jitter, especially in long sequences.Diffusion approaches can improve visual quality and temporal consistency, but face inpainting may lose fine detail and conflict with pretrained priors.Source: §2 · p. 3

Diffusion-based video editing

  • Video editing methods include inversion, inpainting, and reference guidance; inversion can struggle to balance editability and fidelity, while inpainting can lose fine-grained detail.Inpainting methods use structural guidance such as optical flow, depth, or keypoints to edit masked regions.Source: §2 · p. 3
  • Reference-guided editing uses reference images for flexible, high-fidelity reconstruction, but long-video use faces scarce paired data and difficulty maintaining identity or attributes over time.The paper applies this paradigm to face swapping using detailed source-video references and Face2Face with a reversing data strategy for high-fidelity supervision.Source: §2 · p. 3

3. Preliminary: Video Generation with DiT and Rectified Flow

This section describes DiT video generation with Rectified Flow as a continuous denoising process that predicts velocity between latent points. Training minimizes mean squared error between predicted and target velocities, a formulation the passage associates with efficient generation.

Rectified Flow

  • A latent x_t is linearly interpolated between a ground-truth sample x_1 and standard Gaussian noise x_0.The timestep t is sampled from a predefined distribution over [0, 1].Source: §3 · p. 3
  • The target velocity is the derivative of x_t with respect to time, and equals x_1 - x_0.Source: §3 · p. 3

DiT training

  • The DiT model estimates velocity from latent x_t, conditioning signal c, and timestep t, using parameters θ.The training objective minimizes mean squared error between predicted and target velocities.Source: §3 · p. 3
  • The training formulation is described as enabling high-quality video generation with fewer steps and greater computational efficiency.Source: §3 · p. 3

4. Method

LIVINGSWAP combines keyframe identity guidance, source-video references, and chunk stitching in a controllable pipeline; Face2Face supplies paired training data for reference-guided swapping.

4.1. Keyframes Identity Injection

  • Representative keyframes capture pose, expression, and illumination changes, then receive high-quality per-frame edits that guide generation between neighboring frames.Only a small set of boundary keyframes needs editing, retaining frame-level editing flexibility while improving efficiency.Source: §4.1 · p. 4

4.2. Video Reference Completion

  • Source-video segments serve as visual references, preserving fine-grained lighting, expression, and background cues during completion.An optional target identity image can improve identity fidelity in challenging cases such as occlusions or low-quality frames.Source: §4.2 · p. 4

4.3. Temporal Stitching

  • Chronological chunk generation uses the previous chunk’s final frame as the next start guidance and a keyframe as end guidance to support coherent transitions.Independent chunks without temporal overlap can produce boundary discontinuities and jumps.Source: §4.3 · p. 5
  • Approximately 40× less manual labor is required because editing typically targets only the first and last frames of each chunk.The reference model produces a fixed 81-frame output per inference.Source: §4.3 · p. 5

4.4. Dataset Construction

  • Face2Face pairs GAN-swapped video inputs with original-video keyframes and ground truth, providing artifact-free, high-quality, temporally consistent supervision.Role reversal and pretrained-model priors support robustness to noisy training samples; rectified flow loss supervises source-attribute preservation.Source: §4.4 · p. 5

5. Experiments

Experiments compare LivingSwap on general face-manipulation and cinematic benchmarks, then examine training-data quality and key model components. LivingSwap achieves the best overall ranking on FF++ and state-of-the-art performance across multiple metrics and identity settings on CineFaceBench.

5.1. Experimental Setup

  • CineFaceBench contains 400 target–source test pairs spanning cinematic challenges, while FF++ mainly contains interview-style or livestream videos.CineFaceBench includes long takes, complex lighting, exaggerated expressions, heavy makeup, and semi-transparent occlusions; each video has easy and hard identity-similarity targets.Source: §5.1 · p. 6
  • Evaluation uses image-level ID Similarity, Expression Error, Lighting Error, Gaze Error, and Face Pose Error, plus video-level FVD; average rankings across all metrics provide a comprehensive performance assessment.Ablation models use 2,000 training steps and evaluation on the 100 most challenging FF++ samples.Source: §5.1 · p. 7

5.2. Comparisons with Existing Methods

  • LivingSwap achieves the best overall ranking on FF++ and state-of-the-art performance across multiple metrics and average ranks on CineFaceBench’s easy and hard identity settings.On CineFaceBench, the comparisons also report stronger temporal consistency and source-attribute preservation than Inswapper, including in side views and occlusions.Source: §5.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

5.3. Ablation Studies

  • Using the full Face2Face dataset improves fidelity through greater diversity, while filtering yields no benefit and the model remains robust to noisy training pairs.The dataset visualization associates low identity similarity with artifacts and distortions, while high similarity can still include failed swaps, inconsistencies, and flickering.Source: §5.3 · p. 8, Table 3 · p. 8, Figure 5 · p. 8
  • Replacing video reference with inpainting reduces fidelity, while removing keyframe guidance lowers identity similarity and temporal consistency; target-image guidance remains useful when identity matters.Removing the target image also reduces identity similarity because occlusion, extreme angles, or closed eyes can hide identity features in keyframes.Source: §5.3 · p. 8, Table 4 · p. 8

6. Conclusion

The conclusion presents LivingSwap as a video-reference-guided face-swapping model that uses keyframes to support fidelity and temporal coherence. It also reports new training and evaluation resources, strong performance, and reduced production effort.

6. Conclusion

  • LivingSwap combines keyframe conditioning with video-reference guidance for stable identity preservation and high-fidelity reconstruction across long videos.The conclusion describes it as the first video-reference-guided face-swapping model.Source: §6 · p. 8
  • Face2Face uses role-reversed paired data for reliable supervision, and CineFaceBench provides a benchmark for cinematic face swapping.Source: §6 · p. 8
  • Experiments report integration of target identity with source-video attributes, alongside reduced manual effort and more efficient, flexible editing for film and entertainment.Source: §6 · p. 8

for Cinematic Quality

The supplementary material provides further experiments and construction details for Face2Face and CineFaceBench.

  • Sections A–G cover generalization, identity injection, keyframe quality and selection, identity differences, source-video attribute variation, and grayscale guidance.Source: for Cinematic Quality · p. 12
  • Sections H and I provide construction details for Face2Face and CineFaceBench, respectively.Source: for Cinematic Quality · p. 12
  • Sections J and K cover comparisons with closed-source methods and LivingSwap’s limitations.Source: for Cinematic Quality · p. 12

A. Generalization Beyond Train Data Quality

The section tests LivingSwap on noisy Face2Face pairs and reports outputs that surpass the original pair quality. It attributes this generalization to role-reversed supervision and pretrained-model priors.

A. Generalization Beyond Train Data Quality

  • The test uses manually selected noisy source–target pairs containing failed swaps, artifacts, and expression misalignment from large identity gaps or occlusions.LivingSwap is run with the same inference process as Inswapper, which constructed the dataset.Source: A. Generalization Beyond Train Data… · p. 12
  • LivingSwap’s outputs surpass the original noisy pairs in expression alignment, visual realism, and local-artifact reduction.Source: A. Generalization Beyond Train Data… · p. 12
  • The authors attribute the result to role reversal, which provides reliable ground-truth supervision despite noisy inputs, and pretrained priors that correct corrupted patterns.Source: A. Generalization Beyond Train Data… · p. 12

B. Keyframe Identity Injection for Accumulated Identity Errors

Using only the first frame for identity guidance lets identity errors accumulate over long videos, while keyframe identity injection with temporal stitching corrects identity across chunks.

B. Keyframe Identity Injection for Accumulated Identity Errors

C. Robustness to Keyframe Quality

LIVINGSWAP can refine flawed keyframes using source-video attributes, and selecting frontal keyframes can further improve results in the tested worst-performing cases.

C. Robustness to Keyframe Quality

  • Even with artifacts or expression misalignment in injected keyframes, LIVINGSWAP produces outputs better aligned with the source video by referencing its attributes and using diffusion-model priors.The paper attributes robustness to correction from source-video details and diffusion priors that enforce temporal realism and semantic consistency.Source: C. Robustness to Keyframe Quality · p. 12
  • LIVINGSWAP can surpass the quality of its Inswapper-based training pairs, avoiding incomplete swaps, mismatched regions, and occlusion-induced artifacts.Source: Figure 6 · p. 13

D. Potential of Keyframe Selection

E. Robustness to Identity Differences

The experiments examine identity and source-attribute variation, introduce grayscale keyframe guidance to reduce color artifacts, and assess performance on cinematic benchmarks and closed-source comparisons.

E. Robustness to Identity Differences

  • LIVINGSWAP produces satisfactory swaps for the same source video across target identities with large or small identity differences.The authors hypothesize that this robustness reflects identity diversity in the training data.Source: E. Robustness to Identity Differences · p. 13

F. Robustness to Attribute Variations in Source Video

G. Grayscale Keyframe Guidance for Robust Color Learning

  • Grayscale keyframe guidance removes explicit chromatic information while preserving structural cues, directing color recovery toward the reference-video branch.The edited keyframe is converted to a grayscale image before VAE encoding; the remaining generation pipeline is unchanged.Source: G. Grayscale Keyframe Guidance for… · p. 16
  • Grayscale guidance significantly reduces color bleeding and flickering in challenging cases while maintaining comparable identity preservation and temporal consistency.The final checkpoint was finetuned for 5,000 steps for the grayscale pipeline.Source: G. Grayscale Keyframe Guidance for… · pp. 14, 16

H. Face2Face Construction Details

  • Face2Face contains 152,221 video samples totaling over 300 hours, with swapped videos used as inputs and original videos as ground-truth targets.The dataset is based on CelebV-Text and VFHQ; training pairs are reversed after Inswapper face swapping.Source: H. Face2Face Construction Details · pp. 17–18

I. CineFaceBench Construction Details

  • CineFaceBench addresses the limited cinematic scope of FaceForensics++ with 200 clips and 400 easy/hard target-face pairs spanning challenging scenes and some videos longer than one minute.The easy and hard targets are selected as the most similar and least similar identities to each source video; the paper reports impressive performance in these film-scene scenarios.Source: §I · pp. 18–19

J. Comparison with Closed-Source Methods

  • Against captured demos of HiFiVFS and FaceAdapter, LIVINGSWAP better preserves source lighting and expression and shows strong stability in occluded cases.The compared methods are inpainting-based approaches and are not open-source.Source: J. Comparison with Closed-Source Methods · p. 19

K. Limitations

LIVINGSWAP’s keyframe dependence and slow inference are the stated limitations.

K. Limitations

  • Poor-quality selected keyframes can bias the final output through identity or expression drift and image distortion.Higher-quality keyframe selection significantly improved the 10 worst-performing cases.Source: K. Limitations · p. 19
  • An 81-frame video takes 195 seconds on one H200 GPU with 108 GB memory, about one minute per second of swapped video.Source: K. Limitations · p. 19
Loading 2512.07951v2…