PaperlayerView exact revision on arXiv

Source-linked AI summary

From Short Clips to Real-Time Streams: Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping

Zekai Luo, Yuwei Liang, Ruiqi Ma, Haoyu Huang, Xuehao Wang, Huanyi Zheng, Mingyu Liu, Linhao Zhong, Chenchen Jing, Yuling Xi, Hao Chen

arXiv:2610.26681v1cs.CV

TL;DR

Reference-guided video face swapping still struggles to transfer identity faithfully while preserving target details and generating efficiently over long streams. StreamingSwap combines identity-aware supervision with face-region generation and short-context rollout, and reports competitive identity transfer and target fidelity at over 20 FPS, with stable generation up to 10 minutes.

  • Problem

    Reference-guided methods struggle to combine faithful source-identity transfer, preservation of target attributes, and efficient generation over long video streams.

    Source: §1 · p. 2

  • Method

    StreamingSwap combines noise-level-balanced Joint Rec-ID Loss with Face-RoI Generation and Short-Context Rollout Training, and introduces HumanFaceVid for training and evaluation.

    Source: §1 · pp. 2–3

  • Results

    Over 20 FPS on a single NVIDIA RTX 4090 GPU accompanies competitive identity transfer and target fidelity, with stable generation on videos up to 10 minutes long.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The results show that task-specific supervision and compact context can make reference-guided generation substantially more efficient.

    Source: §6 · p. 10

    Realistic identity manipulation can be misused for impersonation, misinformation, and non-consensual content creation; demonstrations are synthetic and not authentic recordings or endorsements.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Video face swapping transfers the reference identity to a target video while preserving its other attributes and motion. Despite recent progress in reference-guided flow-based methods, jointly achieving faithful identity transfer, high fidelity, and real-time generation over long video streams remains challenging. To address this challenge, we rethink the task-specific structure of reference-guided video face swapping. First, face swapping changes only identity while preserving most visual content, yet flow-matching supervision is weakly sensitive to these changes. We therefore develop Joint Rec-ID Loss, which balances explicit identity supervision and latent reconstruction according to the noise level. Second, the target video already provides non-edited attributes and temporal dynamics, enabling generation with fewer spatial tokens and shorter temporal contexts through Face-RoI Generation and Short-Context Rollout Training. By integrating these designs, we propose StreamingSwap, a reference-guided flow-matching framework for real-time streaming video face swapping. We also introduce HumanFaceVid, comprising 1,000 hours of cinematic training data and evaluation sets of 200 short and 200 long videos. Experiments demonstrate that our method achieves competitive identity transfer and visual quality at 20 FPS on a single NVIDIA RTX 4090 GPU, with stable generation over videos up to 10 minutes long. Project webpage: https://aim-uofa.github.io/StreamingSwap/

1 INTRODUCTION

Reference-guided face swapping can preserve target details, but transferring the source identity and generating efficiently over long streams remain challenging. StreamingSwap addresses these challenges with identity-aware supervision and compact spatial and temporal context, alongside a new training and evaluation dataset.

1 INTRODUCTION

  • Reference-guided methods preserve fine-grained target attributes but face a competing identity signal and costly full-frame denoising over long temporal contexts.Source: §1 · p. 2
  • Face swapping changes identity while preserving most visual content, weakening identity-sensitive flow-matching supervision; target-provided attributes and dynamics also make background processing redundant.Source: §1 · pp. 2–3
  • 20 FPS on a single NVIDIA 4090 GPU, with stable generation on videos up to 10 minutes, accompanies improved identity similarity, preserved target attributes, and high video quality.Source: §1 · pp. 2–3
  • Joint Rec-ID Loss balances clean-latent reconstruction and explicit identity supervision according to the noise level.Source: §1 · p. 2
  • Face-RoI Generation limits conditioning and generation to face-region tokens, while Short-Context Rollout Training uses short generated histories for streaming.The rollout trains a bidirectional model autoregressively to reduce the mismatch between training and inference while retaining bidirectional generation within each window.Source: §1 · p. 2
  • HumanFaceVid comprises 1,000 hours of cinematic training data and evaluation sets of 200 short and 200 long videos.Source: §1 · p. 3

2 PRELIMINARIES

Flow matching learns a velocity field that transports Gaussian noise toward a target distribution. Reference-guided face swapping conditions generation on both source identity and target-video content, with role-reversed paired supervision supporting training.

2.1 FLOW MATCHING

  • Flow matching learns a continuous velocity field that transports Gaussian noise to a target distribution in data or latent space.The clean video latent and Gaussian noise define a linear probability path; the training objective matches predicted and target velocities.Source: §2.1 · p. 3

2.2 REFERENCE-GUIDED VIDEO FACE SWAPPING

  • Reference-guided generation combines noisy video latents with encoded target-video and face-mask conditions, then appends source-image identity tokens for bidirectional Transformer processing.The target video provides attributes and content to preserve while the source image provides identity.Source: §2.2 · p. 3
  • Role-reversed paired supervision uses the real source image and swapped video as conditions, with the original video as ground truth to avoid treating synthesis artifacts as targets.A swapping pipeline produces a video carrying the alternate identity while approximately preserving the original video's non-identity attributes and temporal dynamics.Source: §2.2 · p. 4

3 TASK-SPECIFIC ANALYSIS

The analysis examines why identity transfer receives a weak signal from latent reconstruction and whether face-region processing and shorter temporal context can reduce computation while preserving fidelity. It finds that spatial reduction largely preserves measured quality, while a short temporal context incurs moderate degradation.

3 TASK-SPECIFIC ANALYSIS

  • The task changes facial identity while preserving other facial attributes and content outside the face, but whole-image latent reconstruction may be insufficiently sensitive to identity changes.The analysis frames this as a mismatch between restricted editing scope and reconstruction loss over the entire image latent.Source: §3 · p. 4
  • The target video supplies non-identity attributes, surrounding content, and temporal dynamics, although existing reference-guided methods generate full frames and use extended temporal contexts.The analysis asks how much spatial and temporal context is needed to preserve fidelity while avoiding computation on regions that require no editing.Source: §3 · p. 4

3.1 IDENTITY SENSITIVITY OF FLOW-MATCHING SUPERVISION

  • Identity similarity changes by more than 0.75 across constructed pairs while mean face-region latent MSE varies by less than 0.05, indicating weak flow-matching sensitivity to identity changes.Mild blur and self-swap controls show that visually minor deviations can produce latent MSE around 0.15, comparable to or larger than visible identity changes.Source: §3.1 · p. 5

3.2 SPATIAL AND TEMPORAL CONTEXT REQUIREMENTS

  • Reducing retained video tokens from 100% to 39% largely preserves identity similarity and fidelity-and-quality score, while throughput rises from approximately 1.3 to 2.2 FPS and computation falls from 8,900 to 2,200 TFLOPs.The analysis attributes the savings to background tokens contributing substantial computation but limited information once the target face region is available.Source: §3.2 · p. 5
  • Restricting attention to six neighboring latent frames causes moderate quality degradation while increasing throughput and reducing computational cost; performance drops sharply at K = 2.The target video supplies content and motion cues for long-range consistency, so temporal context primarily maintains local continuity.Source: §3.2 · p. 5
  • The overview separates adaptation with identity-aware loss and face-RoI generation from self-rollout training on short generated histories for streaming inference.Phase II uses the model's own generated short history for autoregressive long-video inference.Source: Figure 3 · p. 6
  • The experiments motivate explicit source-identity supervision alongside computation focused on face-region tokens and short temporal context.The design principles exploit target-provided content and motion to complement latent reconstruction's weak identity sensitivity.Source: §3.1 · p. 5, §3.2 · p. 6

4 METHOD

STREAMINGSWAP adapts a reference-guided flow model in two phases: identity-aware Face-RoI training followed by short-context rollout training for streaming inference.

4.1 OVERVIEW

  • The pipeline initializes from pretrained DreamID-V weights and applies Phase-I identity-aware Face-RoI training before Phase-II rollout for streaming inference.Phase I adapts the reference-guided video flow model.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 PHASE I: IDENTITY-AWARE FACE-ROI TRAINING

  • Face-RoI Generation selects patches intersecting a dilated, downsampled face-mask region, then scatters predicted RoI latents back and feather-blends them with target latents.The blend transitions from full weight inside the face region to zero outside the compute region.Source: §4.2 · p. 6
  • Joint Rec-ID Loss combines clean-latent reconstruction with frozen face-recognition identity supervision, emphasizing reconstruction at high noise and identity-specific detail at low noise.The noise-dependent reconstruction weight shifts the relative contribution of the two losses.Source: §4.2 · p. 7

4.3 PHASE II: SHORT-CONTEXT ROLLOUT TRAINING

  • Short-Context Rollout Training retains bidirectional attention within windows, conditions new frames on short generated history, and self-rolls during training to match streaming inference.The latest composited frames supply history for the next window.Source: §4.3 · p. 7

5 EXPERIMENTS

Across short- and long-video comparisons, STREAMINGSWAP balances identity transfer with target fidelity and realism while exceeding 20 FPS; ablations support the roles of its three components.

5.1 COMPARISONS WITH EXISTING METHODS

  • STREAMINGSWAP balances identity transfer, target fidelity, and visual realism across both benchmarks, exceeds 20 FPS, and achieves substantially higher throughput than evaluated diffusion baselines.The comparisons include two GAN-based and four diffusion-based methods, with three strong-performing methods in the long-video comparison.Source: §5.1 · pp. 8–9

5.2 ABLATION OF IDENTITY-AWARE SUPERVISION AND EFFICIENT CONTEXT

  • Removing identity supervision weakens source-identity transfer despite improving target-attribute scores, supporting that flow-matching supervision alone is insufficient for the intended identity edit.The qualitative ablation also shows stronger source-identity transfer with identity supervision.Source: §5.2 · p. 9, Figure 5 · p. 9
  • Face-RoI substantially increases throughput while maintaining comparable identity and target-attribute scores, consistent with background tokens adding computation but little useful information.The qualitative ablation indicates that Face-RoI preserves visual quality.Source: §5.2 · p. 9, Figure 5 · p. 9
  • Self-rollout training mitigates error accumulation across streaming windows by exposing the model to its own generated history.The qualitative ablation also reports mitigation of progressive visual degradation.Source: §5.2 · p. 9, Figure 5 · p. 9

6 CONCLUSION

STREAMINGSWAP aligns supervision and computation with video face-swapping needs, while HumanFaceVid supplies data for training and short- and long-video evaluation. The paper also describes applications in movies, live videos, and AI-generated dramas.

6 CONCLUSION

  • STREAMINGSWAP combines Joint Rec-ID Loss, Face-RoI Generation, and Short-Context Rollout Training; experiments report competitive identity transfer and target fidelity above 20 FPS, with stable generation up to 10 minutes.HumanFaceVid supports training and evaluation on short and long videos.Source: §6 · pp. 9–10

A RELATED WORK

  • The paper frames video face swapping as transferring source identity while preserving target attributes such as expression, pose, illumination, motion, and background.Source: A RELATED WORK · p. 13

A.1 GAN-BASED VIDEO FACE SWAPPING

  • GAN-based face swapping emphasizes efficient feed-forward synthesis, while many image-based variants process frames independently without explicitly modeling temporal dependencies.Subject-agnostic methods inject source-identity features into target representations; CanonSwap modifies identity in canonical space before restoring target motion.Source: §A.1 · p. 13, A RELATED WORK · p. 13

A.2 DIFFUSION-BASED VIDEO FACE SWAPPING

  • Diffusion face-swapping methods use generative priors for quality and temporal modeling, but reconstructing removed faces from auxiliary conditions can lose fine-grained appearance cues.Reference-guided methods retain the full target video, but can respond weakly to source identity and retain the cost of full-frame, chunk-based generation.Source: §A.2 · p. 13

B DATA PREPARATION

  • HumanFaceVid is constructed from real same-identity image–video pairs, with identity-swapped and contour-warped videos used as inputs and original real videos as reconstruction targets.The pipeline detects and tracks faces, filters short or unreliable tracks, and creates aligned face masks.Source: B DATA PREPARATION · p. 14

C APPLICATIONS IN DIFFERENT SCENARIOS

  • Movie examples retain target illumination, expressions, poses, and scene character while integrating source identity, including in partially occluded shots.Live-video examples show continuous processing, while AI-generated drama examples apply consistent identity without regenerating the whole scene.Source: C APPLICATIONS IN DIFFERENT SCENARIOS · p. 14

D EXPERIMENTAL SETTINGS

The experimental settings define metrics for identity, attribute preservation, video quality, and efficiency, and describe model initialization, training, and acceleration choices.

D EXPERIMENTAL SETTINGS

  • Efficiency is measured by FPS for newly generated frames and end-to-end latency; both include VAE encoding, denoising, decoding, and output compositing.FPS excludes reused history frames.Source: D EXPERIMENTAL SETTINGS · pp. 16–17
  • The model is initialized from DreamID-V and trained in two 1,000-iteration phases; inference-speed measurements use one RTX 4090 GPU.Training uses four H100 GPUs; inference acceleration includes SageAttention, individually compiled Transformer blocks, and LightTAE, with compatible methods receiving the same acceleration strategies.Source: D EXPERIMENTAL SETTINGS · pp. 17–18
  • Table 3 covers training-phase and denoising-step ablations, while Figure 10 plots identity and attribute metrics along the short200 denoising trajectory with inverted error axes.Higher positions in Figure 10 indicate better performance.Source: Table 3 · p. 18, Figure 10 · p. 18

E ABLATION OF TRAINING PHASE AND DENOISING STEPS

The ablations compare training phases and denoising-step budgets, then examine how identity and attribute metrics change along denoising trajectories.

E ABLATION OF TRAINING PHASE AND DENOISING STEPS

  • The evaluation compares DreamID-V initialization at 16 steps, the Phase-I bidirectional model at 25 and 4 steps, and the final Phase-II streaming model at 4 steps.The comparison assesses identity similarity, target-attribute preservation, and end-to-end throughput.Source: E ABLATION OF TRAINING PHASE AND… · p. 18
  • 0.583 ID Sim is achieved by the four-step Phase-I model versus 0.445 for initialization; throughput rises from 6.702 to 30.280 FPS versus 25-step sampling.The final streaming model retains 0.577 ID Sim at 21.174 FPS, with improved pose, lighting, and gaze scores but slightly higher expression error.Source: E ABLATION OF TRAINING PHASE AND… · p. 18
  • Identity similarity for the bidirectional model improves through most of denoising before a slight endpoint decline, while expression and lighting errors rise after early minima.Four-step sampling may sacrifice some fine visual detail relative to 25 steps, so better identity and attribute scores do not imply uniformly better perceptual quality.Source: E ABLATION OF TRAINING PHASE AND… · pp. 18–19

F ABLATION OF JOINT REC-ID SUPERVISION

The ablations study how weighting schedules, switching thresholds, and reconstruction objectives balance identity transfer against target-attribute preservation.

F ABLATION OF JOINT REC-ID SUPERVISION

  • Endpoint reconstruction yields higher identity similarity, lower expression and lighting errors, and slightly higher gaze similarity than velocity reconstruction, at the cost of higher pose error.Gradient norms indicate a more consistent scale relative to identity supervision across noise levels.Source: Table 4 · p. 20, F ABLATION OF JOINT REC-ID SUPERVISION · p. 20, Figure 13 · p. 20
  • The ablations share an initialization, training budget, and four-step evaluation protocol on HumanFaceVid short200.The study compares seven piecewise-constant schedules and sweeps τ from 0.3 to 0.7 under L2.Source: F ABLATION OF JOINT REC-ID SUPERVISION · p. 20
  • L2 retains identity supervision throughout training and reduces reconstruction weight at low noise, strengthening identity transfer over L1 with small changes in attribute preservation and imaging quality.Removing identity supervision weakens transfer, while weakening high-noise reconstruction compromises target fidelity despite stronger identity similarity.Source: F ABLATION OF JOINT REC-ID SUPERVISION · p. 20
  • Under the normalized aggregation, τ = 0.4 has the highest attribute score and τ = 0.7 the highest identity score; the authors use knee-point τ = 0.5.Increasing τ favors identity similarity over attribute preservation, while pose and gaze vary nonmonotonically.Source: F ABLATION OF JOINT REC-ID SUPERVISION · p. 20

G ABLATION OF STREAMING WINDOW SIZE

The ablation varies history and generation window sizes to examine streaming quality–efficiency trade-offs. The authors select (3, 9) as a practical balance between throughput and latency, with identity and attribute preservation varying across window sizes.

G ABLATION OF STREAMING WINDOW SIZE

  • The ablation varies Nctx and Ngen; each continuation generates Ngen latent frames conditioned on Nctx clean composited history frames with bidirectional attention.Unlike the main results’ fixed (3, 9) setup, this ablation model was trained for 1,000 iterations with randomly sampled window sizes.Source: G ABLATION OF STREAMING WINDOW SIZE · pp. 20–21
  • At fixed history length, increasing Ngen improves throughput but increases latency; at fixed generation length, increasing Nctx adds overhead and reduces throughput.Among evaluated configurations, (3, 9) achieves the highest FPS with lower latency than larger-window alternatives.Source: G ABLATION OF STREAMING WINDOW SIZE · p. 21
  • Target-attribute preservation generally improves with larger windows, while identity similarity peaks at (3, 3) and then declines.The authors adopt (3, 9) as a practical balance between throughput and latency while maintaining competitive identity transfer and target-attribute preservation.Source: G ABLATION OF STREAMING WINDOW SIZE · p. 21

H ADDITIONAL QUALITATIVE COMPARISONS ON BENCHMARKS

Additional qualitative comparisons on short and long video benchmarks supplement the main-paper results. The examples illustrate identity transfer and target-attribute preservation across diverse scenes.

H ADDITIONAL QUALITATIVE COMPARISONS ON BENCHMARKS

Related papers

Loading 2610.26681v1…