Source-linked AI summary
ID-V2V: Identity-Preserving Video Restylization
Yuancheng Xu, Mingming He, Pablo Salamanca, Li Ma, Yash Kant, Emmett Steven, Paul Debevec, Ning Yu
TL;DR
Generative video models struggle to preserve human identity and fine-grained performance while flexibly restylizing scenes, especially without paired training data. ID-V2V decouples identity-preserving relighting from edit-driven synthesis and outperforms existing methods while supporting multi-subject restylization.
Problem
Identity-preserving video restylization must retain facial likeness and fine-grained performance during flexible visual edits, but paired training data is scarce and subtle cues are difficult to model.
Method
ID-V2V decouples source-grounded identity preservation, formulated as video relighting, from edit-driven synthesis using complementary facial, keyframe, and depth controls.
Results
ID-V2V significantly outperforms existing baselines in preserving facial identity and fine-grained performance while robustly supporting multi-subject restylization and high visual quality.
Takeaways & Limitations
ID-V2V shows promise as a practical, scalable human-centric approach for real-world content production.
Takeaways & Limitations
Large geometric differences between the source video and edited keyframe can make depth conditioning overly restrictive and reduce edit fidelity.
Abstract
from arXiv · showhide
In visual storytelling, human performances are central to creative intent and narrative meaning. However, preserving human identity and performance while enabling flexible visual edits remains challenging for generative video models. We formalize this challenge as identity-preserving video restylization, which propagates scene, lighting, and style changes specified by an edited keyframe across a source video, while preserving facial likeness and performance, including expressions, eye gaze, and lip synchronization. A key obstacle is the absence of paired training data, as identity-preserving restylized video pairs are rare in real-world settings. To address this, we propose a decoupling of source-grounded identity preservation and edit-driven video synthesis. Our key insight is that facial appearance and expression should remain invariant, with illumination being the primary permissible variation. We therefore cast identity preservation as a video relighting problem, while modeling visual edit propagation as controlled video synthesis guided by the edited keyframe. Building on this formulation, we introduce ID-V2V, a video-to-video generative framework integrating complementary control signals: relit facial regions and facial normal maps tightly constrain facial likeness and performance, while edited keyframes and depth sequences enable flexible and temporally coherent generation. This design enables constructing training pairs from a single video, eliminating the need for scarce paired data. Extensive experiments demonstrate that ID-V2V significantly outperforms existing methods in preserving facial likeness and fine-grained facial performance, supports both single- and multi-subject scenarios, and delivers high visual quality, highlighting its potential as a human-centric tool for real-world content production. The code is available at: https://github.com/Eyeline-Labs/ID-V2V.
1 Introduction
ID-V2V introduces identity-preserving video restylization, propagating edited scene, lighting, and appearance changes across a source video while preserving facial likeness and fine-grained performance. It addresses scarce paired data by decoupling edit-driven synthesis from source-grounded identity preservation and constructing complementary controls from a single training video.
- Motivation: The task is motivated by post-production workflows that capture character performance first and modify backgrounds, lighting, and overall style afterward.This decoupling preserves facial expressions, motion, and interactions while enabling later visual design changes.
- Task: Identity-preserving video restylization propagates edited scene, lighting, and appearance changes across a source video while preserving facial likeness, expressions, eye gaze, and lip synchronization.The target appearance is specified through an edited keyframe, and subjects may be relit to match the edited scene without changing identity or performance.
- Method: ID-V2V addresses scarce paired training data by decoupling flexible edit-driven synthesis from source-grounded identity preservation.The method formulates identity preservation as video relighting, allowing creative edits while maintaining source-grounded human appearance and performance.
- Method: ID-V2V constructs complementary control signals from a single training video, including relit face regions, face normal maps, an edited first frame, and a depth sequence.Relit face regions simulate facial-lighting changes, while face normals provide geometric cues and the edited frame plus depth sequence represent desired visual edits.
- Results: ID-V2V significantly outperforms existing baselines in preserving facial identity and fine-grained performance, supports multi-subject scenarios, and achieves strong overall visual quality.The reported advantages include retaining expressions and lip synchronization more faithfully than methods using abstract controls such as facial landmarks or low-dimensional identity embeddings.
2 Related work
Prior work preserves identity through reference-image conditioning, transfers performance through video controls, stylizes appearance, or manipulates illumination. These approaches remain limited by weak grounding in full source performances, potential identity alteration, and temporal inconsistency in video relighting.
- Identity-Preserving Generation with Reference Images: Reference-image methods improve visual identity consistency but do not use source performance videos, limiting access to subtle expressions and performance cues.They primarily operate in text-to-video or image-to-video settings with one or a few reference images.
- Video-to-Video Generation and Performance Transfer: Because identity cues are not grounded in the full source video, performance-transfer methods struggle with multi-view identity preservation and identity reconstruction under expression or lighting changes.These methods commonly infer facial identity from a single input frame while using facial or motion control signals for generation.
- General Stylization: General image and video stylization applies artistic references or edited exemplars but may alter identity-defining facial or human-subject appearance.Examples include cartoonization, oil-painting styles, and other user-specified appearances.
- Relighting and Illumination Control: Single-image portrait relighting methods modify facial illumination while preserving identity, but applying them to video can produce temporal inconsistencies.These approaches use face priors, real captured data, or physically based rendering models.
- Relighting and Illumination Control: Recent video relighting methods address illumination editing with an emphasis on temporal consistency across frames.Representative methods include Light-X, Lumen, Light-A-Video, RelightVide, and RelightMaster.
3 Method
ID-V2V decouples edit-driven video synthesis from source-grounded identity preservation for keyframe-based video restylization. It combines edited-keyframe and depth-guided generation with relit facial regions and facial normals to preserve facial likeness, expressions, eye gaze, and lip synchronization while enabling scene, lighting, and style edits.
- Task formulation: The task propagates edits from an edited first frame through a source performance video while preserving the subject’s facial likeness and dynamic facial performance.The formulation supports redesigning scenes, environments, lighting, and other visual elements after performance capture while treating the human subject and performance as invariant.
- Method overview: ID-V2V decouples edit-driven synthesis, which determines generated visual content, from identity preservation, which anchors the human subject to the source video.This separation addresses the difficulty of preserving subtle cues such as lip synchronization, micro expressions, eye gaze, and emotional dynamics under visual changes.
- Edit-driven synthesis: Edit-driven synthesis uses the edited keyframe and a DepthAnything 2 depth sequence to generate a temporally coherent video with controlled scene appearance, motion, and layout.The edited keyframe guides desired appearance and stylistic changes, while depth encodes global motion and scene layout.
- Identity preservation: Identity preservation is formulated as video relighting, keeping facial structure and expression fixed while allowing illumination to vary.Relit face and head regions provide pixel-level appearance cues, while facial normal maps predicted by DAViD constrain geometry and reduce drift caused by lighting–shape ambiguity.
- Architecture and inference: The architecture extends VACE on Wan2.1 to multiple ControlNets, fusing relit-face, face-normal, and depth features by summation before the main DiT blocks.At inference, original facial pixels are used directly, while guidance can expand from facial regions to the full body or full frame for broader appearance and relighting control.
4 Experiments
Experiments evaluate ID-V2V on single- and two-subject videos using facial identity, facial performance, and general video-quality metrics, with additional qualitative, user-study, multi-subject, ablation, relighting, and failure-mode analyses. ID-V2V consistently preserves facial likeness and performance while maintaining strong video quality across subject settings, though extreme source lighting can cause artifacts.
- Evaluation Setup: The evaluation uses 100 single-subject and 60 two-subject videos covering diverse actions and facial expressions, while qualitative results additionally examine more than two subjects.Quantitative comparisons restrict evaluation to at most two subjects because existing baselines cannot reliably handle more.
- Qualitative Results: ID-V2V retains teeth shape, subtle facial muscle movements, eye blinks, subject interactions, and facial expressions during restylization, including videos with more than two subjects.The method also preserves each subject’s appearance and interactions while changing scene elements.
- Quantitative Results: ID-V2V outperforms all baselines across AdaFace, Exp-AU, and Exp-EMOCA in both single-subject and two-subject settings.These metrics assess facial identity and fine-grained facial performance preservation.
- Quantitative Results: ID-V2V achieves superior or mostly comparable general video-quality performance without compromising overall video generation capability.General quality covers prompt alignment, temporal consistency, and visual fidelity; face-specific baselines degrade noticeably when moving to two subjects, whereas ID-V2V remains strong.
- User Study: User-study win rates exceed 74% for facial likeness and facial performance and 70% for overall video quality in the single-subject setting.ID-V2V is consistently preferred over all baselines across facial appearance, facial performance, and overall visual quality, with a stronger advantage in the two-subject setting.
- Ablation and Limitations: Joint training with relit face videos and face normals provides strong identity and fine-grained facial-performance preservation, while extreme colored lighting or hard shadows can cause residual casts and temporal inconsistency.The ablation indicates that facial pixels are crucial and face normals provide complementary geometric guidance; regular source lighting is recommended.
5 Conclusion and Limitations
The paper frames identity-preserving video restylization as preserving human identity and performance during flexible scene edits, and presents ID-V2V as a practical solution despite depth-conditioning limitations. Experiments show strong preservation of facial identity, performance cues, and inter-subject interactions, including in multi-subject scenarios.
- Conclusion: ID-V2V addresses identity-preserving video restylization by propagating edited-keyframe visual changes across source videos while preserving faithful human identity and performance.The task targets creative intent and narrative meaning in generative video content production.
- Conclusion: The framework decouples source-grounded identity preservation from edit-driven synthesis to overcome the lack of paired training data.This design supports video-to-video editing while retaining identity and performance information.
- Conclusion: ID-V2V substantially outperforms existing methods in preserving facial identity, subtle performance cues, and inter-subject interactions while supporting multi-subject scenarios and high overall video quality.The results position it as a practical and scalable human-centric approach for real-world content production.
- Limitations: Large geometric discrepancies between the source video and edited keyframe can make source-depth conditioning overly restrictive and reduce edit fidelity.Moderate background changes are generally tolerated, but extreme scene-layout or depth differences may conflict with source geometry.