PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context Sync-LoRA for Portrait Video Editing

Sagi Polaczek, Or Patashnik, Ali Mahdavi-Amiri, Daniel Cohen-Or

arXiv:2512.03013v1cs.CVcs.AIcs.GR

TL;DR

Portrait video editing must modify appearance while keeping each frame aligned with the source’s motion. Sync-LoRA trains an in-context image-to-video model on synchronized pairs, and the authors report a balance of edit fidelity and motion preservation across diverse edits. The approach has limitations with fast or large-scale motion and assumes geometric alignment between the edited first frame and source.

  • Problem

    Portrait editing requires localized appearance changes while preserving the source’s exact frame-by-frame motion, a combination that remains challenging for existing techniques.

    Source: §1 · pp. 1–2

  • Method

    Sync-LoRA fine-tunes an in-context image-to-video model on synchronized original-and-edited video pairs, using the source video for motion cues and the edited first frame to define appearance changes.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Sync-LoRA is reported to generalize across diverse edits while balancing visual fidelity, motion correspondence, and identity preservation.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Frame-accurate synchronization is especially crucial for personalized talking-head applications where preserving the original action, speech, and performance matters.

    Source: §5 · p. 8

    Edit quality may degrade with fast or large-scale motion, and significant geometric misalignment between the edited first frame and source can lead to drift or local artifacts.

    Source: §5 · p. 8

Abstract

from arXiv · show

Editing portrait videos is a challenging task that requires flexible yet precise control over a wide range of modifications, such as appearance changes, expression edits, or the addition of objects. The key difficulty lies in preserving the subject's original temporal behavior, demanding that every edited frame remains precisely synchronized with the corresponding source frame. We present Sync-LoRA, a method for editing portrait videos that achieves high-quality visual modifications while maintaining frame-accurate synchronization and identity consistency. Our approach uses an image-to-video diffusion model, where the edit is defined by modifying the first frame and then propagated to the entire sequence. To enable accurate synchronization, we train an in-context LoRA using paired videos that depict identical motion trajectories but differ in appearance. These pairs are automatically generated and curated through a synchronization-based filtering process that selects only the most temporally aligned examples for training. This training setup teaches the model to combine motion cues from the source video with the visual changes introduced in the edited first frame. Trained on a compact, highly curated set of synchronized human portraits, Sync-LoRA generalizes to unseen identities and diverse edits (e.g., modifying appearance, adding objects, or changing backgrounds), robustly handling variations in pose and expression. Our results demonstrate high visual fidelity and strong temporal coherence, achieving a robust balance between edit fidelity and precise motion preservation.

1. Introduction

Portrait editing must localize appearance changes while preserving each blink, gaze shift, and articulation at the source frame’s exact time. Sync-LoRA addresses this with paired synchronized videos and an in-context editing approach, extending prior image and video methods toward frame-accurate portrait synchronization.

1. Introduction

  • Portrait edits must change only instruction-relevant appearance while preserving every motion event at its source-frame timing, a combination that remains challenging for existing techniques.The stated goal includes temporal identity preservation as well as precise synchronization and a minimal editing footprint.Source: §1 · pp. 1–2
  • Sync-LoRA uses an edited first frame and text prompt to guide a fine-tuned image-to-video model trained on paired videos with identical motion trajectories but different appearances.The paired-video training teaches motion transfer from the source while propagating localized edits.Source: §1 · p. 2
  • A two-stage data pipeline generates portrait-image and talking-head video pairs, then filters them using synchronization scores across speech, gaze, blink, and pose.Generated pairs can contain temporal misalignment or motion drift, so only the most aligned examples are retained.Source: §1 · p. 2

2. Related Work

  • Unlike image-based IC-LoRA approaches that train separately for each editing task, Sync-LoRA is presented as one module that generalizes across diverse edits while balancing visual fidelity and motion preservation.The paper frames its extension of IC-LoRA [26] as video editing focused on frame-accurate portrait synchronization.Source: §1 · p. 2, §2 · p. 2
  • Prior video-diffusion work explored in-context clip generation without ensuring temporal correspondence, while PoseGen targets pose-controllable generation rather than synchronized editing.Sync-LoRA explicitly trains in-context behavior for synchronized editing.Source: §2 · p. 2
  • Related video-editing methods include text-guided edits, first-frame edits propagated through a video, and task-specific object insertion or doodle overlays.Other LoRA video methods focus on personalizing appearance or motion rather than frame-accurate synchronized editing.Source: §2 · p. 3
  • Earlier portrait methods transfer expressions through keypoints or 3DMM, whereas Sync-LoRA uses the edited first frame for appearance context and the source video for motion context.This formulation avoids explicit correspondence modeling.Source: §2 · p. 3

3. Method

Sync-LoRA conditions edited-video generation on a clean source video, an edited first frame, and a prompt. It trains on generated portrait pairs filtered for temporal alignment, using an image-to-video diffusion model and a LoRA adaptation.

3.1. In-Context LoRA

  • During training, the edited view is denoised while the clean source view supplies motion guidance through shared attention; at inference, the edited first frame and source video guide synchronized video generation.The prompt describes the intended relation between the views, and the model propagates the edit across the sequence.Source: §3.1 · p. 3

3.2. Data Generation and Curation

  • The data pipeline generates portrait pairs and side-by-side talking-head videos, then filters pairs using landmark-derived motion signals for speech, gaze, blink, and pose.Pairs are compared at zero temporal lag, and examples with insufficient detection coverage are discarded before top-ranked examples are retained.Source: §3.2 · p. 4, Figure 3 · p. 4
  • More than 20,000 paired examples are generated, but synchronization filtering leaves 512 training pairs in a 3:1 edited-to-identical sample ratio.The ratio is used to stabilize training.Source: §3.2 · p. 5

3.3. Training

  • The model is built on LTX-Video and fine-tuned with rank-128 LoRA; its rectified-flow objective predicts the velocity transforming noise into the edited-video latent.The source-video conditioning stream remains noise-free throughout denoising and provides motion and appearance cues.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate Sync-LoRA on portrait-video editing, comparing its visual fidelity, synchronization, and identity preservation with baselines. Qualitative comparisons, quantitative results, and ablations examine its performance and data-curation choices, alongside expression editing.

4. Experiments

  • The benchmark contains 166 portrait videos and spans object insertion, background replacement, colorization, and appearance modification.The clips come from CelebV, CelebV-HQ, TalkVid, and high-quality YouTube content; the evaluation considers spatial fidelity and temporal alignment.Source: §4 · p. 6

4.1. Qualitative Evaluation

  • In qualitative comparisons, Sync-LoRA preserves source motion while applying edits consistently; baselines show edit failures, temporal artifacts, inconsistent textures, or motion drift.The authors report this pattern across the examples shown in Figure 5.Source: §4.1 · p. 6

4.2. Quantitative Comparisons

  • Sync-LoRA achieves a strong balance of synchronization and edit fidelity: LucyEdit has 0.80 Speech Corr. versus 0.72 for Sync-LoRA, but only 0.11 versus 0.21 Directional CLIP (text-dual).The authors also report that a user study preferred Sync-LoRA for visual quality and temporal alignment.Source: Table 1 · p. 6, §4.2 · p. 7

4.3. Ablation Studies

  • ID-Only preserves alignment but fails the edit, while Edit-Only and Random introduce facial drift; the full model balances synchronization with edit preservation.The comparison includes the painted edit on Batman’s mouth and aligned eye motion; Table 2 covers these dataset-composition baselines.Source: §4.3 · pp. 7–8, Table 2 · p. 8
  • Removing any synchronization cue degrades alignment; removing speech causes Speech Corr. to fall from 0.72 to 0.53, while removing pose lowers Gaze Corr. from 0.75 to 0.67.The authors describe speech, gaze, blink, and pose as complementary motion cues; the table compares cue ablations and dataset-composition baselines.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4. Application - Expression Modification

  • Sync-LoRA propagates happy, angry, or sad expressions while maintaining articulation, structural consistency, and synchronization, including under partial occlusion.The authors report more stable, photorealistic results than LivePortrait in challenging cases such as extreme head rotations or facial occlusions.Source: §4.4 · p. 8, Figure 8 · p. 8

5. Conclusions, Limitations and Future work

Sync-LoRA propagates edits from an edited first frame while aiming to preserve temporal alignment and identity across diverse subjects and edit types. The paper identifies motion and geometric-alignment limitations and points to multimodal video models as a future direction.

5. Conclusions, Limitations and Future work

  • Trained on automatically generated, synchronization-filtered pairs, Sync-LoRA balances edit fidelity, motion correspondence, and visual realism across subjects and edit types.The listed edits include background manipulation, object insertion, and expression modification.Source: §5 · p. 8
  • Fast or large-scale motion can challenge spatial coherence, while geometric misalignment of the edited first frame can lead to temporal drift or local artifacts.The approach builds on a specific base video diffusion model; the authors suggest stronger temporal reasoning and multi-view consistency as possible improvements.Source: §5 · p. 8
  • Frame-accurate synchronization is especially crucial for personalized talking-head applications, and extending the formulation to video models that jointly reason over video and audio is future work.The conclusion connects synchronization to faithfulness to the original action, speech, and performance.Source: §5 · p. 8

Supplementary Material

The supplementary material includes implementation details for differentiating reference and target streams and a summary of Sync-LoRA training hyperparameters. The stream design keeps the reference as context while training on the target.

Supplementary Material

  • The supplementary material includes Table 3, which summarizes the configuration used for fine-tuning Sync-LoRA.The supplied passage identifies the table's scope but does not provide its hyperparameter values.Source: Table 3 · p. 12

6.1. Reference and Target Stream Differentiation

  • Reference and target streams are concatenated along the sequence dimension, with per-token timestep conditioning; masking blocks gradients through the frozen reference stream.Gradients flow only through the target stream, while spatial correspondence is preserved.Source: §6.1 · p. 12

6.2. Hyperparameters

  • The hyperparameter subsection directs readers to Table 3 for the detailed training configuration.The provided passage does not specify individual settings.Source: §6.2 · p. 12

7. Data Curation Pipeline

The data-curation pipeline generates paired portrait videos and filters them using synchronization scores. Those scores combine motion signals for speech, gaze, blinking, and pose, then compare processed source and edited signals at zero temporal lag.

7. Data Curation Pipeline

  • The curation process accepts or rejects generated video pairs based on synchronization scores.Figure 13 illustrates representative accepted and rejected pairs.Source: §7 · p. 12

7.1. Synchronization Scoring Metrics

  • The score weights speech at 40%, gaze at 30%, blink at 15%, and pose at 15%, using mouth, iris, eye, and upper-body motion signals.The pose channel tracks six upper-body geometric features; blink uses a simplified eye aspect ratio averaged across both eyes and negated so events appear as positive peaks.Source: §7.1 · p. 12
  • Signals are interpolated for missing detections, smoothed with a Savitzky-Golay filter, z-score normalized, and compared using weighted Pearson correlations at zero temporal lag.The smoothing filter uses a window length of 9 and polynomial order 2.Source: §7.1 · p. 12

8. Evaluation Benchmark and Metrics

The evaluation benchmark excludes clips with specified visual-quality or editing issues. The section defines metrics for edit-direction alignment, text consistency, and identity preservation.

8.1. Benchmark Curation

  • The benchmark retains clips with consistent lighting and high sharpness, excluding jump cuts, subtitles, significant motion blur, and letterboxing examples.Figure 9 illustrates a typical rejection case.Source: §8.1 · p. 13, Figure 9 · p. 14

8.2. Quantitative Evaluation Metric Definitions

  • Directional CLIP measures alignment between framewise visual transformations and the intended edit, using image-based or text-dual directions with ViT-B-32.The text-dual direction is derived from target and source text embeddings; the score is a mean cosine similarity.Source: §8.2 · pp. 13–14
  • CLIP-Text Alignment measures edited-video consistency with the target description, while ArcFace Similarity measures identity consistency across frames.Higher ArcFace similarity corresponds to stronger identity consistency.Source: §8.2 · p. 14

9. User Study

A blinded, randomized user study compared Sync-LoRA with four baselines on edit fidelity, synchronization, identity preservation, and overall preference. Participants strongly favored Sync-LoRA across all four criteria, with the highest gains in identity preservation and synchronization.

9. User Study

  • The blinded, randomized 2AFC study involved 23 participants, each completing 12 comparisons, for 69 judgments per baseline.The 36 comparison pairs were distributed across three evaluation forms.Source: §9 · p. 14
  • Users strongly favored Sync-LoRA over four state-of-the-art baselines across all four evaluated criteria.The criteria were edit fidelity, synchronization, identity preservation, and overall preference.Source: §9 · p. 14
  • The highest preference gains were in identity preservation and synchronization.The authors connect these results to preserving subject identity and temporal coherence while applying edits.Source: §9 · p. 14

10. Supplementary Videos

The supplementary videos provide interactive side-by-side comparisons with baselines, ablations, expression-editing examples, and failure cases. The authors emphasize that motion stability and temporal coherence cannot be fully assessed from static frames.

10. Supplementary Videos

  • The interactive supplementary file presents side-by-side comparisons against VACE, LucyEdit, FlowEdit, and AnyV2V.It includes comprehensive comparisons against all listed baselines.Source: §10 · p. 14
  • The supplementary material includes ablation studies examining synchronization cues.Source: §10 · p. 14
  • Expression-editing examples and failure cases are also provided, since static frames cannot fully convey motion stability or temporal coherence.Source: §10 · p. 14

11. Expression Modification

The expression-editing comparison presents Sync-LoRA against LivePortrait, focusing on robustness to occlusion and nearby structures. Sync-LoRA’s in-context generation produces plausible pixels in occluded regions, while the described warping approach can distort faces and rigid objects.

11. Expression Modification

  • Figure 10 compares Sync-LoRA with four baselines on user-study criteria, whereas the expression-editing comparison is against LivePortrait.Source: §11 · p. 14, Figure 10 · p. 15
  • Sync-LoRA synthesizes geometrically plausible pixels in occluded regions while transferring the target expression.The passage describes the results as structurally intact and photorealistic.Source: §11 · p. 15
  • LivePortrait’s warping can distort a background person’s face and blur a hand, or bend a microphone near the subject’s jawline.The examples involve partial occlusion or nearby background elements.Source: §11 · p. 15

12. Limitations and Failure Cases

The failure cases include conflicts between the edited first frame’s geometry and the source video, as well as very fast or large-scale motion. These conditions can produce synchronization problems, blur, or loss of temporal coherence.

12. Limitations and Failure Cases

  • A structurally contradictory edited frame, such as a zoom-out, can compromise temporal alignment and cause synchronization issues to propagate through the sequence.The model must reconcile conflicting spatial signals from the edited frame and source video.Source: §12 · p. 15
  • Figure 12 illustrates facial blur and temporal drift for non-aligned geometric edits, plus detail loss and warping on fast-moving hands during complex appearance edits.Source: Figure 12 · p. 15
  • Very fast or large-scale motion can make source-video optical-flow guidance ambiguous, leading to blurred textures or reduced temporal coherence.Examples include rapid hand movements, dancing, and camera pans.Source: §12 · p. 15
Loading 2512.03013v1…