PaperlayerView exact revision on arXiv

Source-linked AI summary

LiveSync: Iterative Self-Training for Lip-Sync Generation

Huaishao Luo, Huan Zhang, Guanbo Bao, Yinglu Liu, Youzheng Wu, Xiaodong He

arXiv:2610.23558v1cs.CV

TL;DR

Naturally recorded videos rarely provide identity- and pose-matched pairs differing mainly in speech articulation, while mask-based training can mismatch the aligned self-reference used at inference. LiveSync iteratively refines one mask-based model using filtered, aligned pseudo pairs mixed with real data. It achieves the strongest synchronization scores on HDTF and remains competitive on the selected AIGC-LipSync subset.

  • Problem

    Naturally recorded videos rarely provide pairs matched in identity and pose that differ primarily in speech-related articulation, and random-reference mask-based training differs from aligned self-reference at inference.

    Source: §3.1 · p. 3, §3.2 · p. 4

  • Method

    LiveSync uses a mask-based model to generate spatially aligned pseudo pairs, filters and mixes them with real data, and iteratively refines that same model.

    Source: §1 · p. 2

  • Results

    LiveSync achieves the strongest synchronization scores among evaluated methods on HDTF and remains competitive in synchronization on the selected AIGC-LipSync subset.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The approach refines the deployment model without requiring a separate mask-free student or changing its single-pass inference path.

    Source: §6 · p. 11

    The framework is instantiated with a single-pass GAN model; adapting the principle to diffusion-based models remains future work.

    Source: §1 · p. 2

Abstract

from arXiv · show

Audio-driven lip synchronization aims to match target speech through localized mouth and jaw articulation while preserving subject identity, head pose, and surrounding facial context. Naturally recorded videos provide synchronized reconstruction targets but rarely offer pairs that are matched in identity and pose yet differ primarily in speech-related articulation. Mask-based training enables reconstruction from these videos, but random-reference inputs differ from the aligned self-reference used at inference. We present LiveSync, an iterative self-training framework in which a single mask-based model generates pseudo pairs and is refined for deployment. The model is first trained on a curated real-video corpus augmented with externally generated clips covering under-represented visual conditions. A frozen checkpoint then uses independently sampled audio to generate mouth-edited references from real-video frames, copying pixels outside the edit mask from each source frame. After confidence filtering, these references are paired with aligned masked inputs and original real-frame targets. The retained pseudo pairs are mixed with real training data to refine the same model from its current weights, updating training supervision without adding inference stages. Although we instantiate the framework with a single-pass GAN-based model, the self-training strategy could in principle also be adapted to diffusion-based models. Experimental results show that LiveSync achieves leading synchronization performance on HDTF and competitive performance compared with the baselines on the AIGC-LipSync subset, while maintaining low inference latency.

1. Introduction

Naturally recorded videos provide synchronized training targets but rarely provide controlled pairs that vary mainly in speech-related articulation, while mask-based training introduces context loss and a training–inference reference mismatch. LiveSync addresses this gap by generating aligned pseudo pairs and iteratively refining the same mask-based model.

Motivation

  • Naturally recorded videos rarely provide pairs matched in identity and pose that differ primarily in speech-related articulation, limiting controlled supervision for lip-sync editing.Mask-based reconstruction makes synchronized videos usable, but removes local context and commonly trains with random references that differ from the aligned self-reference used at inference.Source: §1 · p. 2

LiveSync

  • Unlike prior two-stage approaches that use a mask-based model only to generate data, LiveSync continuously refines that model as the final model.Aligned self-generated references share the target frame’s pose and surrounding appearance, matching the self-reference configuration used at inference.Source: §1 · p. 2
  • LiveSync generates spatially aligned pseudo pairs from real frames, filters them, mixes retained pairs with real data, and refines the same model from its previous-round weights.The model first trains on curated real-video data augmented with externally generated clips for under-represented visual conditions.Source: §1 · p. 2
  • The aligned supervision is designed to strengthen audio-driven mouth control while preserving source-controlled content, with filtering limiting unreliable references and real-data mixing retaining reference diversity.The approach is particularly relevant to pose and occlusion challenges.Source: §1 · p. 2

Contributions

  • LiveSync maintains low inference latency and ranks second-fastest despite operating at three times the linear face resolution of the fastest compared method.The framework is instantiated with a single-pass GAN-based model, and pseudo-pair synthesis and filtering add no inference stages.Source: §1 · p. 2

2. Related Work

Prior lip-sync work changes generators, reference use, or synchronization objectives, while pseudo-paired methods use generated supervision to train separate mask-free models. LiveSync instead uses aligned pseudo pairs to iteratively refine its mask-based model.

2.1. Mask-based Lip Sync

  • LiveSync shifts the focus from generator or objective changes to supervision, using spatially aligned pseudo pairs to refine the same mask-based model and reduce the training–inference reference gap.Mask-based methods include reconstruction, latent-space inpainting, masking, diffusion, and deformation-based approaches.Source: §2.1 · p. 3

2.2. Pseudo-paired Supervision for Lip Sync

  • SyncAnyone and X-Dub use mask-based generators to create pseudo pairs for separate mask-free models, while TBDub extends this strategy to production-domain footage.OmniSync instead uses a diffusion transformer for direct frame editing across diverse visual conditions.Source: §2.2 · p. 3
  • LiveSync differs from two-stage bootstrapping by using filtered generated references with real target frames to iteratively refine the mask-based model itself.The comparison contrasts this approach with pseudo-pair bootstrapping and direct mask-free diffusion-based editing.Source: §2.2 · p. 3

3. Method

LiveSync addresses the mismatch between random-reference training and aligned self-reference inference by refining one mask-based model with confidence-filtered pseudo pairs. Its method combines spatially aligned supervision with real data and adds no inference stages.

3.1. Problem Setup

  • Naturally recorded videos rarely provide pairs matched in identity and pose that differ mainly in speech articulation, while random-reference training mismatches aligned self-reference inference.LiveSync uses the same mask-based model to generate pseudo pairs and refine itself, rather than training a separate mask-free student.Source: §3.1 · p. 3

3.2. Base Mask-Based Lip-Sync Model

  • At inference, the model edits a masked mouth region using the aligned source frame, while real-data training uses a same-identity random reference to avoid exposing the target mouth.Pixels outside the edit mask are copied unchanged from the source frame.Source: §3.2 · p. 4

3.3. Mask Construction and Boundary Handling

  • LiveSync uses a mouth-centered rectangular mask that removes target lip motion while retaining facial context, and pads out-of-frame face crops without changing their size.Boundary-aware padding preserves face and mask coordinates across frames, including partial out-of-frame cases.Source: §3.3 · p. 4, Figure 3 · p. 5

3.4. Diversity-Oriented Synthetic Data Construction

  • Externally generated clips broaden training coverage of under-represented poses, occlusions, stylized avatars, and non-human characters, while pseudo pairs retain real-frame reconstruction targets.Clips are retained based on face tracking, synchronization, appearance preservation, and visual quality.Source: §3.4 · p. 5

3.5. Training Objective

  • Training combines pixel, perceptual, lip-synchronization, and region-specific adversarial losses; fixed SyncNet supervision compares generated lip crops with corresponding audio.Target frames supply mouth-crop coordinates, while synchronization loss is computed from generated lip clips and audio; separate face and lip critics assess complementary spatial scales.Source: §3.5 · p. 5
  • Reference features are injected into intermediate VAE decoder blocks to restore frame-specific details, with decoder and predictor fine-tuning delayed until synchronization and reconstruction losses stabilize.The delayed update limits premature drift of the pretrained VAE.Source: §3.5 · p. 6

3.6. Pseudo-Paired Data Synthesis

  • A frozen checkpoint uses independently sampled audio to edit a real frame’s mouth while copying pixels outside the mask, forming an aligned pseudo pair with the original frame as target.Pairs are filtered by lip-sync, identity, and image-quality thresholds, and trivial examples with negligible mouth change are discarded.Source: §3.6 · p. 6

3.7. Iterative Self-Training

  • Each refinement round starts from current weights, generates pseudo pairs with a frozen copy, and trains on a real–pseudo mixture, changing supervision rather than the single-pass inference path.Real tuples retain reference diversity, while pseudo tuples expose the model to aligned self-reference inputs.Source: §3.7 · p. 6

4. Experiments

Experiments compare LiveSync with baselines on HDTF and a shared 100-video AIGC-LipSync subset, assessing quantitative performance, latency, qualitative outputs, and cumulative ablations.

4.1. Experimental Settings

  • Evaluation uses HDTF and the same randomly selected 100 AIGC-LipSync videos for all methods; LiveSync is a deterministic, single-pass GAN model.The model uses a pretrained VAE, and the reported metrics cover synchronization, frame and distribution similarity, perceptual similarity, and face identity.Source: §4.1 · pp. 6–7

4.2. Quantitative Comparison

  • At 3.1× Wav2Lip latency, LiveSync was second-fastest while operating at three times Wav2Lip’s linear face resolution under native-resolution timing.The reported latency comparison excludes components that can be processed offline and uses each method’s native resolution.Source: §4.2 · p. 7
  • On HDTF, LiveSync leads Sync-C (9.138) and Sync-D (6.395); on AIGC-LipSync, it ranks second in Sync-D and third in Sync-C.On the AIGC-LipSync subset, LiveSync ranks second in PSNR, SSIM, FID, and CSIM, but LatentSync retains advantages in FVD and LPIPS.Source: §4.2 · p. 7, Table 1 · p. 8

4.4. Ablation Studies

  • The cumulative ablation removes self-generated pseudo pairs first, then also removes adversarial training; “Pair” denotes generated—not real or external—pairs.The variants are evaluated on HDTF and the shared AIGC-LipSync subset.Source: §4.4 · pp. 7–8, Table 2 · p. 8

4.3. Qualitative Comparison

  • Across displayed phonemes, LiveSync produced outputs in every case and retained pose, occluders, and character appearance in shown profiles, occlusions, and stylized subjects.Still frames support appearance and mouth-shape comparison but cannot establish temporal stability or audio–visual synchronization.Source: §4.3 · p. 7, Figure 4 · p. 9

4.4. Ablation Studies

  • Pseudo-pair training improves all eight HDTF metrics and seven AIGC-LipSync metrics, but AIGC-LipSync synchronization changes are small and mixed.On AIGC-LipSync, PSNR changes from 28.661 to 30.863 and FID from 24.219 to 15.370 when pseudo-paired supervision is used.Source: §4.4 · pp. 9–10
  • Across matched ablation examples, identity, pose, and overall appearance stay similar while differences cluster in mouth opening, lip contours, and inner-mouth detail, especially for stylized or non-frontal subjects.These still images show appearance differences but cannot establish synchronization or temporal stability.Source: §4.4 · p. 9, Figure 5 · p. 10
  • With pseudo pairs absent, GAN training lowers FID and FVD on both datasets and improves Sync-C, but generally trades pixel, perceptual, and identity scores for distributional and synchronization metrics.The HDTF Sync-D difference is negligible and slightly favors the non-adversarial variant (6.559 vs. 6.566); full LiveSync has the best value on all eight HDTF metrics and seven AIGC-LipSync metrics.Source: Table 2 · p. 8, §4.4 · p. 10

5. Discussion: Avatar Generation versus Lip Sync

LiveSync treats avatar-generated visual coverage and pseudo-paired supervision as complementary. Lip-sync editing must control articulation within an existing performance rather than generate its whole motion trajectory.

5. Discussion: Avatar Generation versus Lip Sync

  • Avatar generation coordinates complete motion trajectories, whereas lip-sync editing changes articulation within an existing performance while retaining source pose and surrounding motion.Avatar generation may coordinate head motion, expression, and body gestures with speech; lip-sync editing preserves source-specific pose and surrounding motion.Source: §5 · p. 10
  • Natural audio-video coupling can let models exploit expression and gesture correlations instead of learning an independently controllable speech-to-lip mapping.Natural recordings rarely contain combinations where target articulation is independent of source-correlated pose, expression, and body movement.Source: §5 · p. 10
  • Lip-sync models inherit input disruptions, so temporal coherence must balance frame fidelity and smoothing according to input reliability.Source trajectories may include pose changes, frozen or duplicated frames, tracking failures, occlusions, and shot cuts.Source: §5 · p. 10
  • Earlier factorized systems separate mouth motion from source-controlled attributes, reflecting an identifiability problem that remains in pixel- and latent-space models.Examples separate mouth shapes from pose, or geometry, pose, texture, and illumination; the paper says modern models still face the underlying identifiability problem.Source: §5 · p. 10
  • Avatar-generated clips broaden rare visual-condition coverage, while pseudo-paired lip-sync data decorrelate target speech from a fixed source performance.The former cover rare identities, extreme poses, occlusions, stylized characters, and non-human appearances; the latter retain unchanged context outside the edit mask.Source: §5 · p. 11

6. Conclusion

LiveSync refines the same mask-based model with aligned pseudo-references, confidence filtering, and mixed-data training. It leads synchronization scores on HDTF, remains competitive on the selected AIGC-LipSync subset, and ranks second-fastest under the reported latency protocol.

6. Conclusion

  • LiveSync achieves the strongest synchronization scores among evaluated methods on HDTF and remains competitive in synchronization on the selected AIGC-LipSync subset.It also achieves strong results on several visual metrics on that subset.Source: §6 · p. 11
  • LiveSync ranks second-fastest in model-side latency despite using a linear face resolution three times that of the fastest method.The comparison uses a deployment-oriented protocol.Source: §6 · p. 11
  • The same mask-based model generates aligned pseudo-references that preserve real target frames outside the edit mask, then uses confidence filtering and mixed-data training for refinement.The process does not require a separate mask-free student or alter the single-pass inference path; externally generated clips broaden training visual conditions.Source: §6 · p. 11
  • Ablations confirm the benefit of pseudo-paired supervision, and the results suggest self-training can complement advances in model architecture.Source: §6 · p. 11
Loading 2610.23558v1…