Source-linked AI summary
StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator
Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, Jingdong Wang
TL;DR
Existing lip-sync methods can require long-term person-specific data or produce similar movements across subjects, while whole-head generation does not meet the need to alter a face within an existing scene. StyleSync combines masked spatial encoding and a style-based generator with few-shot person-specific optimization. The paper reports high-fidelity one-shot results that outperform prior methods, with personalization further improving generation quality.
Problem
Existing methods struggle to balance generation quality and generalization, and whole-head generation can be difficult to blend into an existing scene.
Source: §1 · p. 2Method
StyleSync modifies a style-based generator with masked spatial encoding and uses style-space and generator refinement for personalization from limited target frames.
Source: §1 · p. 2Results
StyleSync outperforms previous methods on one-shot lip-syncing, and personalized optimization further improves generation quality.
Source: §1 · p. 2Takeaways & Limitations
The framework produces high-fidelity lip-sync results in one-shot and few-shot settings, with personalization using only a few clips.
Source: §5 · p. 8With a fixed mask, the method cannot change the target person’s head pose or expressions.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Despite recent advances in syncing lip movements with any audio waves, current methods still struggle to balance generation quality and the model's generalization ability. Previous studies either require long-term data for training or produce a similar movement pattern on all subjects with low quality. In this paper, we propose StyleSync, an effective framework that enables high-fidelity lip synchronization. We identify that a style-based generator would sufficiently enable such a charming property on both one-shot and few-shot scenarios. Specifically, we design a mask-guided spatial information encoding module that preserves the details of the given face. The mouth shapes are accurately modified by audio through modulated convolutions. Moreover, our design also enables personalized lip-sync by introducing style space and generator refinement on only limited frames. Thus the identity and talking style of a target person could be accurately preserved. Extensive experiments demonstrate the effectiveness of our method in producing high-fidelity results on a variety of scenes. Resources can be found at https://hangz-nju-cuhk.github.io/projects/StyleSync.
1. Introduction
StyleSync targets high-fidelity lip synchronization that modifies a face within an existing scene while preserving the rest of the scene. It combines a style-based generator with masked spatial encoding and limited-data personalization.
1. Introduction
- StyleSync modifies the mouth or facial area while preserving the rest of an existing scene, unlike methods that generate a whole talking head.The paper identifies seamless alteration of an existing video as a need in scenarios such as audio dubbing.Source: §1 · p. 2
- The framework targets two challenges: balancing accurate audio expression with target-frame preservation, and adding personalized properties to a generalized model.The paper notes that lip-sync quality and preservation of the original frame can conflict, while incorporating few-shot personalization into lip-syncing had not been explored.Source: §1 · p. 2
- StyleSync modifies a style-based generator for lip-syncing, using masked mouth modeling and mask-based spatial information encoding alongside audio and reference-frame information.Audio dynamics and high-level reference-frame information are injected through style-modulated convolutions.Source: §1 · p. 2
- The personalized optimization uses only a few seconds of a target person’s video and adjusts person-specific parameters, including W+ and the generator.The paper says this preserves speaking styles and details of the mouth and jaw.Source: §1 · p. 2
- Experiments report that StyleSync outperforms prior methods in one-shot lip-syncing, while personalized optimization further improves generation quality.The paper characterizes its one-shot results as high-fidelity and reports a clear margin over previous methods.Source: §1 · p. 2
2. Related Work
Prior audio-driven facial animation methods either generate whole heads or modify only the mouth, with trade-offs in data needs, generalization, and image quality. StyleSync draws on style-based generation and inversion while addressing spatial consistency and personalization.
2.1. Audio-Driven Facial Animation
- Whole-head generation methods often rely on facial structural representations, whose uncertainty can accumulate; some person-specific approaches also generalize poorly or need more than two minutes of data.The paper also notes that NeRF-based person-specific models perform poorly with limited data.Source: §2.1 · pp. 2–3
- Mouth-only approaches preserve other video information, but person-specific methods require long training clips, while Wav2Lip produces blurry low-resolution results and misses personalized patterns.The paper describes Wav2Lip as highly accurate in person-agnostic lip synchronization.Source: §2.1 · p. 3
2.2. Style-based Generator for Faces
- StyleGAN inversion recovers a given image’s latent space; extending W to W+ better preserves image details, and pivotal finetuning refines generator parameters.StyleSync encodes W+ from reference frames to support personalized lip-movement learning.Source: §2.2 · p. 3
- Prior facial-animation methods use W or W+ to control appearance and motion, but W+ does not readily preserve background spatial consistency and can produce artifacts.The paper contrasts this with face-restoration and swapping tasks that aim to preserve facial expression and emotion.Source: §2.2 · p. 3
3. Methodology
StyleSync uses a StyleGAN2-based pipeline to reconstruct a masked target face from audio and a reference frame. It combines mask-guided spatial encoding and style-space conditioning with reconstruction, adversarial, and synchronization training, then personalizes the model using a few-shot template video.
3. Methodology
- The task masks the lower face, including jaw and cheeks, and reconstructs the target face from audio and a reference frame.Training follows self-reconstruction; the reference frame supplies context for the masked mouth and jaw.Source: §3 · p. 3
3.1. Modifying Style-based Generator for Lip Sync
- StyleSync modifies StyleGAN2, whose generator uses style vectors to modulate convolution layers and supports layer-specific vectors in W+.The paper uses this generator as the backbone for lip-sync-oriented modifications.Source: §3.1 · p. 3
- The model concatenates masked and reference frames, encodes their features as spatial noise, and masks low-level face features to limit reference-driven changes to mouth shape.The paper describes this as mask-based spatial information encoding for preserving spatial structure while generating different mouth dynamics.Source: §3.1 · pp. 3–4
- Audio dynamics and facial information are encoded into W by concatenating the audio feature with the face-encoder bottleneck, though the paper reports subtle differences with or without the face feature.Reference-frame information is already supplied through spatial noise encoding.Source: §3.1 · p. 4
3.2. Backbone Training Objectives
- Generalized training predicts reconstructed frames from masked and reference images plus audio, using reconstruction, adversarial, and synchronization objectives.The reconstruction term combines L1 and perceptual losses; the adversarial discriminator is initialized from StyleGAN2, and SyncNet supervises five consecutive predicted frames.Source: §3.2 · pp. 4–5
- The generalized loss combines adversarial, reconstruction, and synchronization losses.The paper denotes their weighted combination as L_g = L_adv + λ_r L_rec + λ_s L_sync.Source: §3.2 · p. 5
3.3. Personalized Optimization
- Personalization addresses similar mouth-motion patterns across identities by training on a template video with less than one minute of footage.The audio encoder is fixed, and the face encoder is not optimized on a single person to avoid overfitting.Source: §3.3 · p. 5
- The personalized procedure learns W+ displacements from the face feature and allows limited generator-parameter shifts, while retaining the generalized loss and penalizing parameter changes.The paper restricts the displacements to a limited step.Source: §3.3 · p. 5
4. Experiments
Experiments on LRW and VoxCeleb2 evaluate StyleSync in generalized and personalized settings. The results report strong generation quality, competitive lip synchronization, and gains in identity preservation from personalization, alongside trade-offs in mouth opening and synchronization scores.
4. Experiments
- Experiments use LRW and VoxCeleb2 with their original train/test splits, and compare generalized StyleSync against MakeitTalk, PC-AVS, Wav2Lip, and reproduced Wav2Lip-H.Personalized optimization is also compared with AD-NeRF.Source: §4 · pp. 5–6
4.1. Quantitative Evaluation
- On LRW, generalized StyleSync outperforms previous methods by a large margin for generation quality; the gap is less obvious on VoxCeleb2.The authors suggest that predominantly frontal LRW training data versus more complicated VoxCeleb2 scenes might explain the difference.Source: §4.1 · pp. 6–7
- Personalized optimization makes generated results more similar to targets, while lip-sync scores degrade after personalization.The authors associate the SyncNet decline with target speaking style and less pronounced mouth opening.Source: §4.1 · p. 7
- On both datasets, StyleSync achieves comparable lip-sync performance, with slightly better LMD scores than competing methods.On LRW, its SyncNet score is comparable and closer to ground truth; the authors caution that SyncNet confidence above ground truth does not imply better perceptual synchronization.Source: §4.1 · p. 7
4.2. Qualitative Evaluation
- In cross-driven comparisons, generalized StyleSync produces the most accurate mouth shapes and highest-fidelity results among the compared methods.MakeitTalk lacks accurate lip movements, while PC-AVS and Wav2Lip show visible artifacts or blurriness; reproduced Wav2Lip-H produces plausible results for most cases.Source: Figure 3 · p. 6, §4.2 · p. 7
- In the user study, StyleSync outperforms its counterparts by a large margin in lip-sync quality, generation quality, and video realness.The study used Mean Opinion Scores from 1 to 5; Wav2Lip-H had less blurry mouth textures than PC-AVS and Wav2Lip.Source: §4.2 · pp. 7–8, Table 2 · p. 8
4.3. Ablation Study
- Removing the mask or SyncNet loss preserves comparable image-quality metrics but reduces lip-sync performance; the mask also alleviates reference-frame influence.Without SyncNet supervision, the visual results show inconsistent lip movements relative to the synced video.Source: §4.3 · p. 8
- Personalization makes identity and mouth-opening patterns more like the template, while the mouth opens less; removing ∆W slightly degrades identity scores.The authors attribute the less obvious mouth opening to the target person's talking style; ∆W removal leaves image-quality and lip-sync scores similar.Source: §4.3 · p. 8
5. Conclusion and Discussions
StyleSync produces high-fidelity lip-sync results in one-shot and few-shot settings, with a generalized model and personalized optimization. Its fixed-mask blending constrains changes to the target’s head pose and expressions, and extreme jaw sizes may exceed the mask.
5. Conclusion and Discussions
- The generalized StyleSync model clearly outperforms previous state-of-the-art methods in video results.Source: §5 · p. 8
- StyleSync builds on style-based generators with simple modifications and is described as easy to implement and friendly to train.Source: §5 · p. 8
- Personalized optimization can further improve results for a specific person using only a few clips.Source: §5 · p. 8
- Because blending uses a fixed mask, the method cannot change the target’s head pose or expressions, and an extremely large jaw may extend beyond the masked area.Source: §5 · p. 8
- The authors note that the method could be misused to create non-existing talks and speeches, and state that core models will be issued strictly to research institutions.Source: §5 · p. 8