Source-linked AI summary
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, Yujiu Yang
TL;DR
One-shot talking face generation seeks high-quality, generalizable animation from a single portrait despite limited training resolution and spatial-control challenges. StyleHEAT uses a pre-trained StyleGAN’s feature space with motion generators and feature refinement to provide video/audio-driven animation and editing. The framework produces 1024×1024 talking faces and reports superior video quality, controllability, and editability, while remaining limited by inversion errors and facial occlusions.
Problem
Subject-agnostic one-shot talking-face generation has limited source information and prior methods remain constrained in resolution and spatial motion control.
Method
StyleHEAT transforms pre-trained StyleGAN feature maps using video- or audio-based motion generators, then refines them with calibration and domain-loss components.
Results
The framework generates 1024×1024 talking faces and supports video-driven, audio-driven, jointly driven, and editable reenactment.
Takeaways & Limitations
Pre-trained StyleGAN features provide a unified basis for high-resolution talking-face generation with disentangled motion control and flexible face editing.
Takeaways & Limitations
The framework cannot currently handle facial occlusions, producing notable artifacts around occluded regions.
Abstract
from arXiv · showhide
One-shot talking face generation aims at synthesizing a high-quality talking face video from an arbitrary portrait image, driven by a video or an audio segment. One challenging quality factor is the resolution of the output video: higher resolution conveys more details. In this work, we investigate the latent feature space of a pre-trained StyleGAN and discover some excellent spatial transformation properties. Upon the observation, we explore the possibility of using a pre-trained StyleGAN to break through the resolution limit of training datasets. We propose a novel unified framework based on a pre-trained StyleGAN that enables a set of powerful functionalities, i.e., high-resolution video generation, disentangled control by driving video or audio, and flexible face editing. Our framework elevates the resolution of the synthesized talking face to 1024*1024 for the first time, even though the training dataset has a lower resolution. We design a video-based motion generation module and an audio-based one, which can be plugged into the framework either individually or jointly to drive the video generation. The predicted motion is used to transform the latent features of StyleGAN for visual animation. To compensate for the transformation distortion, we propose a calibration network as well as a domain loss to refine the features. Moreover, our framework allows two types of facial editing, i.e., global editing via GAN inversion and intuitive editing based on 3D morphable models. Comprehensive experiments show superior video quality, flexible controllability, and editability over state-of-the-art methods.
1 INTRODUCTION
One-shot talking face generation must animate arbitrary portraits under a subject-agnostic setting while overcoming the limited resolution of existing training data. StyleHEAT addresses this with a pre-trained StyleGAN framework that supports high-resolution generation, video/audio control, and face editing.
- Task and challenges: One-shot talking face generation transfers identity from a portrait while deriving pose and expression from driving video or audio.The task supports applications including digital human animation and film production.
- Task and challenges: Subject-agnostic one-shot generation is challenging because it uses minimal source information, reducing synthesis quality compared with subject-dependent methods.Subject-dependent methods learn person-specific models from long source videos, whereas one-shot methods must generalize across subjects.
- Task and challenges: Existing methods commonly generate 256×256 frames, while 512×512 approaches remain bounded by newly collected high-resolution training data.Higher resolution also requires carefully designed network architectures and training strategies.
- Proposed framework: StyleHEAT investigates StyleGAN feature-space transformations and uses them to build a unified framework for high-resolution, controllable, and editable talking faces.The framework exploits spatial transformations of latent feature maps rather than relying solely on style codes.
- Proposed framework: The framework reaches 1024×1024 resolution without training on new datasets and supports video-driven, audio-driven, or jointly driven motion generation.A calibration network and domain loss refine artifacts introduced when predicted motion spatially warps StyleGAN feature maps.
- Proposed framework: Face editing is supported globally through GAN inversion and intuitively through 3D morphable models.Global attributes are edited through inverted style codes, while 3DMM-based editing supports pose and expression manipulation.
2 RELATED WORK
Prior talking-head methods use 3D face priors, landmarks, image translation, neural rendering, or StyleGAN spaces, but remain limited in generalization, controllability, spatial fidelity, or resolution. StyleHEAT instead exploits StyleGAN’s spatial feature space for controlled high-resolution animation and editing.
- Talking-head generation: Subject-specific image-translation and fine-tuning methods can model individual identities but require target footage or samples and struggle in complex real-world settings.Subject-agnostic methods reduce identity-specific requirements but must generalize from limited source information.
- Resolution limitations: Previous talking-head methods generally cannot generate high-resolution videos because high-resolution data are difficult to collect and networks require specialized design.Style-based encoders had been used before, but prior approaches focused on implicit style-convolution modularization rather than the spatial feature space.
- StyleGAN spaces: W/W+ spaces support semantic face editing but lack explicit spatial priors, causing motion-control and alignment limitations for video generation and out-of-alignment images.W+ trajectory methods can randomly sample motion without control, and content may be corrupted when pose changes substantially.
- StyleGAN spaces: F space offers spatially local editing and restoration capabilities, motivating StyleHEAT’s investigation of geometric transformations for talking-face generation.Prior examples include segmentation-guided compositing, blind face restoration, and spatial semantic manipulation.
3 INVESTIGATING FEATURE SPACE OF STYLEGAN
The section examines whether StyleGAN’s intermediate feature space supports geometric transformation, addressing limitations of W+ space for controlled talking-face generation. Experiments identify spatially meaningful feature behavior and motivate editing in F space.
- Motivation: W+ space lacks explicit spatial prior, causing uncontrolled motion and content corruption when pose changes, while its edits cannot produce out-of-alignment images.These limitations arise because StyleGAN is trained on aligned faces and W+ is highly semantically condensed.
- F-space properties: F space represents spatial feature maps at generator scales and has been explored for inversion, composition, and face enhancement.The section focuses on whether these spatial features remain realistic after geometric transformations.
- Layer selection: The layer experiment warps feature maps at multiple resolutions to determine which layer most accurately controls pose and expression.The tested feature maps range from f_4×4 through f_1024×1024.
- Spatial transformation experiment: Applying affine transformations to f_64×64 with fixed w preserves identity and appearance with only minor differences.This observation indicates translation-invariant behavior in the pretrained generator’s convolutional kernels.
- Spatial transformation experiment: The observed feature-space transformation corresponds closely to image-space geometry after adjusting for relative scale, supporting F-space editing for talking-face generation.The framework formalizes image generation as I = G(f,w) and maps image transformation T to feature operator T′.
4 METHODOLOGY
StyleHEAT uses pre-trained StyleGAN feature-space transformations for one-shot talking-face generation, with video- or audio-driven motion, feature calibration, and joint training for high-resolution outputs.
- Unified Framework: The framework inverts a source portrait into StyleGAN latent codes and feature maps, then predicts dense flow fields from video or audio to inject motion.The flow fields manipulate StyleGAN feature maps for talking-face generation.
- Video-Driven Motion Generator: 3DMM motion descriptors encode expression, head rotation, and translation, while neighboring frames are windowed to improve temporal consistency.The motion descriptor excludes identity coefficients and uses a temporal window around each driving frame.
- Video-Driven Motion Generator: The video-driven generator uses a U-Net with AdaIN-conditioned convolutional layers to produce flow fields from the source image and driving video.The generator is trained by comparing backward-warped outputs with target frames using perceptual and related losses.
- Audio-Driven Motion Generator: The audio-driven generator predicts lower-half-face flow fields directly from Mel-Spectrogram features, avoiding intermediate visual-semantic parameters.Its training pairs are constructed with the pre-trained video-driven generator because matched-pose, different-lip-shape audio data are difficult to obtain.
- Fully-Controllable Motion Fields: Video- and audio-driven generators can be combined to control head motion and lip movement independently.The two generators are trained separately before joint use in the unified framework.
- Feature Calibration and Joint Training: A U-Net calibration network applies spatial feature transformation to warped features, while domain loss and joint training address artifacts and blur.The calibrated features are passed to StyleGAN to produce the final high-quality, high-resolution image.
5 EXPERIMENTS
Experiments evaluate training, controllability, editing, image quality, and ablations. The method produces detailed high-resolution faces, supports video and audio control, and benefits from calibration and domain loss.
- Implementation Details: The framework trains motion generators on VoxCeleb and the full system on HDTF, using staged optimization for motion and calibration.The video generator and calibration network are jointly optimized before the audio generator is trained with the video generator fixed.
- High-Resolution Talking Face Generation: The method produces more facial detail than PIRenderer, including visible pupils, wrinkles, splashes, and individual hairs.PIRenderer appears blurry because of its lower resolution, whereas the proposed results retain details around the eyes, mouth, and hair.
- High-Resolution Talking Face Generation: Face restoration raises competing methods to 1024×1024 but introduces smoothing, color and lighting changes, and does not remove generated artifacts.The proposed method preserves skin tone better and retains fine hair and brow details while avoiding these reported restoration side effects.
- Video-Driven Face Reenactment: The method transfers pose and expression more reliably than competing methods, especially for large expression differences and cross-identity face-shape gaps.Other methods show mouth, head, or gaze distortions, while the proposed method generates clearer teeth and works more stably across identities.
- Quantitative Evaluation: Quantitatively, the method achieves the best FID in the evaluated reenactment cases and the best cross-identity CSIM, while AED and APD are comparable to PIRenderer.Its lower same-identity CSIM is attributed to identity information lost during GAN inversion.
- Audio-Driven Talking Face Generation: Audio-driven lip movements match ground-truth motion, and joint audio-video control assigns pose to video and lip movement to audio across identities.Compared with wav2lip, the proposed results have better visual quality, with sharper mouths and synthesized teeth.
- Ablation Study: Ablations show that calibration improves eyes and mouth, while domain loss preserves facial details that otherwise become blurry or disappear.Without calibration, the model cannot correctly generate teeth and closed eyes; removing domain loss loses wrinkles and hair texture.
6 LIMITATION AND DISCUSSION
The framework has limitations in reconstructing source details, handling facial occlusions, and maintaining coherent motion between hair and face textures.
- GAN inversion cannot completely avoid reconstruction errors because information is lost in feature maps, severely distorting headband textures.
- Facial occlusions produce notable artifacts around the occluded region in synthesized images, a problem shared by existing talking face methods.
- StyleGAN2 can produce texture-sticking artifacts in which hair and face do not move in unison during synthesized videos.
7 CONCLUSION
The paper proposes a pre-trained StyleGAN framework for one-shot talking face generation with high resolution, multiple driving modes, and face editing. Its components support video-, audio-, and jointly driven reenactment alongside two editing mechanisms, evaluated through ablations and comparisons with state-of-the-art methods.
- The framework uses a pre-trained StyleGAN for high-resolution talking video generation, disentangled video and audio control, and flexible face editing.
- It combines video-based and audio-based motion modules with a calibration network and pre-trained StyleGAN to generate talking faces.
- The system supports video-driven, audio-driven, and jointly driven reenactment, plus global GAN inversion and intuitive 3DMM-based editing.
- Comprehensive experiments include ablations and comparisons with many state-of-the-art methods to assess the framework's capabilities.