PaperlayerView exact revision on arXiv

Source-linked AI summary

Third Time's the Charm? Image and Video Editing with StyleGAN3

Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, Daniel Cohen-Or

arXiv:2201.13433v1cs.CV

TL;DR

StyleGAN3 raises questions about latent-space disentanglement and inversion of unaligned real images, while video editing also requires temporal consistency. The paper analyzes the architecture and develops aligned-generator inversion and editing workflows for images and video. It reports that aligned training remains useful for unaligned imagery, and that aligned-image inversion with transformation control can alleviate inversion challenges.

  • Problem

    The paper asks how StyleGAN3's latent disentanglement compares with StyleGAN2 and whether existing inversion methods can handle real, unaligned images, while video edits must remain temporally consistent.

    Source: §1 · pp. 1–2

  • Method

    The authors analyze StyleGAN3 and use aligned generators in image and video workflows, including aligned-image inversion with transformation control for unaligned inputs.

    Source: §1 · p. 2, §8 · p. 9

  • Results

    Aligned and unaligned generators are comparable at generating unaligned images and controlling position and rotation, while aligned-image inversion with transformation control may alleviate inversion challenges and aligned-only encoder training adds no overhead and gains higher-quality synthesis.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    StyleGAN3's transformation control supports field-of-view expansion on cropped video subjects, reducing the need for challenging seamless stitching.

    Source: §8 · p. 8

    The work focuses on facial images and videos, and further research is needed to assess StyleGAN3 in other domains.

    Source: §8 · p. 9

Abstract

from arXiv · show

StyleGAN is arguably one of the most intriguing and well-studied generative models, demonstrating impressive performance in image generation, inversion, and manipulation. In this work, we explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks. In particular, we demonstrate that while StyleGAN3 can be trained on unaligned data, one can still use aligned data for training, without hindering the ability to generate unaligned imagery. Next, our analysis of the disentanglement of the different latent spaces of StyleGAN3 indicates that the commonly used W/W+ spaces are more entangled than their StyleGAN2 counterparts, underscoring the benefits of using the StyleSpace for fine-grained editing. Considering image inversion, we observe that existing encoder-based techniques struggle when trained on unaligned data. We therefore propose an encoding scheme trained solely on aligned data, yet can still invert unaligned images. Finally, we introduce a novel video inversion and editing workflow that leverages the capabilities of a fine-tuned StyleGAN3 generator to reduce texture sticking and expand the field of view of the edited video.

1. Introduction

Video editing with StyleGAN3 must combine realistic, disentangled edits with temporal consistency, while inversion and editing of real images raise questions about latent spaces and unaligned inputs. The paper analyzes these issues and presents image and video workflows using aligned generators, including a video approach for reducing texture sticking and expanding the field of view.

1. Introduction

  • StyleGAN3 video editing must be temporally consistent as well as disentangled and realistic; texture sticking in StyleGAN1 and StyleGAN2 hinders that consistency.The paper illustrates the issue with hair and face failing to move in unison during latent interpolation.Source: §1 · p. 1
  • StyleGAN3 is examined for latent-space disentanglement and for how well inversion techniques adapt to real, especially unaligned, images.The paper frames these questions alongside the need for video edits to remain temporally consistent despite texture sticking in earlier StyleGAN versions.Source: §1 · pp. 1–2
  • Aligned and unaligned StyleGAN3 generators are comparable at generating unaligned images and controlling position and rotation, but aligned generators are preferable for disentangled editing and inversion.The authors use aligned generators in their image and video inversion and editing workflows.Source: §1 · p. 2
  • The proposed StyleGAN3 video workflow uses the generator to reduce texture sticking and expand the field of view for videos with cropped subjects.The paper also reports reconstruction and editing results and describes generating videos in different styles by training StyleGAN-NADA on GPTI for a video and text prompt.Source: §1 · p. 2

2. Related Work

Prior StyleGAN work develops image manipulation and inversion methods, while video editing has received less attention. This paper builds on existing techniques and StyleGAN3's transformation controls to pursue consistent editing with less reliance on prior video-processing steps.

2. Related Work

  • StyleGAN manipulation methods operate on semantically rich latent spaces, and this work tests techniques from Shen et al. [60], Wu et al. [73], and Patashnik et al. with StyleGAN3 [34].The cited work on StyleGAN2 latent spaces also covers image inversion and manipulation.Source: §2 · p. 2
  • Existing encoder-based inversion methods [6] [55] and generator-tuning techniques [56] are used for StyleGAN3, with tuning needed for higher-fidelity reconstructions across facial expressions.The passage identifies accurate inversion, including of unaligned images, as a goal for adapting the encoder methods.Source: §2 · p. 2
  • Yao et al. [76] use a pretrained encoder and latent-space transformer for consistent video edits, but rely on facial alignment, segmentation, and Poisson blending.The cited passage is truncated after beginning a description of this method's further reliance.Source: §2 · p. 2
  • The paper aims to use StyleGAN3's rotation and translation equivariance for accurate, consistent video editing while reducing the overhead of earlier techniques.The supplied figures show explicit transformation control over Fourier features and pseudo-alignment by replacing w0 with the average latent w.Source: §2 · p. 2, Figure 2 · p. 2, Figure 3 · p. 2

3. The StyleGAN3 Architecture

StyleGAN3 maps an initial latent code into layer-specific inputs and uses Fourier features in place of StyleGAN2's constant input tensor. Its explicit transformation controls and first-layer latent behavior support rotation, translation, and approximate alignment.

3. The StyleGAN3 Architecture

  • A mapping network translates the initial latent code z into an intermediate code w in the learned W space.The initial code is sampled from a 512-dimensional standard normal distribution.Source: §3 · p. 2
  • StyleGAN3 uses 16 synthesis layers at every output resolution, with Fourier features replacing StyleGAN2's constant 4 × 4 input tensor.The first latent input w0 supplies Fourier-feature rotation and translation parameters; later inputs wi are affinely mapped to convolution-weight modulation factors.Source: §3 · p. 2
  • StyleGAN3's StyleSpace S is defined by the outputs of the affine layers and has 9,894 dimensions for a 1024×1024 generator.These outputs provide the style parameters used to modulate the convolutional layers.Source: §3 · p. 2
  • The first two latent inputs determine translation and rotation for practical purposes, while varying w1 also changes other visual attributes.The figure reports that changing w1 affects rotation along with other aspects, and that fixing w0 and w1 preserves pose when later entries are resampled.Source: Figure 4 · p. 3
  • An explicit transformation parameterized by rotation r and translations tx and ty can be concatenated with the generated image's transformation.With all three parameters set to zero by default, this control also lets a generator trained only on aligned data produce rotated and translated images.Source: §3 · p. 3
  • Replacing w0 with the average latent w can make an unaligned generator produce roughly aligned images.The authors attribute this approximate alignment plausibly to roughly aligned, centered average training poses and to translation and rotation being mainly controlled by the first layer; earlier-layer transformations persist through later layers.Source: §3 · p. 3

4. Analysis

The analysis finds that StyleGAN3's style parameters are more disentangled than its W and Z spaces, while rotation and translation depend primarily on the first two latent inputs. The authors also flag unreliable DCI measurements for W+ under their sampling and classifier setup.

4.1. Rotation Control

  • Changing w1 affects in-plane rotation along with face shape and eyes, while fixing w0 and w1 preserves head pose across resampled later entries.The experiments indicate that subsequent layers do not appear to induce further translation or rotation; those effects are determined primarily by w0 and w1.Source: §4.1 · p. 3

4.2. Disentanglement Analysis

  • The disentanglement analysis computes DCI metrics for latent spaces using pretrained attribute regressors and pseudo-aligned images for classifier scoring.Pseudo-alignment is used because the regressors were trained on aligned images and scoring unaligned images is challenging.Source: §4.2 · p. 3
  • S achieves the highest DCI scores across aligned and unaligned StyleGAN3 generators, and the larger W-to-S gap than in StyleGAN2 suggests S may be especially beneficial for editing.The table caption reports DCI scores increasing consistently from Z through W to S for both architectures and both data alignments.Source: §4.2 · p. 3, Table 1 · p. 3
  • W+ DCI metrics are unreliable in this setup because sampled codes produce unnatural images and pretrained classifiers consequently return inaccurate attribute scores.The authors note W+ is of interest because most inversion methods project images into that extended latent space.Source: §4.2 · p. 4

5. Image Editing

The section compares editing in W, W+, and S, finding that S supports more disentangled edits across aligned and unaligned images and multiple domains. W-based editing can also edit unaligned images through either an unaligned generator or an aligned generator with user-controlled transformations.

Editing via Linear Latent Directions

  • Unaligned images can be edited with either an unaligned generator or an aligned generator plus user-controlled rotation and placement transformations.The aligned-generator approach reuses the same latent directions for aligned and unaligned images.Source: §5 · p. 4, Figure 5 · p. 4
  • Comparable edits on unaligned images are possible with aligned and unaligned generators, but directions learned for the unaligned generator are generally more entangled.Pseudo-aligned inputs may remain out of domain for aligned-image classifiers, and linear directions make disentangled editing more challenging.Source: §5 · p. 4

Editing via Non-Linear Latent Paths

  • W+ edits made with StyleCLIP remain entangled, with local changes such as expression altering backgrounds.These results motivate testing whether editing in S is more disentangled than editing in W and W+.Source: §5 · p. 4, Figure 6 · p. 4

Editing via Latent Directions in S

  • StyleCLIP global directions are used to edit in S across generators trained on FFHQ, AFHQv2, and Landscapes HQ.The analysis focuses on aligned generators because they produce higher-quality images and can also edit unaligned images.Source: §5 · pp. 4–5, Figure 7 · p. 5
  • S-space editing preserves unrelated image regions across aligned and unaligned human faces, animal faces, and landscapes.Examples include preserved backgrounds and identity in faces, fur color and pose in animals, and lakes or roads in landscapes.Source: §5 · p. 5

6. StyleGAN3 Inversion

The paper formulates StyleGAN3 inversion as finding a latent code that reconstructs a target image, and uses learned encoders to support image editing and video workflows. Its encoder is trained on aligned images but can reconstruct unaligned inputs using estimated rotation and translation; encoder comparisons show slightly worse reconstruction performance than StyleGAN2, while editability differs between the ReStyle variants.

6. StyleGAN3 Inversion

  • Inversion seeks a latent code that reconstructs a target image, with L2 or LPIPS reconstruction loss.A learned encoder maps input images to latent codes; the reconstructed image can also use transformations for translation and rotation, followed by latent manipulation for editing.Source: §6 · p. 5

6.1. Designing the Encoder Network

  • Training an encoder on aligned and unaligned images with an unaligned generator struggles to capture the variability of unaligned faces, resulting in poor reconstructions.The proposed alternative trains an encoder solely on aligned images, avoiding the need to capture variable placement and pose during encoder training.Source: §6.1 · p. 5
  • At inference, the method aligns an unaligned input, encodes it, and estimates rotation and translation to reconstruct its original placement.The encoder receives the aligned image, while the extracted transformation lets the reconstruction capture the unaligned input's placement and rotation.Source: §6.1 · p. 6

6.2. Inverting Images into StyleGAN3

  • StyleGAN3 encoders produce visually comparable reconstructions to StyleGAN2 encoders and reproduce input position using landmark-based predicted transformations.The quantitative comparison covers CelebA-HQ facial images and reports inference time, L2, LPIPS, identity similarity, and MS-SSIM; the supplied table passage gives its scope but not its values.Source: §6.2 · p. 6, Table 2 · p. 6, Figure 8 · p. 6
  • StyleGAN3 encoders perform slightly worse than StyleGAN2 encoders, which the authors partly attribute to StyleGAN3's less well-behaved W+ space.They connect this difficulty to rapid quality deterioration when moving away from W, which they believe contributes to encoder-training challenges.Source: §6.2 · p. 6
  • ReStyle-e4e produces realistic, meaningful edits while preserving identity, whereas ReStyle-pSp is less editable despite high-quality reconstructions.The editing-quality gap between these encoders is larger for StyleGAN3 than for StyleGAN2.Source: §6.2 · p. 6, Figure 9 · p. 7

7. Inverting and Editing Videos

The video pipeline aligns cropped frames for encoding, then combines edited latent codes with estimated transformations to generate unaligned frames. Temporal smoothing improves coherence, while field-of-view expansion recovers cropped regions for more complete edits.

Video Preprocessing

  • The pipeline crops a fixed bounding box across frames for a stable-camera appearance, then aligns each crop and estimates its transformation from the unaligned frame.The encoder operates on aligned frames, while the estimated rotation and translations support unaligned reconstruction.Source: §7 · p. 7

Initial Video Encoding

  • The trained ReStylee4e encoder maps each aligned frame to a latent code, which the generator uses with the estimated transformation to reconstruct or edit it.Applying a manipulation to the latent code produces the edited frame.Source: §7 · p. 7

Latent Vector Smoothing

  • Temporal smoothing of edited latent codes and Fourier-feature transformations improves coherence without harming reconstruction quality.Independent frame inversions can create discontinuities from alignment, the encoder, or latent manipulation.Source: §7 · p. 7

Pivotal Tuning for Improved Reconstructions

  • Pivotal tuning fine-tunes StyleGAN3 generator weights using the original unaligned frames to improve their reconstructions.The encoder is trained on aligned images, but PTI losses are computed against unaligned inputs.Source: §7 · p. 7

Video Results

  • The paper reports reconstruction and editing examples, including videos stylized by training StyleGAN-NADA on the PTI-modified generator with a text prompt.The examples include a cartoon-style video of Obama.Source: §7 · p. 7, Figure 11 · p. 8

Expanding the Field of View

  • Field-of-view expansion combines generated images to recover the full head, enabling coherent edits where StyleGAN2 can leave mismatches between edited and untouched regions.The workflow adds non-overlapping regions from a shifted image to widen the output frame.Source: §7 · p. 8

8. Conclusions

The conclusions highlight StyleGAN3's translation and rotation control and field-of-view expansion for video editing, while noting greater latent entanglement and unresolved questions. The accompanying background and fine-tuning analysis situate its latent spaces, editing and inversion methods, and preserved edit directions.

8. Conclusions

  • Field-of-view expansion supports more consistent editing of cropped video frames and can avoid cumbersome seamless stitching, but the paper leaves StyleGAN3's overall advantage unresolved.The authors identify facial images and videos as their focus and call for investigation in other domains.Source: §8 · pp. 8–9
  • StyleGAN3's latent space is somewhat more entangled than its predecessors', making inversion harder and frame inversions less robust across a video.The authors report alleviating this with aligned-image inversion and transformation control; aligned-only encoder training adds no overhead and yields higher-quality synthesis.Source: §8 · pp. 8–9

A.1. StyleGAN Latent Spaces

  • StyleGAN2's W, W+, and S spaces differ in disentanglement; W supports editing, W+ is commonly used for inversion, and S is described as still more disentangled.The background notes that these spaces may suit different tasks.Source: §A.1 · p. 13

A.2. Editing Images with StyleGAN2

  • StyleGAN2 editing work spans supervised, unsupervised, and self-supervised direction discovery, code mixing, local editing, and text-based methods.The cited approaches use semantic labels, latent codes, semantic maps, reference images, and CLIP models.Source: §A.2 · p. 13

A.3. Inverting Images with StyleGAN2

  • GAN inversion methods optimize latent vectors, train encoders, or combine approaches; some also tune the generator per image or with a learned hypernetwork.Inversion is required to apply editing techniques to real images.Source: §A.3 · p. 13

B.1. Preservation of Properties Under Fine-Tuning

  • StyleGAN3 fine-tuned child generators retain latent-space alignment: the parent generator's editing directions have similar effects in StyleGAN-NADA fine-tuned models.The paper illustrates this with the same directions applied across fine-tuned generators.Source: §B.1 · p. 13, Figure 13 · p. 13

B.2. Disentanglement Analysis of W+

The W+ comparison is limited because random samples are unrealistic and StyleGAN3 samples contain more artifacts, making DCI evaluation unreliable. The section also describes landmark-based transformations and a face-size constraint for inverting unaligned images.

B.2. Disentanglement Analysis of W+

  • Random W+ samples from both aligned generators are unrealistic, with significantly more artifacts in StyleGAN3, so the authors omit W+ DCI metrics as unreliable.The figure compares uncurated samples from the two generators.Source: §B.2 · p. 13, Figure 14 · p. 14

C. Landmarks-Based Transformations

  • Landmark-based control estimates rotation from eye-line angles and translation from the left-eye displacement between aligned and unaligned images.These values define the user-specified transformations passed to StyleGAN3's Fourier features.Source: C. Landmarks-Based Transformations · p. 14, Figure 15 · p. 14
  • Unaligned inversion is restricted to faces matching the size used to train the unaligned FFHQ StyleGAN3 generator, so inputs are cropped to that size first.The preprocessing is similar to the official FFHQ-U dataset procedure.Source: C. Landmarks-Based Transformations · p. 14

D. Editing: Additional Details

  • For W-space editing, the authors use InterFaceGAN with attribute-specific classifiers for pose, age, and other attributes.The passage names HopeNet for pose, Rothe et al. for age, and Lin et al.'s classifier for the remaining attributes.Source: D. Editing: Additional Details · p. 14

E.1. Additional Training Details

The StyleGAN3 encoders use familiar pSp and e4e architectures but are trained only on aligned images, while comparisons evaluate aligned CelebA-HQ reconstructions. Training details include reconstruction losses, an e4e training change, and batch-size adjustments for memory use.

E.1. Additional Training Details

  • The StyleGAN3 pSp- and e4e-based encoders are trained solely on aligned images, simplifying the encoder training objective.The paper retains the encoder architectures used for StyleGAN2.Source: §E.1 · p. 14
  • Training combines L2 pixel, LPIPS perceptual, and identity-based reconstruction losses with weights 1, 0.8, and 0.1, respectively.The weights are λ_l2 = 1, λ_lpips = 0.8, and λ_id = 0.1.Source: §E.1 · p. 14
  • ReStylee4e predicts all 16 latent codes simultaneously from the start instead of using progressive training, which the authors report leads to faster convergence.Source: §E.1 · p. 14

E.2. Baselines and Comparisons

  • Encoders are trained on 70,000 aligned FFHQ images and quantitatively compared on aligned CelebA-HQ test reconstructions against StyleGAN2 counterparts.StyleGAN3 qualitative outputs are shown unaligned, whereas StyleGAN2 outputs are shown aligned; StyleGAN3 training uses batch size 2 with gradient accumulation to an effective batch size of 8.Source: §E.2 · p. 14, §E.1 · p. 14

E.3. Ablation Study

The ablation finds that the aligned-trained encoder reconstructs unaligned poses better than the unaligned-training alternative. The video workflow also specifies preprocessing, generator tuning, smoothing, and field-of-view expansion procedures.

E.3. Ablation Study

  • The unaligned-training alternative uses paired images and an unaligned generator, but pseudo-alignment supports identity loss while L2 and LPIPS compare unaligned reconstructions.Replacing the first latent code with the generator's average latent code produces the pseudo-aligned representation used for identity loss.Source: §E.3 · p. 15
  • The aligned-trained encoder yields superior reconstructions to the unaligned-trained encoder while faithfully capturing unaligned poses through user-defined transformations.The transformation procedure supplies the desired pose, letting the encoder focus on identity reconstruction.Source: §E.3 · p. 15, Figure 16 · p. 15

F.1. Video Preprocessing

  • Video preprocessing uses eye distance consistently across frames and assumes the camera-to-face distance does not change throughout the video.Using eye-to-mouth distance may vary across frames; the authors report slightly more stable reconstruction with eye distance alone.Source: §F.1 · p. 15

F.2. Pivotal Tuning of Videos

  • Per-video pivotal tuning runs 8,000 optimization steps at batch size 2, with equally weighted L2 and LPIPS losses, while leaving Fourier-feature layer weights unchanged.A 200-frame video therefore shows each frame an average of 40 times during training.Source: §F.2 · p. 15

F.3. Latent Vector Smoothing

  • Without latent smoothing, video reconstruction is unstable because of pre-alignment and per-frame encoder inversions.The comparison is provided in the accompanying video results.Source: §F.3 · p. 15

F.4. Field-of-View Expansion

  • Field-of-view expansion shifts generated images in desired directions, then joins their non-overlapping regions with the original reconstruction.For horizontal and vertical expansion together, an additional diagonal shift fills the corner regions.Source: §F.4 · p. 16

G. Additional Results

Additional results show editing, inversion, and domain adaptation across StyleGAN3 latent spaces and image domains. They also demonstrate video editing and a field-of-view expansion technique on video sequences.

Additional editing results

  • StyleGAN-NADA fine-tuning results show domain adaptation of pretrained StyleGAN3 generators across multiple domains.The parent and child generator latent spaces remain aligned under fine-tuning.Source: G. Additional Results · p. 16, Figure 21 · p. 21

Additional video results

  • The full StyleGAN3 video pipeline produces reconstructions and edits, with StyleGAN-NADA domain adaptation applied to edited frames for varied styles.Examples use StyleCLIP [48] and InterFaceGAN [60] edits alongside Pixar and Sketch styles from StyleGAN-NADA [21].Source: G. Additional Results · p. 16, Figure 26 · p. 26, Figure 27 · p. 27

Additional editing results

  • Non-linear StyleCLIP mapping edits in W+ remain entangled, with local expression or hairstyle changes also altering background or identity.The example uses an aligned StyleGAN3 face generator.Source: G. Additional Results · p. 16, Figure 17 · p. 17
  • StyleCLIP global-direction edits in S are shown for aligned FFHQ, AFHQv2, and Landscapes HQ generators, including randomly transformed unaligned outputs for FFHQ and AFHQv2.The unaligned outputs use sampled rotation and translation parameters on the input Fourier features.Source: G. Additional Results · p. 16, Figure 18 · p. 18, Figure 19 · p. 19
  • Editing directions from the parent FFHQ StyleGAN3 generator have similar effects in fine-tuned child generators, indicating semantic alignment is preserved.The examples use child generators fine-tuned with StyleGAN-NADA.Source: G. Additional Results · p. 16, Figure 22 · p. 22

Additional inversion results

Additional editing results

  • Real-image edits compare latent codes from different inversion methods using editing techniques from Shen et al. [60] and Patashnik et al..For StyleGAN3 encoders, unaligned originals are reconstructed using predicted landmark-based user transformations.Source: G. Additional Results · p. 16, Figure 25 · p. 25

Additional video results

  • The field-of-view expansion technique completes cropped regions in video reconstructions, including a woman's chin and a man's hair.Figure 28 shows the technique on multiple video sequences.Source: G. Additional Results · p. 16, Figure 28 · p. 28
Loading 2201.13433v1…