PaperlayerView exact revision on arXiv

Source-linked AI summary

SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing

Yichun Shi, Xiao Yang, Yangyue Wan, Xiaohui Shen

arXiv:2112.02236v3cs.CV

TL;DR

StyleGAN priors support image synthesis and editing, but holistic latent controls make fine-grained local editing difficult. SemanticStyleGAN factorizes a generator by semantic parts, with codes controlling each part’s structure and texture. Experiments report disentangled local synthesis and finer edits, as well as adaptation to image-only datasets while preserving disentanglement.

  • Problem

    StyleGAN latent codes provide global-style controls, making fine-grained control over synthesized images difficult.

    Source: §1 · pp. 1–2

  • Method

    SemanticStyleGAN composes images from semantic-part generators whose local latent codes control each part’s structure and texture.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Experiments report high-quality synthesis with disentangled local parts, finer editing when combined with other editing methods, and image-only domain adaptation that preserves disentanglement.

    Source: §5.2.1 · p. 6, §7 · p. 8, §5.3 · p. 8

  • Takeaways & Limitations

    The model serves as a generic prior that can be combined with StyleGAN editing methods to provide more precise local control.

    Source: §1 · p. 2, §7 · p. 8

    The method requires a local generator for each class, limiting its scalability to datasets with many semantic classes; its fully supervised training also prevents training from scratch on image-only datasets.

    Source: §6 · p. 8

Abstract

from arXiv · show

Recent studies have shown that StyleGANs provide promising prior models for downstream tasks on image synthesis and editing. However, since the latent codes of StyleGANs are designed to control global styles, it is hard to achieve a fine-grained control over synthesized images. We present SemanticStyleGAN, where a generator is trained to model local semantic parts separately and synthesizes images in a compositional way. The structure and texture of different local parts are controlled by corresponding latent codes. Experimental results demonstrate that our model provides a strong disentanglement between different spatial areas. When combined with editing methods designed for StyleGANs, it can achieve a more fine-grained control to edit synthesized or real images. The model can also be extended to other domains via transfer learning. Thus, as a generic prior model with built-in disentanglement, it could facilitate the development of GAN-based applications and enable more potential downstream tasks.

1. Introduction

StyleGAN-based priors support synthesis and editing, but their latent codes are ambiguous and can entangle attributes, while other controllable GANs often lack fine-grained local editability. SemanticStyleGAN factorizes codes by semantic part to control local structure and texture and serve as a generic prior.

1. Introduction

  • StyleGAN latent codes have ambiguous meanings and correlated attributes can cause edits to change unintended attributes or local parts.Prior methods also target holistic attributes rather than fine-grained local edits.Source: §1 · pp. 1–2
  • SemanticStyleGAN assigns local latent codes to semantic parts, controlling each part’s structure and texture while composing the image from local features.It is not designed for a specific task and can be combined with StyleGAN latent manipulation methods for finer local control.Source: §1 · p. 2
  • The paper reports experiments combining its generator with latent manipulation methods and adapting it to other domains with limited images while preserving spatial disentanglement.The listed contributions also include joint modeling of images and semantic segmentation masks.Source: §1 · p. 2
  • SemanticStyleGAN factorizes its latent space over image regions, unlike StyleGAN’s codes associated with different granularities.The regional codes control local shape and texture.Source: Figure 2 · p. 2, §1 · p. 2

2. Related Work

Prior work explores latent manipulation, compositional generation, and layout-conditioned local editing. SemanticStyleGAN instead uses human-defined semantic parts to factorize local controls for structure and texture.

2.1. GAN Latent Space for Image Editing

  • Prior latent-editing methods manipulate pretrained GAN spaces or train more disentangled GANs with additional supervision, while SemanticStyleGAN factorizes codes by semantic labels.The proposed codes independently control each semantic part’s shape and texture.Source: §2.1 · p. 2

2.2. Compositional Image Synthesis

  • Unlike object-level compositional methods, SemanticStyleGAN uses human-defined parts to separate correlated regions below the object level, such as hair and face.This decomposition enables finer control during synthesis.Source: §2.2 · pp. 2–3

2.3. Layout-based Generators for Local Editing

  • Layout-conditioned generators with latent codes generally control local texture, whereas SemanticStyleGAN associates local areas directly with codes for both structure and texture.Some semantic-mask methods are conditional-generation systems and cannot generate images from scratch.Source: §2.3 · p. 3

3. Methodology

The method factorizes StyleGAN’s W+ space by semantic area. A shared code controls coarse structure, while local codes control each area’s shape and texture, and the generator maps these codes to an image and segmentation mask.

3. Methodology

  • StyleGAN maps z through an MLP to W, then extends W to W+ for styles at different resolutions, but these codes lack strictly defined individual meanings.SemanticStyleGAN proposes to factorize W+ by semantic area.Source: §3 · p. 3
  • The generator factorizes W+ into semantic-area codes, with a shared base code for coarse structure and local codes for each area’s shape and texture.Each local code is further decomposed into shape and texture codes.Source: §3 · p. 3
  • The generator maps the factorized codes to both an RGB image and a semantic segmentation mask.The input is a labeled dataset of images and their semantic masks.Source: §3 · p. 3

1. How to decouple different local areas?

SemanticStyleGAN decouples local semantic areas with local generators and compositional fusion. A dual-branch discriminator jointly models images and segmentation masks to supervise local-part shapes after composition.

1. How to decouple different local areas?

  • The training framework uses compositional synthesis and joint image-mask modeling to supervise local-part shapes after composition.These are the proposed inductive bias for decoupling local areas and the discriminator’s role in ensuring semantic meaning.Source: §1 · p. 3, §3.2 · p. 4

3.1. Generator

  • Local generators use position encodings and latent codes to produce part feature maps and pseudo-depth maps, with local codes controlling part shape and texture.The pseudo-depth maps are composition scores that mimic z-buffering, not strictly depth maps.Source: §3.1 · p. 4
  • Pseudo-depth maps serve as softmax logits for a coarse segmentation mask, whose weights aggregate local feature maps before rendering.For transparent classes such as glasses, a modified mask is used for feature aggregation.Source: §3.1 · p. 4
  • The render net receives fused features and outputs an RGB image and segmentation mask, combining 16×16 and 64×64 inputs and refining the coarse mask with residuals.The low-resolution input supports blending between parts; the refined mask is regularized not to deviate too much from the coarse mask.Source: §3.1 · p. 4

3.2. Discriminator and Learning Framework

  • A dual-branch discriminator processes RGB images and masks separately before summing outputs, enabling separate gradient-norm regularization of the segmentation branch.The branches jointly model the image-mask distribution; simple concatenation was found not to work because segmentation masks had large gradient magnitude.Source: §3.2 · p. 4

4. Implementation Details

The implementation uses StyleGAN2-based training settings, with adjustments for image-only fine-tuning. The accompanying examples and evaluation cover compositional synthesis and CelebAMask-HQ outputs.

4. Implementation Details

  • Training uses StyleGAN2 optimizer and batch settings, with R1 weights of 10, 1000, and 100 and style-mixing probability and path regularization reduced to 0.3 and 0.5.For image-only fine-tuning, the segmentation discriminator branch is dropped and the original StyleGAN2 losses are used.Source: §4 · p. 4
  • The compositional-synthesis illustration adds components from the background and shows pseudo-depth maps used for fusion; hair remains a complete shape beneath the face, with maps fused without order.Source: Figure 5 · p. 5
  • The synthesis-quality evaluation compares models trained on CelebAMask-HQ at 256×256; “img” denotes RGB image and “seg” denotes segmentation mask.Source: Table 1 · p. 5
  • Example generations use the CelebAMask-HQ-trained model at 512×512 resolution with truncation 0.7.Source: Figure 6 · p. 5

5. Experiments

The experiments examine synthesis quality and spatial disentanglement, then test whether the model supports controlled editing and transfer to other domains. Results show local control alongside comparable synthesis and reconstruction quality, with limits on transfer and attribute editing.

5.1. Semantic-aware and Disentangled Generation

  • The model achieves a similar synthesis quality to SemanticGAN while adding control over each semantic area, and its 512×512 version reaches FID 7.22 and IS 3.47.For reference, StyleGAN2 reaches FID 6.47 and IS 3.55.Source: §5.1 · pp. 5–6
  • Latent interpolation shows smooth transitions globally and within selected semantic areas, allowing one part to change while irrelevant parts remain fixed.Local generators can independently generate semantic parts, and pseudo-depth maps learn meaningful shapes despite lacking 3D supervision.Source: Figure 7 · p. 5, §5.1 · p. 6

5.2.1 Encoding and Editing Real Images

  • Restyle-based reconstruction is comparable, but evaluation uses four attributes on 1,000 held-out CelebAMask-HQ images and requires manually selecting relevant areas for the model.The compared reconstruction table reports Restyle encoder results, with the bottom two rows trained on the same CelebAMask-HQ split.Source: §5.2.1 · p. 6, Table 2 · p. 6
  • Our model achieves better overall performance than StyleGAN2 in local attribute editing, while StyleFlow fails to learn baldness for either generator because bald examples are scarce.The model stops increasing the baldness score after removing all hair; bangs editing can increase overall hair length, reflecting a training-data bias.Source: §5.2.1 · pp. 6–7, Figure 10 · p. 7

5.2.2 Text-guided Synthesis

  • When paired with StyleCLIP, the model can constrain text-prompt edits to selected local parts, unlike edits that often modify the whole StyleGAN2 image.Figure 11 compares sequential text-guided editing from the same average fake face.Source: §5.2.2 · p. 7, Figure 11 · p. 7

5.3. Results on Other Domains

  • Fine-tuning on smaller image-only datasets preserves local controllability when the target domain gap is limited; DeepFashion training also enables control of different semantic parts’ structure and texture.The Toonify, MetFaces, and BitMoji adaptations used hundreds of training steps; no segmentation masks were available for those datasets.Source: §5.3 · p. 8, Figure 12 · p. 8, Figure 13 · p. 8

6. Limitations and Discussion

The method’s scope is constrained by its semantic-class design and supervision, and its pose, shape, and texture factors are not always fully separated.

Applicable Datasets

  • A local generator is needed for each semantic class, limiting scalability to datasets with many classes, such as scenes; training from scratch also requires image-segmentation pairs.The authors identify a semi-supervised version as future work.Source: §6 · p. 8

Disentanglement

  • Pose, shape, and texture can remain ambiguous because their separation relies only on local-generator layer design; shared coarse codes may encode expression, and shape codes may affect beards.The stated focus is spatial disentanglement between semantic parts, with additional regularization or architecture tuning proposed for better decoupling.Source: §6 · p. 8

Societal Impact

  • The authors note potential controversial use in fake-profile generation and argue that synthesized images should be presented as synthetic.The work is focused on GAN controllability and is not specifically designed for malicious uses.Source: §6 · p. 8

7. Conclusion

The paper presents a GAN that composes local semantic areas to control their structure and texture. Experiments report high-quality synthesis, local-part disentanglement, finer editing control, and adaptation to image-only datasets.

7. Conclusion

  • Local generators, masked feature aggregation, and joint modeling of images and segmentation masks let the method model semantic areas’ structure and texture separately.The paper reports fine-grained editing when the model is combined with other editing methods, and adaptation to image-only datasets while preserving disentanglement.Source: §7 · p. 8

A. Implementation Details

Transparent semantic classes are added as residual features rather than replacing the underlying class features, preserving visible content behind them. This fusion leaves feature aggregation unchanged when no transparent classes are present, but assumes transparent classes do not overlap.

A.1. Fusion with Transparent Classes

  • Ordinary mask-weighted aggregation selects the frontal class, which can hide skin behind a transparent class such as glasses.The modified mask is used for feature aggregation in these cases.Source: §A.1 · p. 11
  • For transparent classes, Eq. (7) first aggregates non-transparent features, then adds transparent features with their original mask weights as residuals.This leaves the feature map unchanged when no transparent classes are present; the formulation assumes transparent classes do not overlap.Source: §A.1 · p. 11
  • The experiments designate glasses and earrings as transparent; using the original mask is stable but can distort texture behind transparent classes.The source describes glasses and earrings as transparent classes.Source: §A.1 · p. 11

A.2. Architecture Details

The appendix specifies local-generator and rendering details, reports training and inference costs, and discusses the limits of alternative local-editing approaches.

A.2. Architecture Details

  • Each local generator is a 10-layer modulated MLP that maps Fourier features and latent codes to feature and pseudo-depth maps, with gradients stopped between shape and texture layers except for background.The background generator's pseudo-depth map is fixed to zero.Source: §A.2 · pp. 11–12
  • The render network computes mask loss from the difference between downsampled output segmentation and coarse mask m, rather than explicitly computing Δm, with upsampling and residual operations at every mask layer.The discriminator has a symmetric segmentation branch that is removed during fine-tuning.Source: §A.2 · p. 12

A.3. Efficiency

  • Inference takes 0.137s per image on a single GPU without parallelizing local generators.The 512×512 model trains for about two and a half days for 150,000 steps on eight 32GB Nvidia Tesla V100 GPUs, with batch size 32.Source: §A.3 · p. 12

B. Additional Discussion

  • InterFaceGAN applied to StyleGAN2's S space still shows spatial entanglement, while StyleSpace editing tunes individual channels and controls only limited attributes.The appendix notes it is not yet proved that editing the whole S space achieves the same local disentanglement.Source: B. Additional Discussion · p. 12, Figure 16 · p. 12
  • SEAN-like mask-conditioned models and StyleMapGAN edit texture rather than shape; SemanticStyleGAN reports a CelebA-HQ FID of 6.43 versus 17.66 for [77] and 22.43 for [50].The cited mask-conditioned approaches require a fixed mask or editing-area input, whereas SemanticStyleGAN is described as automatic and not restricted to fixed pixels.Source: B. Additional Discussion · p. 12, Figure 17 · p. 12

C. Style Mixing and Additional Results

Local style mixing transfers semantic components between images, while additional examples show editing results across domains and on real faces. Some local information remains entangled with coarse structure or other components.

C. Style Mixing and Additional Results

Related papers

Loading 2112.02236v3…