Source-linked AI summary
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows
Rameen Abdal, Peihao Zhu, Niloy Mitra, Peter Wonka
TL;DR
StyleFlow addresses limited attribute control and entanglement in photorealistic unconditional GANs. It uses conditional continuous normalizing flows in StyleGAN’s latent space for attribute-conditioned sampling and controlled editing, with reported fine-grained disentangled edits and superior qualitative and quantitative results over competing methods.
Problem
Unconditional GANs provide photorealistic images but offer limited attribute control, and entangled latent spaces can cause unwanted changes during edits.
Method
StyleFlow formulates conditional exploration as conditional continuous normalizing flows in a GAN latent space conditioned on attribute features.
Results
StyleFlow produces high-quality attribute-conditioned samples and fine-grained disentangled edits, with reported superiority over competing methods.
Takeaways & Limitations
The framework supports controlled exploration of unconditional StyleGAN latent spaces across multiple attributes on generated and real images.
Takeaways & Limitations
The method relies on available attributes, performs best with high-quality training datasets, and can produce artifacts when editing real images.
Abstract
from arXiv · showhide
High-quality, diverse, and photorealistic images can now be generated by unconditional GANs (e.g., StyleGAN). However, limited options exist to control the generation process using (semantic) attributes, while still preserving the quality of the output. Further, due to the entangled nature of the GAN latent space, performing edits along one attribute can easily result in unwanted changes along other attributes. In this paper, in the context of conditional exploration of entangled latent spaces, we investigate the two sub-problems of attribute-conditioned sampling and attribute-controlled editing. We present StyleFlow as a simple, effective, and robust solution to both the sub-problems by formulating conditional exploration as an instance of conditional continuous normalizing flows in the GAN latent space conditioned by attribute features. We evaluate our method using the face and the car latent space of StyleGAN, and demonstrate fine-grained disentangled edits along various attributes on both real photographs and StyleGAN generated images. For example, for faces, we vary camera pose, illumination variation, expression, facial hair, gender, and age. Finally, via extensive qualitative and quantitative comparisons, we demonstrate the superiority of StyleFlow to other concurrent works.
1 INTRODUCTION
StyleFlow addresses limited attribute control in photorealistic unconditional GANs by exploring StyleGAN’s entangled latent space conditionally. It supports attribute-conditioned sampling and attribute-controlled editing through conditional normalizing flows, with reported high-quality, identity-preserving results.
- StyleGAN produces highly photorealistic images but offers limited user control over adjustable attributes such as pose, illumination, and expression.
- Conditional GANs provide semantic attribute control but may produce blurrier images than uncontrolled StyleGAN and can change unspecified attributes, losing identity.
- StyleFlow treats attribute editing as conditional exploration in an unsupervised GAN rather than attribute-based retraining.
- The framework addresses attribute-conditioned sampling and attribute-controlled editing by finding nonlinear paths in StyleGAN’s latent space.
- StyleFlow conditions inferred paths on the input image and supports attributes including camera, illumination, expression, gender, and age for faces, plus camera, type, and color for cars.
- The method reports high-quality identity-preserving edits, sequential editing without forcing latent vectors outside the distribution, and superior identity preservation against concurrent techniques.
2 RELATED WORK
Related work spans conditional GANs, latent-code manipulation, image embedding, and neural rendering. StyleFlow belongs to latent-code manipulation while targeting attribute control and disentangled editing in pretrained GAN spaces.
- Comparison: Figure 2 contrasts latent vector arithmetic, which changes facial identity during expression edits, with StyleFlow’s nonlinear paths that retain identity.
- Conditional GANs: Conditional GANs add semantic conditioning for image manipulation, including face attributes, sketches, colors, masks, makeup, and hair editing.
- Latent-code manipulation: Latent-code methods edit pretrained GANs by computing semantic difference vectors and adding them to other latent codes.
- Latent-code manipulation: StyleFlow is positioned among pretrained-GAN latent manipulation methods rather than methods that design a separate conditional architecture.
- Image embedding: GAN image embedding uses encoders, optimization, or combinations of both to infer latent codes corresponding to input images.
- Neural rendering: Neural-rendering methods address tasks such as novel-view synthesis, relighting, and animation, but their overall problem setting is sufficiently different for direct comparison to be difficult.
3 OVERVIEW
StyleFlow formulates attribute-conditioned exploration as mapping a normal latent prior into StyleGAN’s latent distribution conditioned on target attributes. The same framework supports diverse conditional sampling and image editing while preserving source identity.
- StyleFlow supports high-quality attribute-conditioned sampling and attribute-controlled editing, with editing designed to preserve the source image’s identity.
- StyleGAN maps 512-dimensional latent samples into intermediate W-space vectors that control normalization at 18 generator locations, while W+ uses separate vectors at those locations.
- An attribute function A, typically a classifier, maps generated images to an attribute vector used as conditioning information.
- Attribute-conditioned sampling: For conditional sampling, StyleFlow draws z from a multidimensional normal distribution, maps it through Φ(z, a), and decodes the resulting weights into images matching target attributes.
- Conditional continuous normalizing flows provide the formulation for mapping prior samples to latent distributions conditioned on target attributes.
- Attribute-controlled editing: For editing, an input image is projected to StyleGAN space, its current attributes are inferred, and latent variables are recovered before targeting new attributes.
4 NORMALIZING FLOWS
Normalizing flows provide invertible mappings between a simple prior and the StyleGAN latent distribution, while continuous flows use neural ODEs to support reversible, attribute-conditioned exploration.
- Discrete Normalizing Flows: Normalizing flows map an unknown distribution to a known prior distribution through invertible transformations and their inverse mappings.
- Discrete Normalizing Flows: A sequence of transformations maps z0 through intermediate variables to w, with inverse inference recovering the prior variable from the StyleGAN latent vector.
- Continuous Normalizing Flows (CNF): Attribute-conditioned editing performs reverse inference from a source image and forward inference through CNF blocks, using an ODE solver over time.
- Discrete Normalizing Flows: Planar flows parameterize each transformation with a neural network whose learnable vectors and bias define the nonlinear update.
- Continuous Normalizing Flows (CNF): Continuous normalizing flows express latent evolution as a neural ODE, dz/dt = ϕ(z(t),t;θ), with θ representing neural-network parameters.
- Continuous Normalizing Flows (CNF): StyleFlow uses CNFs because the continuous formulation is invertible by definition and replaces costly determinant computation with a continuous alternative.
5 METHOD
StyleFlow learns conditional mappings from StyleGAN’s W-space latent vectors to attribute-conditioned distributions, using jointly conditioned CNF networks for sampling and editing. The method supports multiple attributes and aims to reduce unwanted entanglement during edits.
- Method: StyleFlow models a conditional mapping between StyleGAN W-space latent vectors and a prior distribution so semantic editing applications are realizable.
- Method: Training data pairs 10k sampled StyleGAN latent codes with generated images and classifier-derived attributes for flow-network learning.
- Method: The conditional flow uses gate-bias modulation networks whose scaling and translation components support adaptive, identity-aware edit directions.
- Method: At inference, linear interpolation in the attribute domain smoothly translates between edits, using stacked CNF and moving-batch-normalization functions.
- Method: Training attributes jointly avoids the entangled vector fields learned when each attribute is modeled separately.
- Method: The method’s sampling examples specify combinations such as female faces with glasses, 50-year-old males with facial hair, and smiling five-year-old children.
- Method: The training setup maximizes conditional data likelihood with a standard Gaussian prior and uses Adam optimization with an initial learning rate of 1 × 10^-3.
6 ATTRIBUTE-CONDITIONED SAMPLING AND EDITING
StyleFlow supports attribute-conditioned sampling and semantic editing by learning mappings between attributes and StyleGAN latent variables. Its flow-based formulation enables adaptive, sequential edits while keeping manipulated latents within the W-space distribution.
- Conditional sampling: StyleFlow maps between latent variables and attributes, enabling semantic manipulation in the respective domains.
- Conditional sampling: Conditional sampling fixes desired attributes, transforms Gaussian samples through a trained conditional CNF, and feeds the resulting w vectors to StyleGAN.
- Semantic editing: Semantic editing obtains adaptive vector manipulations by solving a learned vector field with an ODE solver.
- Semantic editing: Joint Reverse Encoding infers attributes and a corresponding z0 from a StyleGAN latent or a projected real image.
- Semantic editing: Conditional Forward Editing fixes z0, changes target attributes, and uses forward flow inference to obtain an editing vector.
- Semantic editing: Edit-specific subset selection applies edits to selected W+ layers, while reprojection keeps subsequent sequential edits stable and identity-aware.
7 RESULTS
The evaluation uses FFHQ and LSUN-Car with realism, identity-preservation, and edit-consistency measures, alongside qualitative comparisons with several latent-editing methods. The section establishes the experimental basis for assessing StyleFlow across generated and real-image settings.
- Datasets and metrics: Experiments use StyleGAN pretrained on FFHQ faces and LSUN-Car images, covering diverse identities, poses, colors, and object types.FFHQ contains 70,000 1024 × 1024 face images; LSUN-Car contains 16,185 512 × 384 car images.
- Datasets and metrics: Evaluation measures realism with FID, identity preservation with face-recognition embeddings, and edit consistency across edit permutations.
- Comparison methods: The comparisons include Image2StyleGAN, InterfaceGAN, GANSpace, and StyleRig, with several methods retrained or configured for comparable edits.
- Comparison methods: Continuous metrics are used because competing methods can produce different edit magnitudes when the same vector is applied to different faces.
7.4 Qalitative comparison of edits
StyleFlow produces high-quality sequential and non-sequential edits on generated and projected real faces while preserving unedited attributes. Quantitative and qualitative analyses associate this behavior with source-conditioned, nonlinear latent trajectories that remain within the latent distribution.
- Qualitative edits: StyleFlow handles diverse real-image cases, including extreme pose, asymmetrical expressions, and varied ages.
- Qualitative edits: Sequential edits cover pose, lighting, expression, gender, age, facial hair, eyeglasses, and baldness, with edit order not affecting reported quality.
- Latent edit paths: Attribute-guided edits follow source-conditioned nonlinear curves rather than relying only on fixed offset vectors that may leave the latent distribution.
- Latent edit paths: 1.5: linear interpolation in w differs from attribute-domain interpolation by a factor of 1.5 on average.Under extreme edits, the nonlinear path better retains hairstyle, clothes, and head coverings along the path.
- Quantitative comparison: StyleFlow has relatively low FID, outperforms other methods across identity-preservation metrics except that GANSpace also performs well for expression edits, and remains consistent under edit permutations.
7.6 Choice of encoding and subset selection
Ablations examine joint attribute encoding and edit-specific subset selection as design choices for preserving identity and unedited attributes. The reported comparisons show that the full V2 framework improves visual edit quality relative to variants.
- Ablation studies: The ablation evaluates joint attribute encoding and edit-specific subset selection as separate design choices.
- Ablation studies: Joint encoding better preserves face identity and unedited attributes such as hairstyle, age, and background than separate encoding.
- Ablation studies: The V2 framework produces higher-quality edits with comparable skin tone, background, and clothes relative to the source image.
- Ablation studies: The study also includes an ablation over different numbers of CNF function blocks.
7.7 User Study
The user study compares StyleFlow with four methods on visual quality and identity preservation across common and sequential edits. The study reports that StyleFlow outperforms the alternatives on both criteria, with real-dataset identity results favoring StyleFlow except for expression edits.
- Study design: The pairwise user study evaluates visual quality and identity preservation for expression, pose, lighting, and sequential lighting-plus-pose edits.The comparisons include InterfaceGAN, GANSpace, StyleRig, and StyleFlow.
- User-study findings: StyleFlow outperforms the other methods in both visual quality and identity preservation in the aggregated user-study results.
- Identity preservation: StyleFlow achieves better identity-preservation results than competing methods across all evaluated edits except expression edits.This pattern is reported for the real-image dataset, while sequential edits also favor StyleFlow quantitatively.
- Conditional sampling: Conditional sampling produces high-quality and diverse images while keeping selected attributes fixed, including examples varying age and glasses with pose fixed.The sampling procedure resamples z to infer w while retaining a set of attributes.
- Interactive editing: The interface supports slider-based edits on real or generated images and stores checkpoints so users can revisit sequential changes.
8 DISCUSSION
The discussion attributes StyleFlow’s editing behavior to design choices including edit-specific subset selection, latent-dependent directions, continuous controls, and joint attribute training. It emphasizes sequential edits as a demanding evaluation setting while noting dependencies on data quality and projection for real images.
- 8.1 Edit specific subset selection: Edit-specific subset selection is shared by GANSpace and StyleFlow and is associated with better disentanglement than methods lacking this design choice.
- 8.2 Conditioning editing direction on the starting image: StyleFlow computes an editing direction that depends on the starting latent, unlike Image2StyleGAN, InterfaceGAN, and GANSpace, which use one direction per attribute.The authors report that this design choice contributes to better disentanglement.
- 8.4 Continuous control: StyleFlow provides continuous attribute parameters, whereas InterfaceGAN and GANSpace require manual scaling of a fixed edit direction.
- 8.5 Joint attribute training: Jointly training StyleFlow on all attributes improves disentanglement and edit quality compared with training a separate network for each attribute.
- 8.6 Sequential edits: Sequential edits provide a more challenging evaluation because small disentanglement errors accumulate and can erode the initial face’s identity.
- 8.8 What type of edits are possible?: Edit quality depends on good attribute labels and a high-quality StyleGAN training dataset, while real-image editing additionally depends on projection into W+ space.The compound effect of embedding and editing is outside the paper’s scope.
9 CONCLUSION
The paper presents StyleFlow as a conditional continuous normalizing-flow framework for attribute-conditioned sampling and editing in StyleGAN latent spaces. It reports high-quality sampling, fine-grained edits, and qualitative and quantitative superiority, while identifying data, dataset, and real-image artifact limitations.
- 9 CONCLUSION: StyleFlow addresses attribute-conditioned sampling and attribute-controlled editing in StyleGAN using conditional continuous normalizing flows.
- 9 CONCLUSION: The framework samples high-quality images given attribute sets and performs fine-grained edits across attributes including pose, illumination, expression, skin tone, gender, and age.
- 9 CONCLUSION: Qualitative and quantitative evaluations report superior StyleFlow results over competing methods, including real-face editing quality.
- 9 CONCLUSION: The work is limited by attribute availability, dependence on high-quality StyleGAN training data, and artifacts that can occur when editing real images.The authors identify improved projection algorithms and broader high-quality datasets as areas needing further work.
10 APPENDIX
Appendix comparisons examine StyleFlow under extreme sequential edits, compatibility with StyleGAN1, and differences from GANSpace. StyleFlow is reported to preserve facial features and gender more reliably in the shown comparisons, while GANSpace benefits from unsupervised discovery but cannot control which attributes are found.
- 10.1 Qalitative comparisons on generated images: Under extreme sequential pose, expression, and lighting edits, StyleFlow produces high-quality outputs while preserving facial features better than Image2StyleGAN and InterfaceGAN.Image2StyleGAN can drive images out of distribution, while InterfaceGAN retains a face-identity problem.
- 10.2 StyleGAN1 compatibility: StyleFlow performs well on selected edits in the more entangled StyleGAN1 latent space.
- 10.3 Comparison with GANSpace: In the shown GANSpace comparison, a transition changes gender from female to male and a lighting edit fails, whereas StyleFlow preserves gender in the top sequence.
- 10 APPENDIX: The appendix includes comparisons with Image2StyleGAN and InterfaceGAN, plus attribute-conditioned edits for gender, expression, facial hair, and pose.
- 10.3 Comparison with GANSpace: GANSpace discovers edit axes without annotations but cannot control which attributes or attribute combinations those axes represent.StyleFlow instead learns nonlinear mappings directly to targeted attribute variations.