Source-linked AI summary
Controllable Person Image Synthesis with Attribute-Decomposed GAN
Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, Zhouhui Lian
TL;DR
Existing person-image synthesis methods provide limited control over human attributes beyond pose and face challenges from insufficient annotation and appearance diversity. The Attribute-Decomposed GAN separates pose and component appearance into latent codes using independent pathways and parser-based decomposition, enabling editable attribute recombination. The paper reports more realistic outputs, improved pose-transfer performance, and effective component attribute transfer, while noting failures for attributes and poses far outside the training manifold.
Problem
Existing methods mainly focus on pose control, while insufficient human-attribute annotation and diverse appearances make controllable synthesis challenging.
Method
The Attribute-Decomposed GAN encodes target pose and parser-separated source components through independent pathways, recombining decomposed latent codes into editable style representations.
Results
The method achieves a significant quality boost for person image synthesis, outperforms state-of-the-art methods in pose transfer, and supports component attribute transfer.
Takeaways & Limitations
Decomposed latent codes enable flexible and continuous user control of pose and component attributes, including transfer and interpolation across source images.
Takeaways & Limitations
The method fails on poses and component attributes that extremely bias the manifold built from the training data.
Abstract
from arXiv · showhide
This paper introduces the Attribute-Decomposed GAN, a novel generative model for controllable person image synthesis, which can produce realistic person images with desired human attributes (e.g., pose, head, upper clothes and pants) provided in various source inputs. The core idea of the proposed model is to embed human attributes into the latent space as independent codes and thus achieve flexible and continuous control of attributes via mixing and interpolation operations in explicit style representations. Specifically, a new architecture consisting of two encoding pathways with style block connections is proposed to decompose the original hard mapping into multiple more accessible subtasks. In source pathway, we further extract component layouts with an off-the-shelf human parser and feed them into a shared global texture encoder for decomposed latent codes. This strategy allows for the synthesis of more realistic output images and automatic separation of un-annotated attributes. Experimental results demonstrate the proposed method's superiority over the state of the art in pose transfer and its effectiveness in the brand-new task of component attribute transfer.
1. Introduction
Person image synthesis is extended from pose-guided generation to controllable synthesis of pose and component attributes supplied by multiple source images. The Attribute-Decomposed GAN addresses limited annotation and appearance–pose complexity through independent latent codes, parser-based decomposition, and editable style representations.
- Person image synthesis supports applications including image editing, movie making, person re-identification, and virtual clothes try-on.
- The proposed task synthesizes person images with controllable pose, head, upper-clothes, and pants attributes from multiple source person images.
- Existing pose-transfer methods mainly control pose, while appearance-transfer methods require complex processing and struggle with spatially misaligned body parts.
- The Attribute-Decomposed GAN uses independent pose and decomposed-component encoding pathways instead of directly mapping concatenated conditions to target images.
- Embedding attributes as editable style and pose codes enables flexible, continuous component-specific control and improves person-image synthesis quality.
- An off-the-shelf human parser extracts component layouts, enabling automatic separation of component attributes without annotation in the training data.
2. Related Work
Prior person-image synthesis work largely uses adversarial or related architectures to generate people in arbitrary poses, but primarily transfers pose rather than independently controlling component attributes. The proposed architecture separates pose and source appearance pathways and reconnects them through style blocks.
- GAN-based image synthesis includes conditional, high-resolution, and unsupervised image-to-image translation methods.
- Pose-guided person-image methods use staged GANs, variational autoencoders, conditional U-Nets, deformable skip connections, and pose-attentional transfer blocks.
- The generator embeds target pose and source person inputs through independent pathways connected by style blocks that inject source texture into pose features.
- Earlier methods primarily transfer a target pose to a reference person, whereas this work adds controllable human attributes.
3. Method Description
The method separates pose and component appearance into independent latent pathways, then transfers texture through style blocks to synthesize person images under target poses. Component decomposition, global texture encoding, fusion, and multiple losses are combined to preserve appearance while improving pose alignment and image quality.
- Generator architecture: The generator embeds target pose and source appearance into separate pose and style codes instead of directly concatenating conditions.The pose pathway encodes the target pose, while decomposed component encoding produces the source style code.
- Decomposed component encoding: DCE extracts semantic component masks from the source image, encodes each masked component with a shared texture encoder, and concatenates the resulting codes into the full style code.The component codes occupy specified positions, enabling component-level recombination and control.
- Decomposed component encoding: DCE supports automatic, unsupervised attribute separation without component annotations and allows desired component codes to be mixed across source persons.The paper illustrates DCE effects through training loss curves and visual comparisons with and without DCE or global texture encoding.
- Texture encoding: Global texture encoding concatenates learnable encoder outputs with fixed VGG features to improve texture representation and generalization.The VGG encoder is pretrained on COCO and provides texture information from varied images.
- Texture style transfer: Style blocks receive prior deep features and the full style code, then use fusion-derived AdaIN parameters within residual convolutional blocks to transfer texture.The fusion module uses three fully connected layers to recombine component features and produce the required AdaIN parameters; eight style blocks are used.
- Training objectives: Adversarial, reconstruction, perceptual, and contextual losses train the model to match target appearance, preserve details, follow target pose, and maintain texture similarity.The two discriminators separately enforce pose alignment and appearance-texture similarity, while contextual loss handles non-aligned images.
4. Experimental Results
Experiments evaluate pose transfer and component attribute transfer, showing realistic synthesis under varied poses and controllable replacement of clothing attributes. The method also supports style interpolation, but fails for attributes that lie far outside the training-data manifold.
- Evaluation setup: The evaluation covers pose transfer and component attribute transfer, including comparisons with state-of-the-art methods and a user study.The experiments use DeepFashion and assess both qualitative and quantitative pose-transfer performance.
- Pose transfer: The model generates natural-looking images when target poses differ greatly from the source in scale and viewpoint.These results are reported for pose-controlled synthesis on test-set poses.
- Pose transfer: The user study reports that the generated images are more realistic, natural, and faithful than outputs from four pose-transfer methods.Participants evaluated realness and faithfulness, with over 70% of the authors’ results selected in the reported preference comparison.
- Component attribute transfer: Component transfer combines attributes from multiple source people, such as preserving one person’s appearance while changing upper clothes or pants.The method introduces new attributes harmoniously while preserving textures of remaining components.
- Component attribute transfer: Style interpolation travels between component codes to synthesize an animation from one attribute to another.The interpolation mixes upper-clothes codes using a coefficient β that decreases from 1 to 0.
- Failure cases: Failures occur when pose or component attributes extremely bias the manifold constructed from training data.Examples include a rare pose and a specific T-shirt pattern that cannot be seamlessly interpolated with seen examples.
5. Conclusion
The Attribute-Decomposed GAN enables flexible, continuous control of human attributes by decomposing source person images into component codes and recombining them. This produces more realistic outputs and supports broader adaptation to image synthesis tasks.
- Attribute-Decomposed GAN decomposes source person images into component codes and recombines them into a full style code.The codes represent decomposed human attributes for controllable synthesis.
- Flexible and continuous control of human attributes is achieved through the decomposed representation.
- The decomposition strategy enables more realistic output images and flexible user control of component attributes.
- An off-the-shelf human parser automatically separates component attributes without requiring extensive data annotation.The authors suggest this strategy may inspire research where human-attribute annotation is insufficient.
- The solution is potentially adaptable beyond person-image generation to other image synthesis tasks.