Source-linked AI summary
STGAN: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing
Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, Shilei Wen
TL;DR
Arbitrary attribute editing must manipulate desired attributes without sacrificing image quality or source-image fidelity. STGAN addresses this by conditioning on changed attributes and selectively transferring encoder features, with experiments reporting simultaneous improvements in manipulation and perception quality. The paper reports favorable performance in facial attribute editing and season translation.
Problem
Encoder-decoder bottlenecks produce blurry edits, skip connections trade improved image quality for weaker attribute manipulation, and full target vectors may affect unchanged attributes.
Method
STGAN uses the difference between target and source attribute vectors and selective transfer units to adaptively select and modify encoder features.
Results
STGAN simultaneously improves attribute manipulation ability and image quality, performing favorably against state-of-the-art methods in facial attribute editing and season translation.
Takeaways & Limitations
Selective transfer provides a unified approach for editing local and global attributes while preserving more source-image information.
Takeaways & Limitations
Downsampling irreversibly diminishes feature-map resolution and fine details, which transposed convolutions cannot completely recover.
Abstract
from arXiv · showhide
Arbitrary attribute editing generally can be tackled by incorporating encoder-decoder and generative adversarial networks. However, the bottleneck layer in encoder-decoder usually gives rise to blurry and low quality editing result. And adding skip connections improves image quality at the cost of weakened attribute manipulation ability. Moreover, existing methods exploit target attribute vector to guide the flexible translation to desired target domain. In this work, we suggest to address these issues from selective transfer perspective. Considering that specific editing task is certainly only related to the changed attributes instead of all target attributes, our model selectively takes the difference between target and source attribute vectors as input. Furthermore, selective transfer units are incorporated with encoder-decoder to adaptively select and modify encoder feature for enhanced attribute editing. Experiments show that our method (i.e., STGAN) simultaneously improves attribute manipulation accuracy as well as perception quality, and performs favorably against state-of-the-arts in arbitrary facial attribute editing and season translation.
1. Introduction
Arbitrary attribute editing requires scalable unsupervised translation while preserving source-image details and manipulating only desired attributes. STGAN addresses these tensions with difference attribute vectors and selective transfer units, reporting favorable results in facial editing and season translation.
- Motivation: Arbitrary attribute editing is a multi-domain translation problem for which separate models waste training data and scale poorly as attributes increase.Existing arbitrary-editing methods therefore use unified encoder-decoder models conditioned on source images and target attribute vectors.
- Limitations of Existing Architectures: Bottleneck layers can produce blurry, low-quality edits, whereas skip connections improve image quality but weaken attribute manipulation.Skip connections directly pass encoder features to the decoder, creating a quality-versus-manipulation trade-off.
- Selective Attribute Conditioning: Full target attribute vectors may alter unchanged attributes, so editing should focus on the attributes that differ between source and target.The paper motivates replacing the full target vector with the difference between target and source attribute vectors.
- Proposed Approach: STGAN combines difference attribute vectors with selective transfer units that adaptively select and modify encoder features for task-specific editing.The units selectively concatenate transformed encoder features with decoder features and are intended to support both local and global attributes.
- Reported Outcome: Experiments report favorable performance against state-of-the-art methods in arbitrary facial attribute editing and season translation.The reported gains concern both attribute manipulation ability and image quality.
2. Related Work
Related work spans encoder-decoder and GAN foundations, image-to-image translation, and facial attribute editing. The paper positions arbitrary attribute editing as a scalable multi-domain translation problem that existing approaches do not fully resolve.
- Foundations: Encoder-decoder networks map inputs to latent codes and decode them into recovered or generated images, while GANs use adversarially trained generators and discriminators.These architectures support representation learning, image generation, translation, restoration, and editing.
- Image-to-Image Translation: Image-to-image translation learns mappings between domains, but arbitrary attribute editing requires scalable handling of multiple domains and attribute combinations.Prior approaches use paired or unpaired constraints, shared latent spaces, or domain-specific components, yet remain limited when multiple attributes change.
- Facial Attribute Editing: Facial attribute editing methods include single-attribute models and unified arbitrary-editing models such as IcGAN, FaderNet, ModularGAN, StarGAN, and AttGAN.Their limitations include restricted reconstruction, increasing computation with changed attributes, and difficulty scaling arbitrary edits.
3. Proposed Method
STGAN addresses encoder-decoder blurring, skip-connection trade-offs, and redundant target-attribute conditioning through selective transfer. It uses attribute differences and selective transfer units to adapt encoder features for arbitrary editing.
- 3.1. Limitation of Skip Connections in AttGAN: Skip connections improve reconstruction quality but weaken attribute manipulation, especially when multiple connections directly concatenate encoder and decoder features.This motivates selectively transforming encoder features rather than transferring them unchanged.
- 3.2. Taking Difference Attribute Vector as Input: STGAN uses the difference between target and source attribute vectors instead of the full target vector to focus editing on changed attributes.The difference vector indicates whether each attribute changes and the direction of change.
- 3.3. Selective Transfer Units: Selective transfer units adaptively transform encoder features using encoder features, hidden states, and the difference attribute vector before combining them with decoder features.STUs modify GRU-style information passing with reset and update gates, convolutional transformation, and interpolation.
- 3.4. Network Architecture: STUs are applied after the first four encoder layers, while the fifth encoder feature is passed directly to the decoder.Different STUs do not share parameters because layer dimensions and feature abstraction levels differ.
- 3.4. Network Architecture: The generator combines a five-layer convolutional encoder, a five-layer transposed-convolution decoder, and STUs, while the discriminator has adversarial and attribute-prediction branches.The adversarial branch distinguishes real from fake images, and the attribute branch predicts attribute vectors.
- 3.5. Loss Functions: STGAN trains with reconstruction, adversarial, and attribute-manipulation losses to preserve unchanged images and enforce realistic desired attributes.Reconstruction uses an ℓ1 norm when the difference attribute vector is zero; adversarial loss handles unavailable editing ground truth.
4. Experiments
Experiments evaluate STGAN on facial attribute editing and season translation using qualitative, quantitative, reconstruction, and user-study measures. Across these evaluations, STGAN produces accurate attribute changes and high-quality results, outperforming or favorably comparing with competing methods.
- Facial Attribute Editing: STGAN is evaluated for arbitrary facial attribute editing on CelebA using 13 selected facial attributes and 128 × 128 cropped images.The experiments use the CelebA training, validation, and test splits, with 1,000 validation images reserved for assessing training.
- Facial Attribute Editing: Compared with IcGAN, FaderNet, AttGAN, and StarGAN, STGAN more accurately manipulates complex or multiple attributes while producing higher-quality, more realistic images.The competing methods often produce oversmoothed, insufficiently modified, or non-realistic results for attributes such as Bald, Hair, and Age.
- Facial Attribute Editing: 20% accuracy gains are achieved by STGAN over competing methods for Bald, Black Hair, Brown Hair, and Eyebrows in attribute generation accuracy.Attribute generation accuracy is measured by a classifier trained on the STGAN training set, which reaches 94.5% accuracy on the 13 test attributes.
- Facial Attribute Editing: > 7 dB by PSNR is obtained by STGAN over AttGAN and StarGAN in reconstruction quality when target attributes equal source attributes.The comparison uses PSNR and SSIM, and the passage attributes the improvement to STUs and the difference attribute vector.
- Facial Attribute Editing: STGAN is selected as the best method with higher probability on all 11 facial editing tasks in a user study.Participants judged attribute success, image quality, identity preservation, and preservation of source-image details.
- Season Translation: In summer-to-winter and winter-to-summer translation, STGAN performs favorably against AttGAN, StarGAN, and CycleGAN, with over 50% probability of winning the user study.The comparison uses a single STGAN model for both directions, whereas CycleGAN uses two generators.
5. Ablation Study
The ablation study separates the effects of the difference attribute vector and selective transfer units. Difference vectors generally improve accuracy, stability, and reconstruction, while STUs outperform convolutional, residual, and GRU variants across most attributes.
- Difference Attribute Vector vs. Target Attribute Vector: Difference attribute vectors generally improve attribute generation accuracy for AttGAN, StarGAN, and STGAN compared with full target attribute vectors.The study also reports improved training stability and image reconstruction performance, although AttGAN-diff and StarGAN-diff retain poor image quality.
- Selective Transfer Unit vs. Its Variants: STGAN-conv and STGAN-conv-res are significantly inferior to STGAN, indicating limitations in their selective transfer of encoder features.Residual learning improves STGAN-conv relative to STGAN-conv-res, but neither convolutional variant matches STGAN.
- Selective Transfer Unit vs. Its Variants: Residual learning adds no gains for most attributes and performs worse for several global and fine attributes, including Gender, Age, Mustache, and Beard.This comparison concerns STGAN and STGAN-res.
- Selective Transfer Unit vs. Its Variants: STGAN is slightly superior to STGAN-gru for most attributes, with notable gains for Gender and Mustache, although STGAN-gru performs better on Bald.STGAN-gru replaces the STU with a GRU using transformed features as the hidden state.
6. Conclusion
The paper presents STGAN as a selective-transfer encoder-decoder that uses changed attributes and adaptive feature modification. Experiments on facial editing and season translation report favorable performance in attribute accuracy and image quality.
- Conclusion: STGAN combines difference attribute vectors with selective transfer units in an encoder-decoder for arbitrary image attribute editing.The difference vector focuses the model on changed attributes, while STUs adaptively select and modify encoder features for each editing task.
- Conclusion: Experiments on arbitrary facial attribute editing and season translation show favorable performance in attribute generation accuracy and editing image quality.The conclusion reports simultaneous improvement of attribute manipulation ability and image quality.