Source-linked AI summary
Image-to-image Translation via Hierarchical Style Disentanglement
Xinyang Li, Shengchuan Zhang, Jie Hu, Liujuan Cao, Xiaopeng Hong, Xudong Mao, Feiyue Huang, Yongjian Wu, Rongrong Ji
TL;DR
Existing image-to-image translation methods struggle with uncontrolled manipulations because label independence and exclusiveness are not adequately modeled. HiSD organizes labels into a hierarchy of independent tags, exclusive attributes, and disentangled styles, then adapts translation to use styles identified with those labels. Qualitative and quantitative experiments on CelebA-HQ verify the method's effectiveness and controllable diversity.
Problem
Existing multi-label and multi-style translation methods can produce uncontrolled manipulations because label independence and exclusiveness are unexplored.
Method
HiSD organizes labels into independent tags, exclusive attributes, and disentangled styles, using generated or extracted tag-relevant styles for translation.
Results
Qualitative and quantitative experiments on CelebA-HQ verify HiSD's effectiveness, while ablations report the full model's best qualitative performance.
Takeaways & Limitations
HiSD provides a hierarchical and controllable approach to multi-label and multi-style image-to-image translation, with reported scalability and controllable diversity.
Abstract
from arXiv · showhide
Recently, image-to-image translation has made significant progress in achieving both multi-label (\ie, translation conditioned on different labels) and multi-style (\ie, generation with diverse styles) tasks. However, due to the unexplored independence and exclusiveness in the labels, existing endeavors are defeated by involving uncontrolled manipulations to the translation results. In this paper, we propose Hierarchical Style Disentanglement (HiSD) to address this issue. Specifically, we organize the labels into a hierarchical tree structure, in which independent tags, exclusive attributes, and disentangled styles are allocated from top to bottom. Correspondingly, a new translation process is designed to adapt the above structure, in which the styles are identified for controllable translations. Both qualitative and quantitative results on the CelebA-HQ dataset verify the ability of the proposed HiSD. We hope our method will serve as a solid baseline and provide fresh insights with the hierarchically organized annotations for future research in image-to-image translation. The code has been released at https://github.com/imlixinyang/HiSD.
1. Introduction
HiSD addresses uncontrolled multi-label and multi-style translation by organizing labels into independent tags, exclusive attributes, and disentangled tag-relevant styles. Its redesigned translation process supports diverse, independently controllable manipulations through generated or extracted styles.
- Existing unified translation methods can introduce unnecessary changes such as facial-identity or background manipulation and cannot independently model styles for bangs, glasses, and hair color.
- The framework divides multi-label translation into multi-attribute translation and simultaneous, independent multi-tag manipulation.
- Tag-relevant styles capture detailed manifestations such as different glasses without requiring supervised annotations.
- HiSD organizes labels hierarchically, assigning independent tags, exclusive attributes, and disentangled styles from top to bottom.
- HiSD redesigns modules, phases, and objectives to identify styles with tags and attributes while avoiding global manipulations and implicit attribute changes.
- Experiments are conducted to evaluate the effectiveness of the proposed model.
2. Related Works
The related work situates HiSD within GANs and image-to-image translation, emphasizing its joint treatment of multi-label and multi-style problems and its label-specific style formulation.
- GANs generate outputs resembling real samples and are optimized for training stability and applications across different areas.
- HiSD uses multi-task GANs for target attributes and conditional GANs to preserve tag-irrelevant conditions during translation.
- The framework jointly addresses multi-label and multi-style image-to-image translation while overcoming disadvantages of previous unified approaches.
- Label-specific style captures a specific binary label's clear image manifestation but does not account for label exclusiveness.
3. Methods
HiSD organizes labels hierarchically and translates tag-relevant styles through feature-level operations, reconstruction and style consistency, and conditional discrimination. Its training and test procedures support multi-style, multi-attribute, and multi-tag translation while limiting changes to tag-irrelevant content.
- Hierarchical representation: HiSD represents each tag with exclusive attributes and tag-relevant styles, enabling styles to capture detailed visual manifestations beyond human-annotated labels.The framework denotes attributes and their associated styles separately, allowing unsupervised distinctions such as different kinds of glasses.
- Style-guided translation: The framework generates styles from latent codes or extracts them from reference images, then uses a translator to manipulate encoded features before generating translated images.Multiple tag translations apply the corresponding translators sequentially to the source feature.
- Training phases: Training combines non-translation, self-translation, and cycle-translation paths to reconstruct source images and connect generated styles with extracted styles.The cycle path renders a generated target style, then uses the original extracted style to reconstruct the source image.
- Training objectives: Reconstruction and style objectives enforce feature consistency, cycle consistency, and agreement between generated and extracted styles.The objectives encourage translators to preserve reconstructive information while making style codes accurately represent detailed tag manifestations.
- Architectural improvements: A feature-based local translator uses spatial- and channel-wise attention to avoid global changes such as background and illumination manipulation.The attention mask is applied to features rather than directly to images, with negligible additional calculation and no regularization objective.
- Architectural improvements: A tag-irrelevant conditional discriminator receives conditions such as Male and Young to discourage translations from altering imbalanced implicit attributes.The testing procedures cover multi-style, multi-attribute, and multi-tag tasks, with multi-tag translation achieved indirectly through the redesigned training phases and objectives.
4. Experiments
Experiments on CelebA-HQ evaluate HiSD across latent-guided and reference-guided multi-style translation, using qualitative comparisons and quantitative measures of realism, diversity, and disentanglement. HiSD preserves tag-irrelevant details while producing diverse, accurate tag-relevant styles and balanced performance across guidance modes.
- CelebA-HQ contains 30,000 facial images annotated with attributes including hair color, gender, and glasses.
- Latent-guided task: In latent-guided translation, HiSD produces accurate tag-relevant styles with high visual quality, diverse attributes, and preserved tag-irrelevant details.
- Reference-guided task: In reference-guided translation, HiSD transfers styles accurately, while StarGANv2 changes hair color, background, and facial identity and ELEGANT produces visible artifacts.
- Quantitative evaluation: HiSD outperforms all baselines in realism measured by FID.FID is computed between translated images and real images using five style codes per test image.
- Quantitative evaluation: Users prefer HiSD's diversity over each baseline because no baseline exceeds the 50% preference threshold.
- Quantitative evaluation: HiSD achieves the most balanced performance between latent-guided and reference-guided tasks according to the capacity gap.The capacity gap is the L1-norm between the two task results.
4.3. Ablation Study
The ablation study shows that each component contributes to HiSD's qualitative and quantitative behavior, particularly for focusing on tag-relevant styles and producing realistic reference-guided translations.
- Removing the feature-based local translator prevents the model from focusing on tag-relevant changes such as image color and illumination.
- Removing the adversarial objective makes reference-guided outputs fuzzy and yields the worst realism and disentanglement metrics among ablations.
- The full model achieves the best qualitative results while sacrificing a small amount of quantitative realism for disentanglement.
5. Conclusion
The conclusion presents HiSD as a hierarchical approach for scalable, controllable image-to-image translation. The authors report effectiveness and identify adaptations to several learning settings.
- HiSD organizes independent tags, exclusive attributes, and unsupervised disentangled styles in a hierarchical tree structure.
- Extensive qualitative and quantitative experiments support the effectiveness of HiSD.
- The method is described as adaptable to joint training, continuous learning, semi-supervised learning, and few-shot learning.
G. Module Architecture
HiSD uses indexed mapper, extractor, encoder, and generator modules to generate and process hierarchical style representations.
- The mapper is an MLP indexed by tag before its first layer and by attribute at its middle layer.
- The extractor uses five downsampling blocks built from pre-activation residual units and indexes the tag before its last layer.
- The encoder has two downsampling blocks, while the generator has two upsampling blocks.
H. Implementation Details
HiSD is trained on 128×128 CelebA-HQ images for 200K iterations, using specified adversarial regularization, optimization settings, and learning rates.
- H. Implementation Details: The model uses batch size 8 and trains for 200K iterations on CelebA-HQ images resized to 128×128.Training takes around 40 hours on a single GTX 1080Ti GPU using PyTorch.
- H. Implementation Details: Training uses hinge adversarial loss with R1-regularization, with γ = 1, and the Adam optimizer with β1 = 0 and β2 = 0.99.
- H. Implementation Details: The learning rate is 0.0001 for the model and 0.000001 for the mapper.
I. Comparison without Cherry-picking
Without cherry-picking, HiSD produces satisfying diversity in latent-guided multi-style translation, while SDIT and StarGANv2 are observed to suffer from mode collapse.
- I. Comparison without Cherry-picking: HiSD generates satisfying diversity in completely random, non-cherry-picked latent-guided multi-style results.The comparisons use randomly produced results without manual selection.
- I. Comparison without Cherry-picking: SDIT and StarGANv2 are observed to suffer from mode collapse in the additional qualitative comparisons.
J. Interpolation of Tag-relevant Styles
Interpolating extracted tag-relevant styles produces smooth transitions, indicating continuous style spaces that support novel styles unseen during training.
- J. Interpolation of Tag-relevant Styles: Interpolation between extracted tag-relevant style codes from reference images produces smooth transitions.The reference images may share the same attribute or have different attributes.
- J. Interpolation of Tag-relevant Styles: Continuous tag-relevant style spaces allow translations to manipulate images with novel styles not seen during training.
K. Visualization of Tag-relevant Styles
HiSD’s extracted tag-relevant styles are visualized with t-SNE, revealing attribute-based grouping and structured directions within each tag’s style space.
- Visualization of Tag-relevant Styles: t-SNE visualizations show that images sharing a tag-specific attribute cluster together even though the attribute is not input to the extractor.
- Visualization of Tag-relevant Styles: Each tag-specific style space contains a main direction and secondary directions between attributes, with brown hair positioned between blond and black.
- Architectural details: HiSD encodes an input image with E, manipulates its feature through one or more T modules, and generates the output with G.The manipulation is guided by style codes generated by M or extracted by F; D evaluates whether an image is real given a tag and attribute.
- Additional qualitative results: Additional latent-guided results manipulate Bangs and Glasses attributes using 20 Gaussian random latent codes to generate diverse outputs.
- Additional qualitative results: Figures 12 and 13 provide further non-cherry-picked qualitative results for latent-guided multi-style translation.