Source-linked AI summary
Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning
Yuxin Zhang, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, Tong-Yee Lee, Changsheng Xu
TL;DR
Arbitrary style transfer needs style representations that capture more than second-order feature statistics, which can limit style fidelity and consistency. CAST learns style directly from image features through contrastive learning, multi-layer style projection, and domain enhancement, and the paper reports superior results against state-of-the-art approaches. Its qualitative evaluations specifically describe better preservation of local style characteristics and content structure.
Problem
Existing arbitrary style-transfer methods rely heavily on second-order statistics, which do not fully capture available style information and can produce local distortions or style inconsistency.
Method
CAST learns style representations directly from image features using a multi-layer style projector, contrastive style learning, domain enhancement, and a generative style-transfer network.
Results
CAST achieves superior arbitrary style-transfer results compared with state-of-the-art approaches, balancing vivid local style characteristics with preserved content structure.
Takeaways & Limitations
Contrastive learning uses relationships among multiple styles and specific generated-reference pairs to guide style representation and style transfer.
Abstract
from arXiv · showhide
In this work, we tackle the challenging problem of arbitrary image style transfer using a novel style feature representation learning method. A suitable style representation, as a key component in image stylization tasks, is essential to achieve satisfactory results. Existing deep neural network based approaches achieve reasonable results with the guidance from second-order statistics such as Gram matrix of content features. However, they do not leverage sufficient style information, which results in artifacts such as local distortions and style inconsistency. To address these issues, we propose to learn style representation directly from image features instead of their second-order statistics, by analyzing the similarities and differences between multiple styles and considering the style distribution. Specifically, we present Contrastive Arbitrary Style Transfer (CAST), which is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer. We conduct qualitative and quantitative evaluations comprehensively to demonstrate that our approach achieves significantly better results compared to those obtained via state-of-the-art methods. Code and models are available at https://github.com/zyxElsa/CAST_pytorch
1 INTRODUCTION
The paper identifies artistic style representation as the central challenge in arbitrary style transfer and argues that second-order statistics are insufficient. CAST instead learns style representations directly from image features using contrastive learning, multi-layer projection, and domain enhancement.
- Artistic style representation is presented as the most important issue in artistic style transfer.
- Second-order statistics such as Gram matrices restrict arbitrary style transfer because styles differ in colors, local textures, layouts, and compositions.
- The paper argues that fitting second-order statistics of style and generated images does not directly represent artistic style or its distribution.
- CAST learns style representations through contrastive learning by comparing similarities and differences among multiple artistic styles.
- CAST combines an encoder-transformation-decoder backbone, a multi-layer style projector, and a domain enhancement module.
- Experiments report state-of-the-art visual style-transfer results, including outputs that could mislead survey participants distinguishing fake paintings from real ones.
2 RELATED WORK
Prior arbitrary style-transfer methods commonly rely on handcrafted or deep-feature statistics, while contrastive learning has been used for related image-translation objectives. CAST differs by using contrastive learning to represent style comprehensively rather than treating it as an auxiliary operation over feature statistics.
- Image style transfer: Traditional style-transfer methods use low-level handcrafted features, whereas neural methods model style through statistics of pretrained convolutional-network features.
- Contrastive learning: CAST uses visual features comprehensively for style representation, with style codes and generated images forming a contrastive-learning pipeline.
- Image style transfer: Arbitrary style-transfer architectures seek flexible feed-forward processing, but statistic alignment and dynamic normalization can introduce distortion artifacts.
- Contrastive learning: Contrastive learning has been applied to image translation for preserving input content and reducing mode collapse.
- Contrastive learning: TUNIT assumes semantic similarity, which does not hold for arbitrary style transfer and can significantly reduce learned style-representation performance.
- Contrastive learning: IEST applies contrastive learning to feature-statistics style priors and computes its loss only within generated results.
3 METHOD
CAST learns style representations directly from multi-layer image features using contrastive relationships and domain-level distribution modeling. Its generator uses these representations alongside adversarial, contrastive, and cycle-consistency objectives for arbitrary style transfer.
- Framework: CAST combines a multi-layer style projector, contrastive style learning, and domain enhancement within an encoder-transformation-decoder framework.The projector encodes artistic features, contrastive learning models positive and negative style relationships, and domain enhancement learns the artistic-image distribution.
- Multi-layer Style Projector: The MSP extracts VGG-19 features at multiple layers and projects each layer into a separate latent style space.The implementation uses ReLU1_2, ReLU2_2, ReLU3_3, and ReLU4_3, with pooling, convolution, and multilayer perceptrons.
- Contrastive Style Learning: MSP training contrasts an image with augmented views against negative artistic images, normalizing style vectors to prevent collapse.Augmentations include random resizing, cropping, and rotations; negatives are maintained in a 4096-example memory bank.
- Contrastive Style Learning: The generator applies contrastive guidance between each generated image and its specific reference style image in a one-on-one matching process.The generated image and reference image are positive examples, while negatives come from the MSP training dictionary.
- Domain Enhancement: Domain enhancement divides training images into realistic and artistic domains and uses separate discriminators to learn their distributions.A cycle-consistency loss is added to maintain content information during style transfer between the two domains.
4 EXPERIMENTS
Experiments compare CAST with state-of-the-art style-transfer methods through qualitative evaluations, user studies, perceptual metrics, and ablations. CAST better preserves style characteristics and content structure, while its contrastive style loss and domain enhancement improve stylization fidelity.
- 4.1 Qualitative Evaluation: Compared with prior methods, CAST better transfers detailed stylized patterns and avoids artifacts such as repetitive patterns, haloing, color distortion, and unexpected colors.The comparisons identify limitations across NST, AdaIN, SANet, MCCNet, AdaAttN, ArtFlow, and IEST.
- 4.1 Qualitative Evaluation: CAST achieves superior visual quality by balancing style-pattern characteristics with content structures.It represents vivid local strokes and overall appearance while preserving content structure.
- 4.1 Qualitative Evaluation: CAST successfully transfers large empty regions and vague appearances from style images while retaining content structures.Examples include Chinese painting’s blank regions and Claude Monet’s vague appearance.
- 4.2 Quantitative Evaluation: CAST obtains the highest accuracy in deception-rate evaluation, surpassing other methods by a large margin.The deception rate measures stylized images classified as the correct target styles by a VGG-19 classifier.
- 4.2 Quantitative Evaluation: CAST receives significantly higher human preferences for Sketch, Chinese painting, and Impressionism.User Study I compares CAST against eight state-of-the-art methods using randomly ordered stylized results.
- 4.2 Quantitative Evaluation: CAST-generated paintings have the lowest chance of being identified as fake, with precision and recall below 50% in the authenticity study.The study collected 2125 result groups from 85 participants.
- 4.3 Ablation Study: Replacing contrastive style loss with Gram-matrix loss loses stroke characteristics, whereas the full CAST model faithfully transfers brushstrokes, textures, and colors.Ablations show pencil lines becoming black blocks and oil-painting strokes becoming smooth blocks with murky colors.
- 4.3 Ablation Study: Removing realistic-image reconstruction slightly degrades stylization results.The ablation isolates the effect of asymmetric cycle-consistent training.
5 CONCLUSION AND FUTURE WORK
The paper presents CAST for arbitrary image style transfer and concludes that it improves results by learning style representations directly from image features. Future work will incorporate artist and category information into contrastive style learning.
- 5 CONCLUSION AND FUTURE WORK: CAST replaces second-order style statistics with direct image-feature encoding through a multi-layer style projector.The framework also uses contrastive loss and domain enhancement to model multi-style information and realistic/artistic image distributions.
- 5 CONCLUSION AND FUTURE WORK: Extensive experiments demonstrate superior arbitrary style-transfer results compared with state-of-the-art approaches.
- 5 CONCLUSION AND FUTURE WORK: Future work will improve contrastive style learning by considering artist and category information.