PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPstyler: Image Style Transfer with a Single Text Condition

Gihyun Kwon, Jong Chul Ye

arXiv:2112.00374v3cs.CVcs.CLeess.IV

TL;DR

Existing neural style transfer requires reference style images, and text-guided generative methods can be confined to trained image domains. CLIPstyler trains a lightweight CNN with patch-wise CLIP matching, multiview augmentation, and threshold rejection to transfer semantic textures from text alone. Experiments show varied, realistic text-driven styles without style images.

  • Problem

    Existing style transfer requires reference style images, while text-guided generative approaches depend heavily on pretrained models and are confined to trained image domains.

  • Method

    CLIPstyler trains a lightweight CNN using patch-wise CLIP loss, perspective augmentation, and threshold rejection to transfer text-conditioned semantic textures.

  • Results

    Extensive experiments show that the framework transfers a variety of unique styles from text conditions without requiring style images.

  • Takeaways & Limitations

    The framework produces realistic style-transfer results by changing text conditions alone, including across content images and image domains.

  • Takeaways & Limitations

    Existing text-guided generative methods are confined to trained image domains, whereas the paper's framework targets source images regardless of domain.

Abstract

from arXiv · show

Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. However, in many practical situations, users may not have reference style images but still be interested in transferring styles by just imagining them. In order to deal with such applications, we propose a new framework that enables a style transfer `without' a style image, but only with a text description of the desired style. Using the pre-trained text-image embedding model of CLIP, we demonstrate the modulation of the style of content images only with a single text condition. Specifically, we propose a patch-wise text-image matching loss with multiview augmentations for realistic texture transfer. Extensive experimental results confirmed the successful image style transfer with realistic textures that reflect semantic query texts.

1. Introduction

Existing style transfer methods need reference style images, while CLIPstyler transfers semantic textures from a single text condition. It uses patch-wise CLIP matching, multiview augmentation, and threshold regularization to produce realistic text-driven styles across content images.

  • Existing methods: Traditional neural style transfer transfers style texture by matching content and style features, commonly through Gram-matrix style loss.This formulation underlies pixel-optimization, arbitrary-style, and feedforward stylization methods.
  • Motivation: Reference style images limit applications where users want imagined styles, such as Monet, Van Gogh, or nighttime appearance.The stated motivation is style transfer without owning or supplying a corresponding painting or style image.
  • Motivation: Text-guided approaches use text-image embeddings but may inadequately reflect semantics or remain restricted to domains supported by pretrained generative models.These limitations motivate a more general text-conditioned style-transfer framework.
  • Proposed framework: CLIPstyler trains a lightweight CNN to transform content images toward text conditions by matching CLIP outputs of transferred images with the query text.The method is designed to transfer styles across multiple content images rather than optimize only one image.
  • Technical innovations: Patch-wise CLIP loss treats local output patches as brushstrokes, while perspective augmentation promotes vivid, diverse, and locally consistent style transfer.The loss samples patches, applies multiple perspective views, and matches the processed patches to the text condition.
  • Results: Extensive experiments show that the model transfers varied unique styles from text conditions, widening style-transfer options beyond methods requiring style images.

2. Related works

Related work spans image-based style transfer and text-guided image manipulation. Existing methods either require style images or depend on pretrained generative models, constraining the domains they can address.

  • Image-based style transfer: Image-based style transfer evolved from VGG feature and Gram-matrix matching to feedforward, whitening-coloring, attention, wavelet, graph, and domain-aware approaches.
  • Image-based style transfer: Despite successful results, conventional style-transfer methods require style images to make content follow a target texture.
  • Text-guided synthesis: Text-guided synthesis methods use text embeddings as conditions for generative models, with attention and joint text-image feature modules improving text-conditioned generation.
  • Text-guided synthesis: CLIP connects text and image domains using representations learned from 400M text-image pairs, enabling text-conditioned image manipulation.StyleCLIP explores StyleGAN's latent space, while StyleGAN-NADA modifies a trained model toward a novel domain without additional training images.
  • Text-guided synthesis: Generative-model-based text manipulation is confined to trained image domains because it depends heavily on pretrained generative models.The paper contrasts this with transferring text-conditioned texture to source images regardless of image domain.

3. Method

CLIPstyler optimizes a lightweight CNN to transform a content image toward a text-specified style without using a reference style image. Its objective combines global and directional CLIP guidance with patch-wise texture matching and content-preserving regularization.

  • 3.1. Basic framework of CLIPstyler: The framework optimizes a CNN f so the stylized output Ics=f(Ic) follows the target style text without a reference style image.The encoder-decoder captures hierarchical content features while stylizing in deep feature space.
  • 3.2. Loss function: Global CLIP loss aligns the whole output with the target text, but can corrupt output quality and make optimization unstable.The method therefore supplements global guidance with directional and patch-wise losses.
  • 3.2. Loss function: Directional CLIP loss aligns the image change from content to output with the text change from source content semantics to target style semantics.For natural-image content, the source text is set to “Photo”; the loss uses CLIP image and text encoders.
  • 3.2. Loss function: PatchCLIP minimizes CLIP loss over randomly cropped output patches, extending style-transfer guidance toward spatially invariant local texture information.The patches are sampled from arbitrary locations, and the resulting local supervision is intended for texture transfer.
  • 3.2. Loss function: Random geometrical, including perspective, augmentation presents each patch from multiple views, encouraging more vivid and diverse texture representations.Perspective augmentation guides patches toward the same semantics across viewpoints, supporting more 3D-like semantic structures.
  • 3.2. Loss function: Threshold rejection nullifies losses for patches whose scores fall at or below τ, preventing easy-to-optimize patches from driving over-stylization.The total objective combines directional CLIP, patch-wise CLIP, content, and total-variation losses.

4. Results

Experiments evaluate implementation settings, qualitative transfer, comparisons with style-transfer and text-guided baselines, manipulation alternatives, and ablations of the proposed components.

  • Experiment settings: Content images use 512×512 resolution, while the lightweight U-net uses three downsample and three upsample layers with channel sizes 16, 32, and 64.Training uses Adam with learning rate 5×10^-4 and 200 iterations.
  • Experiment settings: The default configuration samples 64 patches of size 128, applies perspective augmentation, and uses threshold rejection with τ = 0.7.Patch size 128 produced the best perceptual quality, while varying crop size produced different effects.
  • Experiment settings: Prompt engineering averages embeddings from several texts with the same meaning to reduce text-embedding noise.The authors also apply the same contrast enhancement to all outputs, including baselines.
  • Qualitative evaluations: The method transfers artistic and general texture styles from text conditions while preserving content structure and producing vivid patterns across image locations.Text can specify texture, color, and a target pattern object, such as “white wool,” “green crystal,” or oil painting with flowers.
  • Comparison with baselines: Compared with baseline style-transfer methods, the method uses only text conditions yet produces complex textures matching the queried semantics, including non-artistic styles.The comparison includes AdaIN, Gatys et al., AdaAttn, SANet, and CST.
  • Comparison with baselines: Against text-guided manipulation and alternative CLIP-loss approaches, the method produces realistic whole-image textures while better preserving content than models that modify identity or deteriorate structure.Pixel optimization failed to reflect text semantics, while VQGAN+CLIP and AdaIN+CLIP severely deteriorated content structures.
  • Ablation studies: Ablations show that the full loss combination gives the best perceptual results, while removing global loss, threshold rejection, augmentation, perspective augmentation, or patch loss causes distinct degradations.Observed degradations include irregular color patterns, over-stylization, missing realistic texture, artifacts, and limited texture change.
  • Ablation studies: The fast training scheme uses randomly cropped patches as content images and the same proposed losses to train a decoder network.The framework is illustrated in Figure 8.

5. Further Extensions

Patch-based training extends fast text-conditioned style transfer from a single content image to arbitrary content images, including high-resolution inputs. The method produces realistic semantic textures in real time while preserving content details, though some styles can affect unnecessary regions.

  • Fast style transfer: Training on diverse texture patches lets the style network transfer text-conditioned styles to arbitrary content images in real time.The model is trained with various texture patches rather than a single content image.
  • Fast training scheme: A fast training scheme uses randomly cropped high-resolution DIV2K texture patches and fine-tunes only a pretrained VGG decoder for about 40 seconds.The model is updated for 200 Adam steps with learning rate 1 × 10^-4.
  • Results: The resulting outputs reflect semantic textures across content images with different structural diversity while retaining similar texture-transfer quality to single-content training.The authors report high-quality style transfer for arbitrary content images.
  • Observed limitation: Some styles, including “Neon Light” and “Fire,” place textures on unnecessary background regions.This is reported as a limitation of some fast-transfer outputs.
  • High-resolution transfer: High-resolution transfer can process a 3000×2000 image in about 4 seconds for the text condition “A sketch with black pencil”.The result is produced by the fast style transfer method.
  • High-resolution transfer: Patch-based training enables higher-resolution style transfer while maintaining content details under text conditions.The method feeds high-resolution images directly into the trained network.

6. Conclusion

The paper concludes that text conditions alone can drive semantic texture transfer without reference style images. Its framework combines patchCLIP loss and augmentation to produce realistic style-transfer results by changing the text condition.

  • Conclusion: The proposed framework transfers semantic texture information using only a text condition, without requiring a style image.This is the paper’s central stated contribution.
  • Conclusion: PatchCLIP loss and an augmentation scheme support realistic style-transfer results from changing text conditions.The conclusion attributes the result to these proposed components.
  • Conclusion: Experimental results are reported to demonstrate state-of-the-art image style transfer.The paper directs readers to supplementary materials for limitations and social impacts.

A. More details of implementation

The implementation uses a lightweight residual U-Net style network and a combined loss for text-guided style transfer. Architectural limits and specified loss weights support fast training while preserving content information.

  • Network architecture: The style network is a lightweight U-Net with residual blocks to improve content preservation and training stability.Its maximum channel count is 128, and the highest-resolution layer has 16 channels.
  • Loss function: The total loss combines directional, patch, content, and total-variation terms: L_total = λ_d L_dir + λ_p L_patch + λ_c L_c + λ_tv L_tv.The equation is stated to match the loss of the basic method.
  • Hyperparameters: The implementation sets λ_d, λ_p, λ_c, and λ_tv to 1, 10, 1, and an incompletely specified final value.The supplied implementation passage does not state the final λ_tv value.
  • Implementation details: Further implementation details are deferred to the source code.No additional implementation specification is provided in the passage.

B. Additional comparison results

The paper compares its method with style transfer using retrieved or generated images as style references. Those baselines can produce reasonable outputs, but degrade when retrieval or generation provides an inadequate style image.

  • Compared baselines: The comparison uses two baselines: existing style transfer with a text-retrieved image, and with an image generated by VQGAN+CLIP.The retrieved image is selected by highest CLIP score.
  • Results: Both baselines produce reasonable results when the retrieved or generated style image is adequate.The comparison is shown in the upper row of Figure 12.
  • Results: When the retrieved or generated style image is inadequate, both baseline outputs degrade despite efficient inference.The authors identify dependence on retrieval and generation model performance as a major disadvantage.

B.2. User study

The evaluation combines user studies, CLIP-based texture correspondence, ablations, and failure-case analysis to assess content preservation, stylization, and patch-wise design choices.

  • Evaluation setup: The user study compared CLIP-guided models using artistic and non-artistic text conditions, evaluating content preservation and text customization with ratings from 25 participants.The study generated 1,120 stylized images across models and text conditions.
  • User-study results: Our model achieved the best combined performance in content preservation and stylization, with the second-best content-preservation score and the best stylization score.StyleCLIP had the best content-preservation score but the worst stylization score, while AdaIN+CLIP traded weaker content preservation for the second-best style-transfer score.
  • Quantitative comparison: The model obtained the highest averaged patch-wise CLIP scores, while StyleCLIP scored lowest and AdaIN+CLIP scored second among the baselines.These CLIP-score tendencies closely matched the user-study stylization scores.
  • Ablation study: Removing the patch-wise CLIP loss produced the worst ablation preference score, identifying Lpatch as the most important loss for stylization.The ablation user study compared settings without Ldir, Lpatch, threshold rejection, or augmentations.
  • Patch analysis: Larger training patches create coarser brushstrokes, whereas smaller patches produce finer style patterns, with little perceptual-quality difference from cropping content patches.The default patch size was 128, and content-image patch cropping was omitted for computational efficiency.
  • Failure cases: Failure cases arise from poorly sampled patches or direct visualization of text, while overly specific conditions can produce unwanted patterns instead of texture.A larger total-variation loss can partially reduce some artifacts, but does not fully solve them.

F. Additional results

Additional experiments show CLIPstyler transferring realistic textures across varied content images, while comparisons and high-resolution results further demonstrate its qualitative capability.

  • Additional qualitative results: CLIPstyler applies realistic, text-guided textures across various content images and achieves strong content preservation and texture synthesis in qualitative comparisons.The comparisons include Retrieve + Sty, Text2Image + Sty, StyleGAN-NADA, StyleCLIP, VQGAN+CLIP, and AdaIN+CLIP.
  • High-resolution results: The fast style-transfer method produces high-resolution outputs from text conditions, although the passage does not quantify their performance.

G. Limitations and Negative social impact

The method has practical and technical limitations: each text condition requires network training, poor patch samples can reduce quality, and malicious prompts may create harmful effects.

  • Technical limitations: Each given text condition requires network training, so real-time style transfer is not yet possible.The fast style-transfer method partially addresses this issue but produces lower quality than default single-image optimization.
  • Technical limitations: Badly sampled patches can produce low-quality results, motivating future improvements to the sampling process.
  • Negative social impact: Malicious words can produce negative social effects, especially when applied to personal photos containing sensitive information.
Loading 2112.00374v3…