Source-linked AI summary
StyTr$^2$: Image Style Transfer with Transformers
Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, Changsheng Xu
TL;DR
CNN locality makes global information difficult to preserve in neural style transfer, producing biased content representations. StyTr2 uses separate content and style transformer encoders, a transformer decoder, and scale-invariant CAPE; experiments report effectiveness against CNN- and flow-based approaches.
Problem
CNN locality makes extracting and maintaining global input-image information difficult, leading neural style-transfer methods toward biased content representations.
Method
StyTr2 uses separate content and style transformer encoders, a transformer decoder for stylization, and content-aware positional encoding that adapts to image semantics and scale.
Results
Qualitative and quantitative experiments report StyTr2's effectiveness compared with state-of-the-art CNN-based and flow-based approaches.
Takeaways & Limitations
StyTr2 alleviates the content-leak problem of CNN-based models and provides a transformer-based approach to image style transfer.
Takeaways & Limitations
At present, StyTr2's test-time speed is not as fast as some CNN-based approaches.
Abstract
from arXiv · showhide
The goal of image style transfer is to render an image with artistic features guided by a style reference while maintaining the original content. Owing to the locality in convolutional neural networks (CNNs), extracting and maintaining the global information of input images is difficult. Therefore, traditional neural style transfer methods face biased content representation. To address this critical issue, we take long-range dependencies of input images into account for image style transfer by proposing a transformer-based approach called StyTr$^2$. In contrast with visual transformers for other vision tasks, StyTr$^2$ contains two different transformer encoders to generate domain-specific sequences for content and style, respectively. Following the encoders, a multi-layer transformer decoder is adopted to stylize the content sequence according to the style sequence. We also analyze the deficiency of existing positional encoding methods and propose the content-aware positional encoding (CAPE), which is scale-invariant and more suitable for image style transfer tasks. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed StyTr$^2$ compared with state-of-the-art CNN-based and flow-based approaches. Code and models are available at https://github.com/diyiiyiii/StyTR-2.
I. INTRODUCTION
CNN-based style transfer can lose long-range information and fine details, biasing content representation. StyTr2 addresses this with domain-specific transformer encoders, a transformer decoder, and content-aware positional encoding.
- Motivation: Existing neural style-transfer methods may fail when modeling relationships between content and style, despite efficient feed-forward generation.Self-attention has therefore been explored to improve stylization.
- Motivation: CNNs struggle to capture long-range dependencies without deep networks, whose increased depth can lose feature resolution and fine details.These losses can damage content-structure preservation and style display.
- Method: StyTr2 uses separate content and style transformer encoders, followed by a transformer decoder that progressively generates output image-patch sequences.The encoders obtain domain-specific information before decoding stylized content.
- Method: CAPE learns positional encoding from image semantic features and dynamically expands positions for different image sizes.It is designed to be scale-invariant and suitable for style-transfer generation.
- Results: StyTr2 is reported to outperform baseline methods while preserving desirable content structures and style patterns.The introduction attributes this conclusion to comprehensive qualitative and quantitative experiments.
II. RELATED WORK
Related work spans neural style transfer, vision transformers, and positional encoding. StyTr2 formulates style transfer as transformer-based sequence-to-sequence generation of image patches.
- Image style transfer: Traditional and neural style-transfer methods progressed from computationally intensive texture synthesis and optimization to efficient feed-forward CNN pipelines.Some CNN approaches target one style, while others combine multiple styles in one model.
- Transformer for vision tasks: Vision transformers use self-attention to capture long-term dependencies and have been applied across image and video tasks.The paper extends transformer-based structures to style transfer as sequence-to-sequence image-patch generation.
- Positional encoding: Positional encoding methods include functional, parametric, and relative formulations for representing token positions or distances.Functional encoding includes sinusoidal functions, while parametric encoding is learned during training.
- StyTr2 formulation: StyTr2 splits content and style images into patches and linearly projects them into sequential feature embeddings.The patch sequence has shape L×C, with L determined by image dimensions and patch size.
A. Content-Aware Positional Encoding
CAPE addresses positional encoding's semantic and scale limitations by conditioning positional information on image content and resizing a fixed encoding across image resolutions.
- Pipeline integration: The framework splits content and style images into patches, encodes them as sequences, and adds CAPE to the content sequence before transformer encoding.The pipeline then decodes stylized patch sequences and progressively upsamples them into the output image.
- Motivation: Traditional positional encoding depends on patch distance, while image patches also carry semantic content and may change relative distances when resized.These issues motivate encoding that reflects image semantics and remains stable across scales.
- CAPE design: CAPE conditions positional encoding on image-content semantics rather than only on relative patch distance.It assumes an n×n positional encoding can represent image semantics before adapting it to the target image.
- CAPE design: CAPE rescales a fixed n×n positional encoding to the input image dimensions, making spatial relations invariant across image scales.The rescaled encoding is used to avoid scale-dependent changes in positional relations.
- CAPE computation: The CAPE map is computed from average-pooled patch embeddings through a learnable 1×1 convolution and then added to each patch embedding.In the experiments, the average-pooling size n is set to 18.
B. Style Transfer Transformer
StyTr2 uses separate transformer encoders for content and style, then decodes the content sequence according to the style sequence before CNN-based refinement and upsampling.
- Transformer encoders: StyTr2 uses separate transformer encoders to learn domain-specific representations from natural content images and artistic style images.The style encoder omits positional encoding because the style image's input structure need not be maintained in the final output.
- Transformer encoders: The content encoder applies multi-head self-attention and a feed-forward network to produce an encoded content sequence.The input content embedding combines patch features with CAPE before encoding.
- Transformer decoder: The transformer decoder translates the encoded content sequence according to the encoded style sequence using two multi-head self-attention layers and one feed-forward network per layer.All sequential patches are supplied simultaneously rather than generated autoregressively as in NLP.
- Transformer decoder: The decoder uses content features to form queries and style features to form keys and values for cross-domain stylization.This connects content-driven output positions with style-derived transformation information.
- CNN decoder: A three-layer CNN decoder refines transformer outputs with convolution, ReLU, and 2× upsampling operations to produce an H × W × 3 image.The sequence is progressively expanded instead of directly upsampled in one step.
C. Network Optimization
Network optimization combines perceptual content and style losses with identity losses so outputs preserve content structures while matching reference style patterns.
- Perceptual losses: Content and style perceptual losses measure differences between the output and the content image, and between the output and style reference, respectively.Both losses use feature maps extracted by a pretrained VGG model.
- Perceptual losses: The style loss compares the means and variances of features extracted from the output and style images.The feature statistics are computed from pretrained VGG19 representations.
- Identity losses: Identity losses train StyTr2 to reproduce identical content or style images when the two inputs are the same.The generated Icc and Iss outputs are compared with their corresponding input images.
A. Implementation Details
Training uses MS-COCO content images and WikiArt style images with fixed-resolution crops, Adam optimization, warm-up learning-rate adjustment, and 160,000 iterations.
- Data: MS-COCO supplies content images and WikiArt supplies style images for training.Images are randomly cropped to 256 × 256 during training.
- Resolution: At test time, the model supports any image resolution.This contrasts with the fixed 256 × 256 training crops.
- Optimization: Training uses Adam with learning rate 0.0005, warm-up adjustment, batch size 8, and 160,000 iterations.These settings define the reported optimization configuration.
B. Comparisons with SOTA Methods
StyTr2 is compared with CNN-based and flow-based style transfer methods through runtime, qualitative, and quantitative evaluations. It preserves content structures and style patterns more effectively than the compared approaches.
- Evaluation setup: StyTr2 is evaluated against CNN-based methods, ArtFlow, and other approaches using inference-time, qualitative, and quantitative comparisons.The compared methods include AdaIN, Avater, SANet, AAMS, MAST, MCC, ArtFlow, AdaAttN, IEST, and StyleFormer.
- Qualitative evaluation: Qualitative comparisons show that StyTr2 avoids missing content and style details, producing well-preserved structures and desirable style patterns.The method uses transformer-based feature representation to capture long-range dependencies in input image features.
- Quantitative evaluation: StyTr2 achieves the lowest content losses, while SANet and StyTr2 outperform the other methods in style loss.The evaluation uses 800 stylized images generated from 40 style images and 20 content images.
- Quantitative evaluation: The quantitative metrics measure content and style differences between generated images and their respective input images, with smaller differences indicating better preservation.Content and style differences are used as indirect metrics of style-transfer quality.
C. Analysis of Content Leak
The analysis examines content leakage during repeated stylization, where CNN-based methods lose content details. StyTr2 preserves clearer content structures and alleviates this issue through transformer-based long-range dependency modeling.
- Content leak problem: Content leakage can emerge after repeated stylization because CNN-based feature representations may not sufficiently capture image-content details.The artifact becomes readily visible after several rounds of stylization.
- Content leak problem: The repeated stylization process applies successive generators to the content and style images, producing the result of the final round.The formulation represents the output as nested applications of G_i across stylization rounds.
- Experimental analysis: After the first round, CNN-based methods damage content structures to varying degrees, whereas StyTr2 still presents clear content details.The comparison also examines results after the 20th round of repeated stylization.
- Experimental analysis: StyTr2 captures precise content representations and uses transformer-based long-range dependencies to effectively alleviate content leakage.ArtFlow maintains clear content structures but has less satisfactory stylized effects in the comparison.
D. Analysis of CAPE
CAPE uses content-image semantics and dynamically adapts positional encoding to support consistent stylization across repetitive patterns and resolutions. Compared with sinusoidal positional encoding, CAPE avoids reported artifacts and supports input resolutions beyond training dimensions.
- CAPE incorporates content-image semantic information when computing positional encoding for style transfer.
- Sinusoidal positional encoding produces inconsistent stylized regions on content images with repetitive patterns or repeated collages.These comparisons use 256 × 256 inputs, matching the training resolution.
- At 512 × 512, sinusoidal positional encoding causes vertical track artifacts, whereas CAPE produces clear content structures and proper stylized patterns.512 × 512 is twice the training image resolution; CAPE supports arbitrary input resolutions by design.
- CAPE dynamically encodes each input and exhibits similar learnable positional patterns, helping it generalize across resolutions.The reported patterns include highlighted diagonal, repeat, and periodic structures.
- The user study collected 4,000 votes per question and found StyTr2 superior to competing methods on overall quality, content preservation, and style consistency.The study involved 100 participants and compared AdaAttN, ArtFlow, MCC, AAMS, and AdaIN.
V. CONCLUSION
StyTr2 combines separate content and style transformer encoders with a transformer decoder and content-aware positional encoding for image style transfer. The framework alleviates content leak in CNN-based models, but its test-time speed remains slower than some CNN approaches.
- StyTr2 uses content and style transformer encoders to capture domain-specific long-range information, then decodes content sequences using reference style sequences.
- Content-aware positional encoding is semantics-aware and designed for scale-invariant visual generation.
- StyTr2 alleviates the content leak problem of CNN-based models and provides fresh insight into image style transfer.
- Test-time speed is not as fast as some CNN-based approaches.The authors identify incorporating CNN priors to speed computation as future work.