PaperlayerView exact revision on arXiv

Source-linked AI summary

CSGO: Content-Style Composition in Text-to-Image Generation

Peng Xing, Haofan Wang, Yanpeng Sun, Qixun Wang, Xu Bai, Hao Ai, Renyuan Huang, Zechao Li

arXiv:2408.16766v2cs.CV

TL;DR

Image style transfer remains challenging, and the scarcity of large-scale stylized data has constrained end-to-end training. The paper builds a triplet dataset and an end-to-end model with separate content and style feature injection. Experiments report strong style control across image-driven transfer, text-driven synthesis, and text editing-driven synthesis.

  • Problem

    Large-scale stylized datasets are scarce, while image style transfer remains challenging and existing methods face limitations such as slower inference, content loss, or fine-tuning needs.

    Source: §1 · p. 2

  • Method

    The authors construct IMAGStyle with 210K triplets and train CSGO end-to-end using independent content and style feature injection for multiple style-transfer tasks.

    Source: §1 · p. 2, §4.1 · p. 5, §6 · p. 10

  • Results

    Experiments report strong style control for image-driven transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis in a unified CSGO framework.

    Source: §5.2 · pp. 7–8, §6 · p. 10

  • Takeaways & Limitations

    IMAGStyle and CSGO provide a shared dataset and framework for studying image style transfer and stylized synthesis without inference-stage fine-tuning.

    Source: §1 · p. 2, §6 · p. 10

    The dataset contains only 210K triplets, and CSGO is described as a basic version whose feature extraction and fusion methods could be further improved.

    Source: §6 · p. 10

Abstract

from arXiv · show

The diffusion model has shown exceptional capabilities in controlled image generation, which has further fueled interest in image style transfer. Existing works mainly focus on training free-based methods (e.g., image inversion) due to the scarcity of specific data. In this study, we present a data construction pipeline for content-style-stylized image triplets that generates and automatically cleanses stylized data triplets. Based on this pipeline, we construct a dataset IMAGStyle, the first large-scale style transfer dataset containing 210k image triplets, available for the community to explore and research. Equipped with IMAGStyle, we propose CSGO, a style transfer model based on end-to-end training, which explicitly decouples content and style features employing independent feature injection. The unified CSGO implements image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis. Extensive experiments demonstrate the effectiveness of our approach in enhancing style control capabilities in image generation. Additional visualization and access to the source code can be located on the project page: \url{https://csgo-gen.github.io/}.

1 INTRODUCTION

Diffusion models have advanced text-to-image generation, but image style transfer remains challenging because it requires fine-grained control over content and style. The paper addresses the lack of large-scale stylized data with IMAGStyle and an end-to-end style-transfer framework, CSGO.

1 INTRODUCTION

  • Although diffusion models such as SD generate visually appealing images from text, image style transfer remains challenging despite advances in general controllability.Style transfer must preserve content semantics while adopting a desired style across texture, color, composition, and visual quality.Source: §1 · pp. 1–2
  • A major obstacle is the lack of large-scale stylized datasets, which prevents end-to-end training and leaves non-end-to-end methods with suboptimal style-transfer quality.Existing approaches include inversion-based feature extraction and LoRA fine-tuning, which can increase inference time or reduce stability.Source: §1 · p. 2
  • The paper constructs IMAGStyle with 210K content-style-stylized triplets and introduces end-to-end CSGO with independent content and style feature injection.CSGO supports image-driven transfer, text-driven synthesis, and text editing-driven synthesis without inference-stage fine-tuning.Source: §1 · p. 2

2 RELATED WORK

Related work spans controllable text-to-image models and style-transfer methods, from optimization and feature-statistics approaches to inversion, trained controls, and fine-tuning. The paper presents CSGO as a zero-shot style-transfer framework addressing limitations in existing methods.

Text-to-Image Model

  • Controllable text-to-image models such as ControlNet, T2Iadapter, and IP-Adapter add image conditions and integrate extracted features to improve layout control.The paper positions CSGO as a style-transfer framework based on image-conditional generation.Source: §2 · p. 2

Style Transfer

  • Earlier optimization- and inference-based transfer methods face speed or style-diversity limits, while AdaIN-inspired and transformer approaches remain limited in arbitrary style transfer.The related work also notes that these methods primarily focus on color or stroke transfer.Source: §2 · pp. 2–3
  • Inversion-based methods can lose content detail or increase inference time, while trained controls and fine-tuning approaches also face content-preservation or per-style tuning challenges.The paper proposes CSGO and a new dataset for high-quality arbitrary image style transfer without fine-tuning.Source: §2 · p. 3

3 DATA PIPELINE

The paper constructs content-style-stylized triplets through stylized-image generation and automatic cleaning, then uses the pipeline to build IMAGStyle. The dataset draws on varied content and style image sources and includes 210K training triplets plus a separate testing set.

3.1 PIPELINE FOR CONSTRUCTING CONTENT-STYLE-STYLIZED IMAGE TRIPLETS

  • The pipeline generates stylized candidates from content and style images, then filters the candidates to select a suitable target image.The generation process trains content LoRAs on individual content images, separates content and style LoRAs, combines them, and selects the candidate with the minimum CAS; the implicit separation can be unstable.Source: §3.1 · pp. 3–4
  • Automatic cleaning uses CAS, a feature distance between style-stripped content semantics of the generated and original images, with smaller scores indicating closer content alignment.The feature transform removes style information using feature mean and variance; DINO-V2 was closest to human filtering results among the compared feature extractors.Source: §3.1 · p. 4

3.2 IMAGSTYLE DATASET DETAILS

  • Content images come from MSRA10K, MSRA-B, and ImageNet-Sketch, while style images come from Wikiart and Midjourney-generated images.The content sources provide salient images and sampled sketches; the style sources include painting styles and a range of generated styles.Source: §3.2 · p. 4
  • IMAGStyle contains 210K training triplets and a test set of 248 content images and 206 style images, with each test content image transferred to all 206 styles.The test content covers real scenes, sketched scenes, faces, and style scenes.Source: §3.2 · p. 4

4 APPROACH

CSGO uses independently injected content and style features to support image-driven transfer and text-driven stylized synthesis. It is trained end-to-end on content, caption, style, and target-image conditions, with classifier-free guidance at inference.

4.1 CSGO FRAMEWORK

  • Content control combines ControlNet injections in up-sampling blocks with cross-attention injections in down-sampling blocks.The passage says these strategies ensure small content loss during style transfer.Source: §4.1 · p. 5
  • Style control projects style-image embeddings into new embeddings and injects style features through cross-attention in up-sampling blocks and an independent module.The independent module addresses the risk that content-image style information leaks through ControlNet injections.Source: §4.1 · p. 5
  • CSGO explicitly learns separate modules to inject content and style features at different locations in the base model.The framework supports image-driven transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis.Source: §4.1 · pp. 4–5

4.2 MODEL TRAINING AND INFERENCE.

  • Training uses content image C, caption P, style image S, and target image T to train a style-transfer network based on a pre-trained diffusion model.The objective models the relationship between the styled image and Gaussian noise under the content and style image conditions.Source: §4.2 · p. 6
  • During inference, classifier-free guidance combines the conditioned prediction with the unconditioned prediction using guidance factor w.The passage defines w as the classifier-free guidance factor.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments evaluate CSGO on image-driven style transfer, text-driven stylized synthesis, and text editing-driven synthesis, alongside ablations of its feature-injection design. Results report strong style control and content retention, with the ablations examining how design choices and scales affect those outcomes.

5.1 EXPERIMENTAL SETUP

  • Experiments use IMAGStyle for training and its test set for evaluation, measuring style similarity with CSD and content similarity with CAS.The compared methods include inversion-based, Transformer-based, and ControlNet/IP-Adapter approaches; CSGO uses Stable Diffusion XL, a ViT-H image encoder, and ControlNet.Source: §5.1 · pp. 6–7

5.2 EXPERIMENTAL RESULTS

  • CSGO achieves the highest CSD score among the compared methods and maintains high-quality content retention in image-driven style transfer.The comparison evaluates style similarity and content alignment on the test dataset.Source: §5.2 · pp. 7–8, Table 1 · p. 7
  • In text-driven synthesis, CSGO better follows prompts in simple scenes and controls style while converting text meaning in complex scenes.The authors attribute this to decoupled content and style features and a style feature injection block.Source: Figure 6 · p. 7, §5.2 · p. 8
  • Text editing-driven synthesis preserves the original content image’s semantics and layout while allowing simple edits to textual prompts.Source: Figure 8 · p. 7, §5.2 · p. 8

5.3 ABLATION STUDIES.

  • ControlNet injection preserves content layout and semantics, while injecting style features into the ControlNet branch strengthens style transfer.Cross-attention injection alone guarantees semantic information but not full content preservation; style injection into the base UNet alone weakens generated-image style.Source: §5.3 · pp. 8–9, Figure 9 · p. 8
  • The Resampler projection captures more detailed style features while avoiding content leakage, compared with the explored linear layer.Source: Figure 10 · p. 8, §5.3 · p. 9
  • Increasing style token number t progressively improves style control in image transfer and text-driven synthesis.The authors associate larger t with extraction of richer style features.Source: Figure 11 · p. 8, §5.3 · p. 9
  • Small δc weakens content injection and favors text prompts and style, while larger δc improves content retention but can severely weaken style; CFG enhances style transfer.The style scale is suggested between 1.0 and 1.5 because values below 1.0 severely weaken style, while λc is most effective near 1.0.Source: §5.3 · p. 9, Figure 13 · p. 9

6 CONCLUSION.

The work introduces IMAGStyle, a 210K-triplet dataset, and CSGO, an end-to-end framework for style transfer. CSGO supports image style transfer and two text-driven synthesis tasks; the authors identify dataset scale and feature extraction and fusion as areas for further improvement.

6 CONCLUSION.

  • IMAGStyle and CSGO are presented as beneficial for style transfer, with experiments validating their contributions.IMAGStyle contains 210K image triplets.Source: §6 · p. 10
  • CSGO unifies image style transfer, text-driven style synthesis, and text editing-driven style synthesis.The conclusion describes the framework as an end-to-end training style transfer framework.Source: §6 · p. 10
  • The dataset contains 210K triplets because of time and computational resource constraints; the authors expect expansion to improve style transfer quality.They also describe CSGO as a basic version and suggest optimizing style and content feature extraction and fusion.Source: §6 · p. 10

Related papers

Loading 2408.16766v2…