PaperlayerView exact revision on arXiv

Source-linked AI summary

CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

Xiaoman Feng, Mingkun Lei, Yang Wang, Dingwen Fu, Chi Zhang

arXiv:2602.20721v1cs.CV

TL;DR

Encoder-based style transfer can leak semantic details from a style reference, while generic unconditional inputs may not suppress style-specific, prompt-irrelevant content. CleanStyle filters style-embedding tail components with a time-aware SVD strategy and reuses them for style-specific guidance. Experiments report reduced leakage and improved prompt fidelity, stylistic detail, and overall visual quality; the method adds inference cost from repeated SVD.

  • Problem

    Encoder-based style transfer can leak semantic details from the reference, and generic unconditional embeddings may fail to suppress style-specific, prompt-irrelevant content.

    Source: §1 · p. 2, §3.3 · p. 5

  • Method

    CleanStyle uses time-aware SVD suppression of style-embedding tail components and reuses them as style-aware unconditional inputs in SS-CFG.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Experiments report that CleanStyle suppresses content leakage, enhances prompt fidelity, preserves stylistic details, and improves overall visual quality.

    Source: §5 · p. 9

  • Takeaways & Limitations

    CleanStyle is a training-free, plug-and-play approach that integrates with multiple representative style-transfer frameworks.

    Source: §1 · p. 2, §5 · p. 8

    Repeated CS-SVD across denoising steps adds inference cost, which accumulates with cross-attention layer depth and sampling length.

    Source: G. Limitation and future work · p. 13

Abstract

from arXiv · show

Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.

1. Introduction

CleanStyle targets content leakage in encoder-based style transfer with training-free filtering of style embeddings and style-specific negative guidance. The reported experiments show reduced leakage and improved stylization quality and prompt adherence.

  • CS-SVD applies SVD to cross-attention style embeddings and dynamically suppresses tail components, filtering content-related artifacts while retaining stylistic expressiveness.Filtering is stronger early in denoising to establish a clean global layout, then relaxes to preserve fine-grained style.Source: §1 · p. 2
  • SS-CFG replaces generic unconditional inputs with a style-aware negative embedding built from the tail components separated by CS-SVD.This creates contrast between clean dominant style components and content-contaminated tail components.Source: §1 · p. 2
  • CleanStyle reduces content leakage and improves prompt adherence and generation quality, better preserving prompt semantics than InstantStyle while reflecting the reference style.Source: Figure 1 · p. 1, §1 · p. 2
  • The plug-and-play framework integrates with encoder-based diffusion models, including InstantStyle, DEADiff, and StyleShot, with minimal changes and no retraining.Source: §1 · p. 2

2. Related work

Prior style-transfer work includes encoder-based conditioning as well as approaches requiring fine-tuning, inversion, or adaptation. CleanStyle instead applies SVD analytically to image-based style embeddings to filter residual content signals without training or fine-tuning.

2.1. Style transfer with Diffusion Models

  • Encoder-based methods inject image-encoder style representations into diffusion models, while other approaches use fine-tuning, inversion, adaptation, or attention changes to address stylization and content leakage.CleanStyle offers an analytical, training-free alternative within encoder-based pipelines.Source: §2.1 · pp. 2–3

2.2. Singular Value Decomposition (SVD) in Diffusion Models

  • Prior diffusion uses of SVD include parameter reduction and manipulation of text embeddings; CleanStyle applies it to image-based style embeddings, whose tail components correlate with content-specific artifacts.Source: §2.2 · p. 3

3. Method

CleanStyle filters style embeddings at the cross-attention level and reuses content-related tail components as targeted negative guidance. Its components are designed to integrate into existing encoder-based stylization pipelines without retraining or architectural changes.

3.1. Preliminaries

  • Encoder-based pipelines inject image-encoder style embeddings into cross-attention Key and Value branches alongside text conditioning.The unconditional style input is typically null or zero, yielding a generic signal that does not represent instance-specific residual content.Source: §3.1 · p. 3

3.2. CleanStyleSVD

  • CS-SVD identifies tail components as a source of content-related signals and exponentially attenuates singular values beyond the top-k components.Suppression is stronger early in denoising and gradually relaxes to preserve fine-grained style details.Source: §3.2 · pp. 3–5
  • Filtering projected Key and Value matrices suppresses content leakage more effectively than filtering the encoder feature alone.The comparison in Fig. 6 reports insufficient suppression when filtering the feature alone.Source: §3.2 · p. 4

3.3. Style-Specific CFG

  • SS-CFG uses CS-SVD-filtered embeddings conditionally and the isolated tail component as a style-specific negative signal.This replaces generic unconditional embeddings to suppress unwanted content features and strengthen prompt alignment.Source: §3.3 · p. 5

3.4. Integration with other SOTAs

  • CleanStyle integrates with StyleShot and DEADiff as a plug-and-play module without retraining or architectural modifications.For DEADiff, it is applied exclusively to style components, leaving the text pathway unchanged.Source: §3.4 · p. 5

4. Evaluation and experiments

CleanStyle is evaluated against encoder-based style-transfer methods and integrated with multiple frameworks; the experiments examine its visual quality, prompt alignment, component contributions, and design choices.

4.1. Comparisons with State-of-the-Arts

  • In qualitative comparisons, CleanStyle reduces content leakage, better follows prompts, and maintains visual fidelity with fewer structural or stylistic distortions.Examples include more accurate scene composition and fewer semantic distortions than several baselines.Source: §4.1 · p. 6, Figure 4 · p. 6
  • Across 2,580 judgments from 43 participants, CleanStyle receives the highest overall preference for prompt adherence, style retention, and perceptual quality.The study used 60 questions derived from 20 diverse pairs and evaluated text alignment, style similarity, and image quality.Source: §4.1 · p. 6
  • CleanStyle achieves superior CLIP-TA scores, while its CLIP-SS and DINO-SS scores are slightly lower than each method’s original baseline.The authors describe this as a more favorable balance between semantic consistency and style preservation; Table 2 reports comparisons on CleanStyle.Source: §4.1 · p. 7, Table 2 · p. 7

4.2. Further analysis.

  • On StyleShot and DEADiff, CleanStyle mitigates content leakage while retaining stylistic features such as color, texture, and stroke.The integrations require no additional training according to the evaluation setup.Source: §4.2 · p. 7
  • The full model provides the most balanced performance: SS-CFG alone offers limited improvement, while CS-SVD alone improves prompt alignment but degrades style similarity.The ablation attributes the limited effect of SS-CFG alone to its need for a properly filtered conditional branch.Source: Table 3 · p. 7, §4.2 · p. 8, Figure 7 · p. 8
  • A larger top-k can reintroduce content leakage, while stronger suppression can discard fine-grained style textures.A moderate setting balances style strength and content cleanliness.Source: Figure 8 · p. 8, §4.2 · p. 8
  • Filtering the projected Key and Value matrices suppresses content leakage more effectively than filtering the image-encoder output directly.Without timestep awareness, stylistic textures can be missing; fixed linear or exponential schedules offer limited improvement.Source: §4.2 · p. 8
  • SS-CFG maintains high visual quality at large guidance weights, including stable color distribution, coherent structure, and consistent style expression.The authors present this as resilience to prompt amplification and flexible tuning without content distortion or style degradation.Source: §4.2 · p. 8, Figure 9 · p. 8

5. Conclusion

CleanStyle targets content leakage by suppressing embedding tail components with CS-SVD and using those components to form a style-specific negative condition in SS-CFG. Experiments across representative frameworks report reduced leakage, stronger prompt fidelity, preserved stylistic detail, and improved visual quality.

5. Conclusion

  • CleanStyle is designed to mitigate content leakage in stylized image generation by attenuating SVD tail components with a time-aware soft exponential strategy.The paper reports that these tail components often encode undesired content information.Source: §5 · p. 8
  • SS-CFG constructs a style-image-specific negative condition from suppressed tail components to guide suppression of undesired content.The conclusion contrasts this with existing encoder-based CFG formulations.Source: §5 · p. 8
  • Across multiple representative frameworks, qualitative and quantitative results show reduced content leakage, stronger prompt fidelity, preserved stylistic details, and improved visual quality.The paper characterizes CleanStyle as a versatile, training-free solution for stylized image generation.Source: §5 · pp. 8–9

D. Integration with other methods

The appendix presents additional qualitative comparisons with state-of-the-art methods and reports better text alignment, reduced content leakage, and preserved stylistic attributes.

A. More qualitative comparisons

  • The appendix compares InstantStyle, CSGO, IP-Adapter, StyleShot, DEADiff, and StyleStudio in additional qualitative results.The comparisons are shown in Figures A.1–A.4.Source: A. More qualitative comparisons · p. 11
  • Across the qualitative comparisons, CleanStyle shows better text alignment and reduced content leakage while preserving stylistic attributes such as color and texture.The appendix characterizes these improvements as consistent across various models and conditions.Source: A. More qualitative comparisons · p. 11

B. Quantitative comparison on StyleBench [7]

StyleBench results report improved CLIP Text Alignment across base models, while style-similarity metrics fluctuate; parameter analyses describe trade-offs between prompt alignment, content leakage, and stylistic detail.

B. Quantitative comparison on StyleBench [7]

  • Across base models on StyleBench, CleanStyle consistently improves CLIP Text Alignment, while CLIP Style Similarity and DINO Style Similarity show slight fluctuations.The authors relate these fluctuations to reducing style-induced content leakage.Source: B. Quantitative comparison on… · p. 11
  • CLIP-SS and DINO-SS do not explicitly penalize content leakage, and outputs containing reference-image semantics can still score highly on style similarity.The authors argue these metrics may conflate content with style and overlook prompt-fidelity violations.Source: B. Quantitative comparison on… · p. 11

C. Impact of parameters

  • In the parameter analysis, smaller k generally yields higher CLIP-TA but can lose stylistic textures, whereas larger k can raise style-similarity scores while reducing prompt alignment and increasing leakage.Overly large α0 may also wash out essential stylistic cues; the evaluation uses five randomly selected prompts and all 100 style images.Source: C. Impact of parameters · p. 11

D.1. Qualitative results on StyleShot [7]

On StyleShot, CleanStyle improves text alignment and stylistic consistency while preserving fine-grained visual attributes and suppressing content leakage. Qualitative examples show fewer unrelated elements and support the approach’s applicability across styles and conditions.

D.1. Qualitative results on StyleShot [7]

  • CleanStyle improves StyleShot’s text alignment and stylistic consistency while preserving fine-grained color and texture attributes.The section reports clear advantages over the baseline but provides no numerical values here.Source: §D.1 · pp. 11–12
  • For prompts including a kitten, a mountain, a lake, and a palm tree, CleanStyle suppresses unrelated faces and wooden shelves while maintaining stylistic fidelity.The baseline also introduces faces for other prompts, and CleanStyle avoids these artifacts.Source: §D.1 · p. 12
  • CleanStyle corrects a misleading green tree in a snowy-house prompt and filters spurious character faces from baseline generations.These examples are reported as evidence of reducing leakage while preserving stylistic expressiveness.Source: §D.1 · p. 12
  • The StyleShot results support applying CleanStyle across different styles and conditions; the CS-SVD and SS-CFG settings match those used with InstantStyle.StyleShot is based on SD1.5 and uses an encoder to extract style features.Source: §D.1 · pp. 11–12

D.2. Qualitative results on DEADiff [26]

On DEADiff, CleanStyle reduces content leakage in qualitative examples while preserving stylistic details. The accompanying user study gathered 2,580 judgments across text alignment, style similarity, and image quality.

D.2. Qualitative results on DEADiff [26]

  • On DEADiff, CleanStyle corrects unrelated or inaccurate visual elements across lotus, palm-tree, duck, cooking, cat, mouse, and water-bottle prompts while retaining artistic style.Examples include replacing green leaves, flowers, and red berries with content aligned to the prompts.Source: §D.2 · p. 12
  • For a lotus prompt, DEADiff adds unrelated green leaves, whereas CleanStyle mitigates the leakage while preserving stylistic integrity.Source: §D.2 · p. 12

E. Details of the user study

  • The user study collected 2,580 judgments from 43 complete responses across text alignment, style similarity, and overall image quality.Participants chose the best output among methods shown in randomized order for 60 questions derived from 20 style–prompt pairs.Source: E. Details of the user study · p. 12

F. Details of the CleanStyle dataset

The CleanStyle dataset pairs a diverse set of prompts with curated style images for controlled evaluation. Its hyperparameter study identifies a balance between retaining style components and suppressing content leakage.

F. Details of the CleanStyle dataset

  • The CleanStyle dataset pairs 52 prompts with 100 curated style images to create 5,200 text-style pairs.The prompts cover objects, scenes, and multi-clause descriptions, and originate from a widely adopted stylization benchmark.Source: F. Details of the CleanStyle dataset · p. 12
  • The 100 style images span artistic genres, color palettes, and texture characteristics, enabling evaluation of text adherence and stylistic consistency across generation conditions.Every prompt is paired with every style reference.Source: F. Details of the CleanStyle dataset · p. 12
  • The hyperparameter study reports a balanced trade-off around k=1 and α0∈[0.01, 0.02]; larger k reintroduces leakage, while larger α0 may weaken style richness.The study covers CLIP-TA, CLIP-SS, and DINO-SS.Source: Table A.2 · p. 13

G. Limitation and future work

CleanStyle’s main stated limitation is the inference cost of repeatedly applying CS-SVD during denoising. The authors identify image-to-image style transfer as a natural extension of the framework.

G. Limitation and future work

  • Repeated CS-SVD across denoising steps adds inference cost, with overhead accumulating according to cross-attention layer depth and sampling length.The number of cross-attention layers is the dominant runtime factor, while the size of the style-related Key/Value matrices also influences runtime.Source: G. Limitation and future work · p. 13
  • Approximate or cached decompositions are proposed as possible ways to reduce the computational burden.Source: G. Limitation and future work · p. 13
  • Adapting the framework to image-to-image style transfer is proposed as a natural extension because its mechanism operates on style embeddings rather than text conditioning.The authors suggest this may offer more controllable and content-preserving stylization across broader tasks.Source: G. Limitation and future work · p. 13

Related papers

Loading 2602.20721v1…