PaperlayerView exact revision on arXiv

Source-linked AI summary

CLeaR: A Unified Framework for Resolving the Leakage-Degradation Dilemma in Style Transfer

Teng Zhou, Yunhao Chen

arXiv:2609.38136v2cs.CV

TL;DR

Style transfer seeks to render target content in a reference style, but content leakage and the leakage-degradation dilemma complicate this task across the pipeline. CLeaR combines content-reduced style targets, a shared multi-VFM anchor, and diffusion-time calibration. On StyleBench, it improves style alignment, content leakage metrics, and LLM-as-Judge evaluation.

  • Problem

    Style transfer can leak objects, layouts, or semantics from a style reference, and stronger content suppression may weaken style fidelity.

    Source: §1 · p. 1

  • Method

    CLeaR is a training-free framework that combines Orthogonal Subspace Projection, Ensemble Inversion, and Energy-Guided Calibration to reduce leakage and maintain style alignment.

    Source: §1 · p. 2

  • Results

    Experiments on StyleBench show that CLeaR improves style alignment, content leakage metrics, and LLM-as-Judge evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CLeaR addresses content leakage across feature separation, feature-space grounding, and diffusion generation with a unified framework.

    Source: §5 · p. 9

    CLeaR adds test-time computation and remains slower than simpler feed-forward methods, despite runtime comparable to several training-free baselines.

    Source: §5 · p. 10

Abstract

from arXiv · show

Style transfer aims to render target content in the style of a reference image, but existing methods often suffer from content leakage, where objects, layouts, or semantics from the style reference appear in the generated output. Although prior data-driven and training-free methods can reduce leakage, they often face a leakage-degradation dilemma: stronger content suppression may weaken style fidelity, while richer style preservation may reintroduce unwanted reference content. We identify this dilemma across the full style-transfer pipeline, including feature separation, feature-space grounding, and diffusion generation. To address these issues, we propose CLeaR, a training-free framework for content-leakage-resistant style transfer. CLeaR first uses Orthogonal Subspace Projection to define content-reduced style targets in each vision foundation model (VFM) feature space. It then performs Ensemble Inversion, which optimizes a shared pixel-space style anchor satisfying style constraints across multiple VFMs. Finally, Energy-Guided Calibration maintains style alignment during diffusion sampling by steering the denoising trajectory toward the ensemble-defined style manifold. We further provide a theoretical analysis showing that the style-anchor estimation error decreases with the number of VFMs. Experiments on StyleBench demonstrate that CLeaR improves style alignment, reduces content leakage, and achieves better LLM-as-Judge evaluation compared with existing methods. The code is available at \href{https://github.com/0606zt/CLeaR}{https://github.com/0606zt/CLeaR}.

1 Introduction

Existing style-transfer methods face a leakage-degradation dilemma across feature separation, representation grounding, and diffusion generation. CLeaR addresses these stages with content-reduced targets, a shared multi-VFM style anchor, and test-time calibration.

  • Content leakage can bring objects, layouts, or semantics from the style reference into the output, while stronger suppression may remove stylistic details.Source: §1 · pp. 1–2
  • Orthogonal Subspace Projection removes the reference feature component aligned with a content proxy and retains the orthogonal residual as the style target.Source: §1 · p. 2
  • Ensemble Inversion optimizes an image in shared pixel space so its features satisfy heterogeneous style constraints across multiple VFMs.Source: §1 · p. 2
  • Energy-Guided Calibration uses ensemble feature-alignment energy and its gradient at selected denoising steps to steer generation toward the VFM-defined style manifold.Source: §1 · p. 2
  • The theoretical analysis shows that style-anchor estimation error decreases with the number of VFMs under limited cross-model dependence.Source: §1 · p. 2
  • Experiments on StyleBench show that CLeaR improves style alignment, content leakage metrics, and LLM-as-Judge evaluation.Source: §1 · p. 2

2 Related Work

Style transfer has shifted from optimizing deep feature statistics toward conditional generation with diffusion models, while content-style separation remains central. Prior methods use learned representations or manipulate pretrained features, and model inversion offers a route for combining feature spaces through pixels.

  • Style transfer was formalized through optimization of deep feature statistics and has increasingly become conditional generation with diffusion models and conditioning adapters [10] [26] [23] [52].Source: §2 · p. 2
  • Prior methods either train style representations or projection modules, or manipulate pretrained VFM and diffusion features; both approaches can face leakage-degradation across the pipeline [19] [9] [44] [24] [58] [18] [49] [43] [62] [20] [29, 5, 13, 12] [57] [13] [61].Source: §2 · p. 3
  • Model inversion provides a bridge from heterogeneous feature spaces to pixel space, which CLeaR uses as a shared grounding domain for combining representations [56] [14] [39] [3].Source: §2 · p. 3

3 Methodology

CLeaR addresses failure sources across feature separation, feature-space grounding, and diffusion generation with three corresponding training-free components. Its method defines content-reduced style targets, combines them in a shared pixel-space anchor, and calibrates diffusion sampling toward the resulting style target.

3 Methodology

  • CLeaR pairs Orthogonal Subspace Projection, Ensemble Inversion, and Energy-Guided Calibration with feature separation, feature-space grounding, and diffusion generation, respectively.Together, the components target three failure sources across the style-transfer pipeline.Source: Figure 1 · p. 3, §3 · p. 3

3.1 Orthogonal Subspace Projection

  • Orthogonal Subspace Projection removes the reference feature’s component along a content proxy, defining a content-reduced style target in each VFM feature space.The proxy is generated from the content description; the projection makes the resulting style feature orthogonal to the content feature.Source: §3.1 · pp. 3–4

3.2 Ensemble Inversion

  • Ensemble Inversion optimizes one pixel-space image whose features match model-specific style targets across VFMs with different feature-space dimensions and geometries.The resulting image serves as a shared style anchor across the ensemble.Source: §3.2 · p. 4

3.3 Energy-Guided Calibration

  • Energy-Guided Calibration uses the style-alignment energy gradient to adjust the latent during denoising, steering sampling toward the ensemble-defined style target without retraining.A lower energy indicates stronger alignment with the target style representation.Source: §3.3 · pp. 4–5

3.4 Theoretical Analysis

  • When centered residuals are weakly correlated across models, the dominant inversion-error term decreases approximately as 1/K; the bound’s gain depends on cross-model dependence.The reported VFM-composition ablation is consistent with this prediction: adding VFMs improves both SA and CL.Source: §3.4 · pp. 5–6

4 Experiments

Experiments compare CLeaR with style-transfer baselines on StyleBench and examine how content descriptions, VFM composition, and calibration timing affect disentanglement. The reported comparisons and ablations show gains in style alignment and content-leakage suppression, with performance depending on complementary VFMs and guidance timing.

4.1 Settings

  • On StyleBench, evaluation uses 2,920 style-content pairs, compares eight baselines with matching input modalities, and measures style, content, leakage, and aesthetic quality.The pairs come from 40 content images and 73 styles; each style uses one representative image, with content prompts as generation conditions.Source: §4.1 · p. 6

4.2 Comparisons

  • Qualitative comparisons show CLeaR with stronger style transfer and less visible leakage, while baselines exhibit style degradation, residual content, or dependence on unavailable style-name priors.The comparisons mark leakage in magenta and style degradation in cyan.Source: §4.2 · p. 7, Figure 3 · p. 7
  • CLeaR improves CSD by 8.3%, Style Loss by 64.9%, DINO by 60.7%, KID by 51.1%, and AI-Scoring by 8.05% over the strongest baseline, with the second-best CLIP score.It achieves the best performance on most metrics; its aesthetic score is slightly lower, possibly because inversion prioritizes disentanglement and structural preservation over aesthetic smoothing.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Ablation Studies

  • Changing the content description controls which reference objects are removed: undescribed objects can reappear in the anchor, while an empty description leaves content artifacts.Editing the natural-language description provides fine-grained disentanglement control without retraining.Source: §4.3 · p. 8, Figure 4 · p. 8
  • CLIP, DINO, and Inception mainly suppress leakage, while VGG and CSD-CLIP preserve fine-grained style patterns; increasing the ensemble size improves both SA and CL.The five-VFM ensemble achieves the best performance.Source: §4.3 · p. 8, Figure 5 · p. 9
  • Guidance is ineffective in the chaotic stage; in semantic and refinement stages, it improves SA while CL remains relatively stable, with the final 20% of refinement steps giving the best efficiency-performance trade-off.This setting is the final T/10 timesteps and is used by default.Source: §4.3 · p. 9, Figure 6 · p. 9

5 Conclusion

CLeaR addresses content leakage across feature separation, style representation, and diffusion generation. Its pipeline extracts orthogonal style targets, inverts them into a shared pixel-space anchor, then applies energy-guided calibration during generation.

5 Conclusion

  • CLeaR addresses the style-transfer dilemma across feature separation, style representation, and generation, with experiments showing improved style alignment and content leakage suppression.The paper also reports theoretical support for using multiple VFMs.Source: §5 · p. 9
  • CLeaR adds test-time computation through Ensemble Inversion and Energy-Guided Calibration, and remains slower than simpler feed-forward methods despite runtime comparable to several training-free baselines.Ambiguous content descriptions, degenerate anchors, and adapter misinterpretation of inversion noise can still lead to leakage and artifacts.Source: §5 · p. 10

A Overall Pipeline of CLeaR

  • The pipeline computes orthogonal style anchors for multiple VFMs, optimizes a pixel-space image to match them, then uses the resulting anchor and target content during diffusion generation.Energy guidance is applied only during the semantic stage, such as intermediate timesteps, to balance efficiency and style fidelity.Source: A Overall Pipeline of CLeaR · p. 14

B Proof of the Multi-Model Inversion Error Bound

The appendix analyzes ensemble inversion locally using a first-order regime. Under its stated assumptions, the leading-order bound also applies to the exact estimator.

B Proof of the Multi-Model Inversion Error Bound

  • The analysis assumes ensemble-average image-space bias cancellation and uniformly bounded local Jacobians.These conditions accompany the local first-order regime.Source: B Proof of the Multi-Model Inversion… · p. 15
  • The proof linearizes the objective, expands and differentiates its quadratic form, then decomposes bK into bias and centered fluctuation terms.The proposition is stated under Assumption 1 and Appendix Assumptions 2–4.Source: B Proof of the Multi-Model Inversion… · pp. 15–16
  • Under Assumption 2, the neglected remainder is lower-order locally, so the leading-order bound also applies to the exact estimator.The local regime requires differentiable feature maps near x⋆ and sufficiently small ∆.Source: B Proof of the Multi-Model Inversion… · pp. 15–16

C More Qualitative Comparison Results

Additional qualitative comparisons show CLeaR maintaining stylization without compromising content leakage or style fidelity across diverse examples.

C More Qualitative Comparison Results

  • Across diverse examples, CLeaR achieves clean stylization without compromising content leakage or style fidelity, while existing methods show one of those problems.Magenta boxes highlight leakage and cyan boxes highlight style degradation.Source: C More Qualitative Comparison Results · p. 18

D More Examples of Style Anchors

Additional extracted style anchors suppress content semantics while preserving visual style attributes across varied artistic styles.

D More Examples of Style Anchors

  • Across artistic styles, Ensemble Inversion anchors suppress objects, faces, and text while preserving color, texture, and brushstrokes.Each figure pair places the original reference on the left and the extracted anchor on the right.Source: D More Examples of Style Anchors · p. 18, Figure A2 · p. 18

E Comparisons on Held-Out VFMs and AI Evaluator

CLeaR is evaluated with held-out vision foundation models and independent AI evaluators to test generalization beyond its optimization models and reduce concern about closed-loop evaluation bias. It achieves the best or competitive performance across these evaluations.

E Comparisons on Held-Out VFMs and AI Evaluator

F Ablation on Style Categories

The analysis compares style-alignment (SA) and content-leakage (CL) scores across all 73 StyleBench styles. Scores differ between styles whose identity is mainly rendering and those tied to subject matter.

F Ablation on Style Categories

G Analysis on Failure Modes

The analysis identifies three representative CLeaR failure modes, attributing them to limitations in current pretrained modules rather than the disentanglement formulation. They involve incomplete content descriptions, degenerate style anchors, and generation artifacts.

G Analysis on Failure Modes

  • The authors frame the three representative failure modes as limitations of current pretrained modules, not the disentanglement formulation.Source: G Analysis on Failure Modes · p. 19
  • When styles are difficult to describe, incomplete content descriptions can leave residual content leakage despite the idealized potential of OSP.The text states that an ideal content description would allow OSP to remove all content-related components.Source: G Analysis on Failure Modes · p. 19
  • When style is tightly coupled to an object, Ensemble Inversion can produce an almost information-free anchor or one that copies the reference.This failure is frequently observed for Line Art, Stick Figure, and Icon styles.Source: G Analysis on Failure Modes · p. 20, Figure A3 · p. 20
  • An inverted anchor outside the adapter’s training distribution may be misread as semantic content, producing noisy lines and blobs in generated images.Source: G Analysis on Failure Modes · p. 20, Figure A4 · p. 20

Related papers

Loading 2609.38136v2…