Local Content-Style Control for Diffusion-based Image Stylization

1 Introduction

• Image stylization transforms an image to mimic a specific artistic medium or art style.Source: §1 · p. 1
• LDMs resynthesize images globally, so neither structure nor appearance stays pinned to input pixels, while professional workflows demand region-by-region control.Stylization negotiates preserving underlying structure and re-rendering appearance as artistic marks.Source: §1 · p. 1
• Existing ControlNet and IP-Adapter conditioning leaves the strength of structural adherence and image-prompt style uniform across the image.ControlNet specifies what structure to follow per location, but not how strongly to enforce it.Source: §1 · p. 1
• The method lifts ControlNet strength and per-block IP-Adapter scale from scalars to spatial maps, providing local content and style control through disjoint pathways.The two controls are validated as local and axis-specific despite global attention mixing.Source: §1 · p. 1

Spatial diffusion control

• Spatial diffusion methods often spatialize what is conditioned, while SmartControl spatializes ControlNet scale and practitioner techniques spatialize IP-Adapter scale for style alone.The paper identifies coupling spatial content strength with spatial style scale as its contribution, rather than regional control itself.Source: §2 · p. 2

Image-prompt stylization

• Image-prompt stylization uses IP-Adapter image features, with prior work separating style and content by attention blocks or globally across a forward pass.The paper builds on image-prompt conditioning and a ControlNet Tile branch used for stylization.Source: §2 · p. 2

Diffusion image editing

• Mask-, stroke-, and instruction-guided diffusion editing targets content but does not expose per-location control of image-style strength.The paper contrasts this with maps that control both content and style strength by location.Source: §2 · p. 2

Level-of-abstraction control

• Earlier abstraction control varied image abstraction locally using gaze, saliency, depth, or parameter maps, with later neural work using perceptual factors and guided style transfer.The paper presents its maps as a generative counterpart that trades content preservation against style transfer per region.Source: §2 · p. 2

3.1 Pipeline and Modulation Points

• The pipeline combines ControlNet content structure and IP-Adapter image-prompt style in a latent-diffusion stylizer driven by a Tile control image.The formulation depends on these two pathways, not on a particular backbone, sampler, or distillation.Source: §3.1 · p. 2

3.2 Content Axis: Spatial ControlNet

• The content map replaces scalar ControlNet strength at latent resolution and is broadcast across channels; a uniform map reproduces the scalar setting.ControlNet combines the noisy-latent feature map with the control embedding scaled by the conditioning strength.Source: §3.2 · p. 2

3.3 Style Axis: Spatial IP-Adapter

• The style map replaces each IP-Adapter block’s scalar image scale, broadcasting over channels without changing the attention computation; a uniform map reproduces the scalar setting.The decoupled image-attention term is added to text attention, and a zero image scale recovers the base model.Source: §3.3 · pp. 2–3

3.4 Resolution Semantics

• Pixel-resolution maps are anti-alias downsampled at modulation points, so content modulation propagates across multiple scales while style modulation remains coarser.The paper states that structure benefits from fine spatial control, while spatial style assignment is a low-frequency semantic choice.Source: §3.4 · p. 3

Implementation

• The implementation uses a four-step distilled SDXL-Lightning pipeline converted for Apple Neural Engine, with map inputs adding negligible latency and memory over the scalar pipeline.At 1024^2 output resolution, maps downsample to latent 128^2 for ControlNet and 32^2 or 64^2 for injected attention blocks.Source: §4 · p. 3

The 2×2 retouching vocabulary

• The 2×2 vocabulary combines content lock or relaxation with style strength to preserve structure, weaken stylization, preserve identity, or give the prompt more weight.Content-lock can displace some stylization because the Tile input is the content image; raising the style scale inside the region can offset this.Source: §4 · p. 3

Evaluation

• All conditions shared the same noise, and all 11 IP-Adapter blocks carried style, testing locality under the stated worst-case setting.The evaluation used objects and scenes from DIV2K.Source: §4 · p. 3
• 0.13–0.24×: far-field mean spatial LPIPS beyond 0.25 image widths from R stayed below the inside-R response, stylization magnitude, and a seed re-roll.Across 300 content-style-seed triples, the reported specificity probes found content-lock structure-dominant; style-relax strengthened texture with dose, left palette flat, and left structure unchanged or slightly restored.Source: Table 1 · p. 3, §4 · p. 3

Use cases

• Sweeping content strength and style scale over a fixed region moves it gradually from photographic to fully stylized, while depth or segmentation priors can generate control maps.The paper contrasts this with neural style transfer, which can restyle given content but cannot regenerate a region through the content pathway.Source: §4 · p. 3

Spatial control

• Style localizes more coarsely than content because SDXL caps style modulation at 64^2 latent resolution.The 8× pixel-to-latent downsampling also leaves very small objects hard to control.Source: §5 · p. 3

Sampling

• The four-step distilled sampler used in the experiments resolves intermediate strengths only coarsely.Source: §5 · p. 3

Color behavior

• Relaxing style can reintroduce color into near-monochromatic styles such as pen-and-ink, either from the source or through hallucination.Source: §5 · p. 3

• Spatial maps for two ControlNet and IP-Adapter conditioning weights provide local, axis-specific content and style control in one coherent generation pass without retraining.The conclusion describes region-by-region retouching and notes that per-pixel priors can author maps automatically.Source: §6 · p. 4