PaperlayerView exact revision on arXiv

Source-linked AI summary

TexSliders: Diffusion-Based Texture Editing in CLIP Space

Julia Guerrero-Viu, Milos Hasan, Arthur Roullier, Midhun Harikumar, Yiwei Hu, Paul Guerrero, Diego Gutierrez, Belen Masia, Valentin Deschaintre

arXiv:2405.00672v1cs.GRcs.CV

TL;DR

Precise texture editing is challenging because attention-based editing methods are less suited to textures. TexSliders derives editing directions from text prompts in CLIP image-embedding space and reports better performance than other image-editing methods on textures. Its directions can also be computed for custom concepts, though edits may not fully change appearance or preserve identity.

  • Problem

    Attention-map editing methods that work for photographs are less informative in textures, making precise texture manipulation a challenge.

    Source: §1 · p. 1

  • Method

    TexSliders samples texture-prior image embeddings for two text prompts, computes their cluster-center direction, and selects dimensions to reduce identity variation.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    TexSliders is reported to be superior to other image-editing methods when applied to textures.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Custom editing directions can be specified with two unrestricted text prompts and computed in under two minutes.

    Source: §5 · p. 10

    Some edits may not significantly change texture appearance, and input texture identity is not always perfectly preserved.

    Source: §5 · p. 8

Abstract

from arXiv · show

Generative models have enabled intuitive image creation and manipulation using natural language. In particular, diffusion models have recently shown remarkable results for natural image editing. In this work, we propose to apply diffusion techniques to edit textures, a specific class of images that are an essential part of 3D content creation pipelines. We analyze existing editing methods and show that they are not directly applicable to textures, since their common underlying approach, manipulating attention maps, is unsuitable for the texture domain. To address this, we propose a novel approach that instead manipulates CLIP image embeddings to condition the diffusion generation. We define editing directions using simple text prompts (e.g., "aged wood" to "new wood") and map these to CLIP image embedding space using a texture prior, with a sampling-based approach that gives us identity-preserving directions in CLIP space. To further improve identity preservation, we project these directions to a CLIP subspace that minimizes identity variations resulting from entangled texture attributes. Our editing pipeline facilitates the creation of arbitrary sliders using natural language prompts only, with no ground-truth annotated data necessary.

1 INTRODUCTION

Precise texture editing remains challenging because attention maps that help edit photographs are less informative for textures. TexSliders instead derives editing directions from text prompts in CLIP image-embedding space, selecting dimensions to better preserve texture identity.

1 INTRODUCTION

  • Attention maps that guide photograph edits are less informative for textures, limiting the value of attention-based editing methods in this domain.The authors demonstrate this limitation in their evaluation.Source: §1 · pp. 1–2
  • The method targets sliders that preserve texture identity, avoid annotated training data, and require no changes to the original diffusion-model weights.Identity means retaining all input-texture attributes except the one being edited.Source: §1 · p. 2
  • Sampling image embeddings for two prompts and measuring their cluster-center difference yields an editing direction, while dimension selection reduces identity variation.The method selects dimensions using per-dimension intra- and inter-cluster statistical distances.Source: §1 · p. 2
  • Image-embedding-conditioned diffusion makes identity preservation easier than text-embedding-conditioned diffusion and guarantees tileable texture outputs.The authors attribute the difference to information lost when describing an image with text.Source: §1 · p. 2
  • TexSliders manipulates CLIP image embeddings for image-conditioned diffusion, deriving customizable texture-editing directions from natural-language prompts.The approach uses a texture prior and does not require ground-truth annotated data.Source: §1 · p. 2

2 RELATED WORK

Prior diffusion editing approaches use attention manipulation, fine-tuning, or optimization, while texture synthesis also includes procedural methods. TexSliders instead finds texture-editing directions in CLIP embedding space using text prompts.

Diffusion-Based Image Editing

  • Attention-map editing methods rely on semantic information often missing in textures, and their texture-editing performance degrades.Imagic fine-tunes the text embedding and diffusion-model weights, requiring lengthy tuning for each edit and input image.Source: §2 · p. 2
  • Unlike Concept Sliders’ diffusion-parameter directions, TexSliders finds edit directions in CLIP embedding space without fine-tuning or changing diffusion-model weights.Concept Sliders uses composable low-rank parameter updates.Source: §2 · p. 2

Finding Directions in Generative Models

  • Earlier diffusion work studied manageable latent directions through optimization, whereas TexSliders defines a direction with two text prompts and requires no optimization.The related work describes disentangled semantic directions as more challenging in diffusion models than in GANs.Source: §2 · p. 3

Texture Generation and Editing

  • Procedural texture methods expose parameter sliders but can be difficult to author because parameters may not match intuitive concepts and interactions can be complex.TexSliders targets similar control over existing textures using simple text prompts, without complex procedural representations.Source: §2 · p. 3

3 DIFFUSION-BASED TEXTURE EDITING

Texture editing methods based on attention maps do not capture texture structure, so the approach edits CLIP image embeddings instead. A texture diffusion prior maps prompt pairs to image-space directions, with dimension selection intended to preserve input identity as edits progress.

3.1 Preliminary Observations

  • Cross-attention maps capture structure for object-centered images but fail to represent texture structure and identity, motivating editing in CLIP image-embedding space.The method avoids cross-attention maps for texture editing.Source: §3.1 · p. 3

3.2 Diffusion Model

  • A texture-domain diffusion prior converts text embeddings into image embeddings for conditioning a pre-trained latent diffusion model without fine-tuning the entire model.The prior is trained separately on a texture subset, while the main model is trained on a larger image dataset.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 Computing Sliders for Texture Editing

  • Text prompts define an editing direction by sampling image embeddings for each prompt and taking the difference between the two embedding-cluster centroids.The experiments use 150 embeddings for each prompt.Source: §3.3 · p. 4
  • The method selects dimensions whose inter-cluster variability exceeds intra-cluster variability, zeroing the rest to reduce identity variation while changing the desired attribute.The comparison uses normalized vectors and a threshold typically set to 0.8 in the experiments.Source: §3.3 · p. 5
  • Adding a scaled editing direction to an input texture embedding creates the final conditioning embedding, with positive or negative scale values controlling edit intensity.The resulting embedding conditions the diffusion model to generate the edited texture.Source: §3.3 · p. 5
  • Users can define new sliders from two text prompts in about 2 minutes on a single A10G GPU in the unoptimized implementation.Source: §3.3 · p. 5

4 RESULTS

The evaluation covers generated and photographed textures, comparing editing quality, identity preservation, and the method’s generality across materials and editing directions.

4.1 Test Dataset

  • The test dataset contains 80 images—66 generated and 14 real—across seven material types, paired with original and target prompts.Generated examples use semantically valid material–attribute pairs and two random seeds; the real photographs include two per material type.Source: §4.1 · p. 5

4.2 Qualitative Evaluation

  • The method performs challenging edits across materials, including extrapolated edits, while preserving input identity during global texture changes.The qualitative evaluation includes weathering, metallicness, and touch edits.Source: §4.2 · p. 5, Figure 5 · p. 6
  • For “metal” to “rusty metal,” using one embedding or all 768 direction dimensions produces greater identity shift than selecting a subset of relevant dimensions.The selected-dimension approach edits while maintaining the texture’s original structure.Source: §4.2 · p. 5, Figure 6 · p. 7

4.3 Real Photographs

  • The method successfully edits photographed textures using their CLIP embeddings to reconstruct and condition editing, including changes to wetness and smoothness.The authors note that diffusion-model inversion could further improve reconstruction accuracy, but is outside the work’s scope.Source: §4.3 · p. 7, Figure 7 · p. 8

4.4 Comparison to the State of the Art

  • Against prior methods, the approach shows strong identity preservation and target-prompt adherence; it also outperforms them on real photographs and successfully changes pattern scale.SDEdit trades editing strength against identity preservation, while P2P and Pix2pix-zero show identity loss or artifacts in some cases.Source: §4.4 · p. 7
  • Our approach significantly outperforms previous work on both CLIP-Direction and CLIP-Im2Im across the full test dataset.The dataset includes generated images and photographs.Source: §4.4 · p. 7, Table 1 · p. 7

4.5 Generality

  • Directions transfer across material types with reasonable results and small identity loss; multiple directions can be combined, and image pairs can define edits when text prompts are insufficient.The image-pair example uses 20 texture images to form original and target clusters for a “zoomed in” to “zoomed out” direction.Source: §4.5 · p. 8

5 DISCUSSION AND LIMITATIONS

TexSliders is reported to outperform other image-editing methods on textures, while its identity preservation remains subject to limitations from CLIP, the diffusion model, and the editing direction. The discussion also identifies boundaries in how texture identity is defined and in extending the method beyond color textures.

5 DISCUSSION AND LIMITATIONS

  • CLIP and diffusion-model biases can weaken or derail edits, while residual noise in an editing direction may alter input texture identity.The paper gives vertical-pattern bias and changed cobblestone arrangement as examples.Source: §5 · p. 8, Figure 10 · p. 10
  • Extrapolating too far from the input can place embeddings outside the sampled distribution, where identity preservation is not guaranteed.The paper also notes that texture identity is not formally defined and leaves its in-depth study for future work.Source: §5 · p. 8
  • The work focuses on color textures; material-map adaptation is considered possible, but current material diffusion models use much less data, limiting expressivity.The authors also identify improved understanding of CLIP embedding spaces and user-interface integration as future work.Source: §5 · p. 8
  • TexSliders provides better identity preservation and realistic editing than the compared methods, and is reported superior to other image-editing methods on textures.The comparison includes Prompt-to-Prompt, Pix2pix-zero, and SDEdit.Source: §5 · p. 8, Figure 8 · p. 9
  • The figure presents generalization from brick-derived directions to wood and combines stone-scale and moss directions along a diagonal of edits.The full edit matrices are in the Supplemental Material.Source: Figure 9 · p. 9
  • Editing directions can be precomputed as sliders for attributes of any input texture, and new custom directions can be computed in under two minutes.The authors also suggest possible adaptation to vector-graphics-like images.Source: §5 · p. 10

Related papers

Loading 2405.00672v1…