PaperlayerView exact revision on arXiv

Source-linked AI summary

SAEdit: Token-level control for continuous image editing via Sparse AutoEncoder

Ronen Kamenetsky, Sara Dorfman, Daniel Garibi, Roni Paiss, Or Patashnik, Daniel Cohen-Or

arXiv:2510.05081v1cs.GRcs.AIcs.CV

TL;DR

Text prompts struggle to provide both disentangled edits and smooth control over edit strength. SAEdit manipulates text embeddings along SAE-derived sparse directions; experiments demonstrate continuous, highly disentangled editing across backbones and real images.

  • Problem

    Text prompts struggle to provide smooth intensity adjustment and can unintentionally alter other image elements, limiting continuous, disentangled editing.

    Source: §1 · p. 1

  • Method

    SAEdit compares sparse representations of prompts that differ by an edit and manipulates text embeddings along relevant SAE-derived directions.

    Source: §1 · p. 2

  • Results

    Experiments demonstrate continuous, highly disentangled edits across two image-synthesis backbones without model-specific training, with real-image application through inversion.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Editing is decoupled from rendering: compatible text-to-image backbones can render the modified text embeddings.

    Source: §6 · p. 9

    For strongly out-of-distribution edits, disentanglement can fail because of biases in the underlying text-to-image model.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Large-scale text-to-image diffusion models have become the backbone of modern image editing, yet text prompts alone do not offer adequate control over the editing process. Two properties are especially desirable: disentanglement, where changing one attribute does not unintentionally alter others, and continuous control, where the strength of an edit can be smoothly adjusted. We introduce a method for disentangled and continuous editing through token-level manipulation of text embeddings. The edits are applied by manipulating the embeddings along carefully chosen directions, which control the strength of the target attribute. To identify such directions, we employ a Sparse Autoencoder (SAE), whose sparse latent space exposes semantically isolated dimensions. Our method operates directly on text embeddings without modifying the diffusion process, making it model agnostic and broadly applicable to various image synthesis backbones. Experiments show that it enables intuitive and efficient manipulations with continuous control across diverse attributes and domains.

1. Introduction

Text prompts make smooth, disentangled editing difficult because their discrete, holistic influence can cause unintended changes. SAEdit uses sparse autoencoder directions to manipulate text embeddings at token level, enabling continuous edits across models and real images.

1. Introduction

  • Text prompts struggle to provide continuous intensity adjustment and can cause unintended changes; discrete descriptions such as “a slight smile” and “a wide grin” do not smoothly vary smile intensity.Disentanglement means changing one attribute without unintentionally affecting others.Source: §1 · pp. 1–2
  • General training-free representation manipulation can struggle with disentanglement, while task-specific edit modules can produce high-quality results but require a separate training pipeline for each modification.These limitations motivate a general approach that combines continuous control and disentanglement.Source: §1 · p. 2
  • Comparing sparse representations of prompts that differ by an edit identifies relevant entries for a sparse, edit-specific direction that can be scaled to control attribute magnitude while preserving other image content.The SAE exposes directions that are difficult to identify directly in raw embedding space.Source: §1 · p. 2
  • Editing changes text embeddings while leaving denoising untouched, allowing compatible text-to-image backbones to render edits without additional training or fine-tuning.Compatibility requires a shared text encoder.Source: §1 · p. 2
  • Experiments demonstrate continuous, highly disentangled edits across two image-synthesis backbones without model-specific training, with magnitude-controlled text edits and real-image application through inversion.The method supports intuitive edits from simple text commands.Source: §1 · p. 2

2. Related Work

Diffusion-based real-image editing must balance preservation of original attributes with controlled text-guided changes. Existing approaches include denoising manipulation, partial noise schedules, and methods for continuous semantic control that trade off scalability, disentanglement, or direct text guidance.

Image Editing with Diffusion Models

  • Real-image editing balances preservation of original image attributes with controlled text-guided changes, using denoising feature injection or partial noise schedules and requiring inversion to find reconstructing initial noise.The passage lists inversion techniques as a key requirement for applying these methods to real images.Source: §2 · p. 2

Continuous Image Editing with Diffusion Models

  • Task-specific optimization can yield high-fidelity, disentangled continuous edits but requires a separate costly process for each attribute, while general-purpose training-free directions often lack precision and disentanglement.The passage also notes that SliderSpace explores latent variations but is not designed for direct text-guided editing.Source: §2 · p. 3

3. Preliminary - Sparse AutoEncoders

Sparse autoencoders learn high-dimensional latent representations intended to be interpretable and disentangled. Sparsity encourages dictionary-like features, which have been used to identify semantic concepts and steer model behavior.

3. Preliminary - Sparse AutoEncoders

  • An SAE typically uses an encoder and a linear decoder and is trained with reconstruction and sparsity objectives.The sparse regularization encourages a dictionary-like representation in which a small set of active features often corresponds to a semantic attribute.Source: §3 · p. 3
  • SAE features have been used to uncover semantic features in language models and to steer model behavior by manipulating internal activations.The passage gives features active for mentions of entities such as “US presidents” as an example.Source: §3 · p. 3
  • Initial applications of SAEs to diffusion models have focused on interpretability and concept unlearning.Source: §3 · p. 3

4. Method

The method trains an SAE on frozen text-encoder embeddings and edits token representations along sparse semantic directions. Scaling a direction controls attribute intensity, while an exponential injection schedule increases its influence over diffusion steps.

4. Method

  • The method manipulates frozen text embeddings in SAE latent space, then decodes edited representations for compatible text-to-image models.The SAE serves as a pluggable module; the diffusion model renders the edited semantic instructions.Source: §4 · p. 3

4.1. SAE Training

  • The SAE is trained on individual frozen T5 token embeddings with reconstruction and sparsity losses, controlling the target number of active latent entries.The embeddings are collected from the encoder’s final output, which conditions the diffusion transformer.Source: §4.1 · pp. 3–4, Figure 3 · p. 4

4.2. Obtaining an edit direction

  • Source and target prompts are max-pooled after SAE encoding; their feature ratios identify edit-correlated entries used to form a sparse direction.Directions from multiple prompt pairs are aggregated by selecting the leading singular vector of their matrix.Source: §4.2 · pp. 4–5

4.3. Applying the edit direction

  • Adding a scaled edit direction to the selected source token’s sparse representation, then decoding it, provides continuous control over attribute intensity.A scale of zero recovers the original embedding, and increasing the scale strengthens the visual effect.Source: §4.3 · p. 5

4.4. Injection Schedule

  • An exponential injection schedule increases edit strength over diffusion steps, applying it gently early and more strongly during later refinement.The schedule is designed to preserve global structure while enabling fine-grained modifications.Source: §4.4 · p. 5

5. Experiments

Experiments assess SAEdit’s continuous, disentangled edits across attributes, models, comparisons, and real-image editing. They also examine the contributions of its components and the limits imposed by base-model biases.

5.1. Qualitative Results

  • Qualitative results show continuous, localized human edits and control over object attributes; the same directions produce consistent, faithful edits with SD3.5.Examples include changing one person’s age while leaving others and the background untouched, and modifying object season or shape.Source: §5.1 · p. 6, Figure 6 · p. 6, Figure 8 · p. 6

5.2. Ablation

  • Aggregating prompt-pair directions strengthens the edit but can alter fine details; the exponential injection schedule preserves those details while retaining the semantic change.The ablation illustrates detail preservation with the necklace and hair color.Source: Figure 9 · p. 7, §5.2 · p. 7

5.3. Comparisons

  • Across varying edit intensities, SAEdit outperforms all compared approaches, including task-specific methods, while metrics progress smoothly with edit magnitude.The comparison evaluates image preservation and prompt fidelity.Source: Figure 10 · p. 7, §5.3 · pp. 7–8
  • On the custom evaluation set, 63 source images yield 432 edit scenarios and at least 1,296 images per method; evaluation uses LPIPS and a VQA-based prompt-adherence score.Each scenario is generated at 3–5 magnitude levels.Source: §5.3 · p. 7
  • Users significantly preferred SAEdit across image preservation, prompt alignment, and overall quality in the pairwise study.The study collected 390 pairwise comparison responses.Source: Table 1 · p. 8, §5.3 · p. 8
  • In Flux qualitative comparisons, SAEdit preserves subject details and changes age more naturally and gradually than competing methods, including ConceptSliders and Flux Kontext.Flux Kontext prompts were manually optimized for the comparison.Source: Figure 11 · p. 8, §5.3 · p. 8
  • For real photographs, Uni-Inv Flow inversion followed by text-embedding manipulation supports continuous edits while preserving identity and background details.Demonstrated edits include expressions and age.Source: §5.3 · p. 8, Figure 12 · p. 9

5.4. Limitations

  • Some complex edits benefit from manually selecting sparse-direction entries, while strongly out-of-distribution edits can break disentanglement under base-model biases.Examples include a beard edit changing perceived gender and a green-dog edit producing unnatural texture.Source: §5.4 · p. 9, Figure 13 · p. 9

6. Conclusions

SAEdit uses a Sparse Autoencoder on text embeddings to isolate semantic attributes and support fine-grained, continuous editing. Because editing is decoupled from rendering, compatible text-to-image backbones can serve as renderers.

6. Conclusions

  • SAEdit uses a Sparse Autoencoder on text embeddings to create a sparse representation with isolated semantic attributes.The isolated attributes support disentangled edits that preserve a subject’s core identity.Source: §6 · p. 9
  • Token-level manipulation provides fine-grained, continuous control over an attribute’s magnitude.The conclusion presents this as a way to address the open challenge of precise, fine-grained image-editing control.Source: §6 · p. 9
  • Editing modifies only text embeddings, allowing any compatible text-to-image backbone to act as the renderer.The authors also demonstrate using SAEs for image generation, yielding fine-grained editing capabilities.Source: §6 · p. 9

A. Implementation Details

The implementation trains an SAE on 12 million text prompts and derives edit directions from prompt pairs. It uses a 65,536-dimensional latent space and an exponential injection parameter tied to edit scale.

A. Implementation Details

  • The SAE training set combines 2M DiffusionDB captions and 10M HumanCaption-10M captions, totaling 12M prompts and approximately 800M filtered text tokens.The method uses the T5-XXL text encoder, also used by Flux.dev and Stable Diffusion 3.5.Source: A. Implementation Details · p. 12
  • The SAE uses a 65,536-dimensional latent space with 300 active entries per token and is trained for 200,000 steps at a 0.003 learning rate.Each edit direction is derived from n = 100 source-target prompt pairs; the exponential injection parameter is τ = 15 · ω.Source: A. Implementation Details · p. 12

B.1. Additional Qualitative Results

The additional qualitative results show continuous edits, compositional control, and localization across subjects, while the benchmark details describe how the evaluation images and edits were assembled.

B.1. Additional Qualitative Results

  • A single set of four directions—smile, angry, surprised, and old—generalizes across diverse source images, subjects, scenes, and identities.The same direction is applied to each image in the corresponding figure column.Source: §B.1 · p. 12
  • Smile strength and glasses can be varied independently while identity and background details remain consistent across the grid.The edits are shown on the horizontal and vertical axes, respectively.Source: §B.1 · p. 12
  • Applying laugh to a woman and old to a man in one scene confines each edit to its intended subject without changing the background or other attributes.The results preserve non-targeted attributes across the grid.Source: §B.1 · p. 12
  • Additional figures show continuous editing results on both human and non-human subjects.The cited figures are Figures 22 and 18, respectively.Source: §B.1 · p. 12

B.2. Benchmark Details

  • The comparative benchmark uses 63 source images from 21 prompts, with 3 random seeds per prompt and 6–8 semantic edits per image.The prompts were generated with an LLM; edit counts varied with applicability to the subject.Source: §B.2 · p. 12

B.3. Quantitative Ablation

The ablation reports that prompt aggregation improves alignment and the exponential injection schedule improves preservation at intermediate edit intensities. Qualitative comparisons show gradual, identity-preserving edits, while the Flux Kontext baselines lack native continuous scaling.

B.3. Quantitative Ablation

  • The full method retains the high VQA score from aggregating N prompts while improving LPIPS scores at all intermediate intensity levels.The ablation compares a single-prompt direction, aggregation without the injection schedule, and the full method with the exponential schedule; the authors describe the full method as balancing edit accuracy and preservation.Source: §B.3 · p. 12

B.4. Flux Kontext Baseline

  • Flux Kontext lacks native continuous edit scaling, so the evaluation uses prompt-level light, medium, and extreme instructions or varies CFG with a fixed medium instruction.The first proxy uses LLM-generated prompts; the second varies Classifier-Free Guidance scale.Source: §B.4 · p. 12
  • The same four learned directions—smile, angry, surprised, and old—are applied across diverse source images to illustrate their generality.The figure shows each direction applied to different source images.Source: Figure 14 · p. 13

B.5. Qualitative Comparisons (Continued)

  • The qualitative comparison covers FluxSpace, Concept-Sliders, and two Flux Kontext variants on the Flux model.The Flux Kontext prompts were manually optimized for the comparison because default outputs were often suboptimal.Source: §B.5 · p. 13
  • Against competing methods on Flux, SAEdit makes clear, consistent edits where competitors fail or cause identity changes; its edits are gradual and identity-preserving.The comparisons include FluxSpace, Concept-Sliders, and two Flux Kontext variants.Source: §B.5 · p. 13
  • For beard edits, SAEdit produces gradual, natural-looking changes, while baselines show abrupt transitions, unnatural appearances, or identity shifts.The comparison also reports that most baselines produce non-gradual changes or shift identity in the other example.Source: §B.5 · p. 13

B.6. User study

The appendix details the user-study protocol and supplemental demonstrations, then describes SAE sparsity techniques and additional manuscript materials.

B.6. User study

  • The user study sampled 20 edit scenarios and compared three intensity levels against a competing method on preservation, graduality, prompt alignment, and overall preference.The appendix also lists attribute-scale descriptions for comparison with Flux Kontext and prompts used for quantitative evaluation and the study.Source: §B.6 · pp. 13–14, Table 2 · p. 14, Table 3 · p. 16
  • Figure 15 varies smile strength horizontally and adds glasses vertically to show two independently controlled edits on one subject.The caption describes the grid results as smooth and accurate.Source: Figure 15 · p. 14
  • Figure 16 compares method versions and marks the top-right as better, but its caption gives no quantitative outcome.Source: Figure 16 · p. 14
  • Figure 17 combines a laugh edit targeted to the woman with an old-age edit targeted to the man, with each intended to avoid the other subject and background.The caption characterizes the results as localized and disentangled.Source: Figure 17 · p. 14

C. Sparse AutoEncoders - Continue

  • For SAE sparsity, BatchTopK keeps the strongest B×K entries across a training batch, then uses a calibrated global threshold at inference; an auxiliary loss can revive dead latents.Dead latents are entries that stop activating and can degrade reconstruction performance.Source: C. Sparse AutoEncoders - Continue · pp. 14–15
  • Figure 18 shows continuous edits on non-human subjects involving seasonal changes, color, and object shape.Source: Figure 18 · p. 15
  • Matryoshka SAEs learn nested dictionary subsets and minimize reconstruction losses across levels, encouraging important features to appear earlier in the dictionary.The reconstruction loss at each level uses only the first m entries.Source: C. Sparse AutoEncoders - Continue · p. 15

D. LLM Usage Statement

  • Additional figures compare editing methods and show text-based sliders; the authors state that an LLM helped improve manuscript grammar, spelling, and clarity, with the authors responsible for the final content.Figures 20 and 21 captions say the proposed method better preserves identity and produces more disentangled results than competing approaches.Source: D. LLM Usage Statement · p. 15, Figure 20 · p. 17, Figure 21 · p. 18

Related papers

Loading 2510.05081v1…