Source-linked AI summary
Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, David Bau
TL;DR
Diffusion image generation lacks precise, low-interference control over continuous and difficult-to-describe visual attributes. Concept Sliders learn low-rank directions from text or image examples, enabling composable continuous editing; experiments report stronger targeted edits with lower interference, while also improving realism and correcting hand distortions. The method remains limited by residual edit interference and a trade-off between edit strength and structural coherence.
Problem
Text-driven diffusion editing provides limited precision for continuous attributes and cannot directly capture some visual concepts that are difficult to describe in words.
Method
Concept Sliders fine-tune low-rank adaptors using text or image examples to control target attributes while preserving selected concepts and reducing interference.
Results
The method supports textual, visual, and StyleGAN-derived sliders, composes over 50 sliders without degrading quality, and improves perceived realism while correcting hand distortions.
Takeaways & Limitations
Concept Sliders provide continuous, single-pass, modular control over textual and visual concepts, including edits that are difficult to express through prompts.
Takeaways & Limitations
Residual interference remains, while SDEdit improves structural coherence at the cost of edit intensity.
Abstract
from arXiv · showhide
We present a method to create interpretable concept sliders that enable precise control over attributes in image generations from diffusion models. Our approach identifies a low-rank parameter direction corresponding to one concept while minimizing interference with other attributes. A slider is created using a small set of prompts or sample images; thus slider directions can be created for either textual or visual concepts. Concept Sliders are plug-and-play: they can be composed efficiently and continuously modulated, enabling precise control over image generation. In quantitative experiments comparing to previous editing techniques, our sliders exhibit stronger targeted edits with lower interference. We showcase sliders for weather, age, styles, and expressions, as well as slider compositions. We show how sliders can transfer latents from StyleGAN for intuitive editing of visual concepts for which textual description is difficult. We also find that our method can help address persistent quality issues in Stable Diffusion XL including repair of object deformations and fixing distorted hands. Our code, data, and trained sliders are available at https://sliders.baulab.info/
1. Introduction
Concept Sliders address the limited precision and entanglement of text-driven diffusion editing by learning low-rank, interpretable controls for textual and visual concepts. They support generalized, composable editing and improve realism and hand quality without altering image content.
- Text prompts struggle to precisely modulate continuous attributes such as age and weather intensity, limiting creators’ control over generated images.
- Concept Sliders use low-rank model modifications to identify minimal concept subspaces for controlled, high-quality, disentangled editing.The low-rank constraint is described as vital for precision control.
- StyleGAN latent directions can be transferred into diffusion models to control complex visual concepts that lack textual descriptions.The transferred directions originate from StyleGAN’s style space trained on FFHQ faces and adapt to diverse image generation.
- Concept Sliders for realistic images and fixed hands significantly improve perceived realism without altering image content.The result is supported by a perceptual user study.
- More than 50 unique sliders can be composed without degrading output quality, bypassing standard prompt token limits.
- Unlike single-image post-hoc editors, Concept Sliders learn semantic changes from small sets of text or image pairs and apply them across the model.
3. Background
Diffusion models generate images by reversing a noise-adding process, while latent diffusion performs this process in a lower-dimensional autoencoder space. LoRA makes adaptation efficient by restricting weight updates to a low-rank subspace and merging them into the pretrained weights at inference.
- The forward diffusion process gradually adds noise to organized data until it becomes Gaussian noise, which the model then learns to reverse.
- Stable Diffusion and Stable Diffusion XL operate in a lower-dimensional latent space produced by a pretrained variational autoencoder.The latent is decoded through the VAE decoder to produce the final image.
- LoRA decomposes the fine-tuning weight update into low-rank matrices while keeping the pretrained weights fixed.
- The small rank r constrains the update to a low-dimensional subspace, reducing the number of trainable parameters.
- At inference, the LoRA update can be merged into the pretrained weights using a scaling factor α without overhead.
4. Method
Concept Sliders fine-tune LoRA adaptors to learn low-rank directions that increase or decrease targeted attributes while preserving selected concepts. They support text- or image-defined concepts, continuous strength control, and transfer from paired visual examples or StyleGAN latents.
- Text-based sliders: Concept Sliders fine-tune LoRA adaptors to increase attribute c+ and decrease attribute c− for a target concept ct.The method modifies the diffusion model’s likelihood of the two attributes while conditioning on ct.
- Text-based sliders: A guided score expresses concept editing through time-varying denoising predictions over partially denoised samples.The score is derived from gradients of the conditional log probability using a noising process and denoising prediction.
- Disentanglement: Preservation concepts constrain optimization to reduce entanglement with attributes such as race when editing age.The objective increases the desired attribute jointly with each preservation concept rather than optimizing the target attribute alone.
- Low-rank adaptation: Low-rank adaptation identifies a minimal concept subspace and produces more controlled, high-quality, disentangled edits than unconstrained fine-tuning.The LoRA modules are trained while pretrained weights remain fixed.
- Inference control: The LoRA scaling factor α continuously adjusts edit strength at inference without retraining.Increasing α strengthens the edit while preserving the trained slider direction.
- Visual sliders: Paired before/after images define visual sliders for concepts that are difficult to specify textually, including directions transferable from StyleGAN.The positive and negative LoRA directions align with the two visual states in the paired data.
5. Experiments
Experiments evaluate textual and visual sliders on Stable Diffusion XL, showing targeted editing across diverse concepts, fine-grained control from image pairs, StyleGAN-latent transfer, and composition of many sliders.
- Experimental setup: The evaluation primarily uses Stable Diffusion XL and also includes experiments on SD v1.4.Sliders are tested on diverse prompts, with inference using SDEdit to help preserve structure and semantics.
- Textual Concept Sliders: Across 30 text-based concepts, Concept Sliders achieve higher CLIP scores and lower LPIPS than the original while enabling precise control.Compared baselines require additional inference passes for each new concept and are more prone to concept entanglement.
- Textual Concept Sliders: Text-based sliders maintain image structure while enabling fine-grained editing of the specified concept.Table 1 characterizes the comparison using ∆CLIP for desired-edit efficacy and LPIPS distance from the original image.
- Visual Concept Sliders: Image-pair sliders control fine-grained attributes such as eyebrow shape and eye size, with optional text guidance focusing the edit spatially.Paired examples can provide stepwise control over the target attribute.
- Visual Concept Sliders: 2.75x average eye-area change is achieved across the slider range while providing more targeted editing than textual inversion and custom diffusion.The evaluation measures eye regions with FaceNet and a face parser, while LPIPS assesses similarity to the original distribution.
- StyleGAN transfer: StyleGAN-v3 style-space neurons transfer into diffusion sliders for structured editing of facial features that are difficult to describe in words.The transferred directions control features such as cheekbone structure, face width, and inter-ocular distance.
6. Concept Sliders to Improve Image Quality
Concept Sliders can steer Stable Diffusion XL toward more realistic outputs by repairing common distortions and fixing hands, while supporting compositional control over image attributes.
- Repair Slider: Stable Diffusion XL can produce warped or blurry objects despite containing latent parameter directions associated with higher-quality outputs.Concept Sliders identify low-rank directions that move generations away from these common distortions.
- Composed quality controls: Composing cooked and fine-dining sliders provides two-dimensional control over food images in both positive and negative directions.This demonstrates how multiple concept controls can be applied together to a generated image.
- Repair Slider: The repair slider targets distorted humans and pets, unnatural objects, blurry natural images, and inconsistent shapes.The slider is described as shifting images toward greater realism and correcting multiple classes of generation errors.
- Fixing Hands: Across 150 SDXL images, 62% were rated as having distorted hands, compared with 22% of 150 images generated with the hand-fixing slider.The user study used the same seeds and prompts for the original and slider conditions.
- Repair Slider: 80.39% of evaluated image pairs were judged more realistic with the repair slider than without it.The perceptual study compared 250 pairs of slider-adjusted and original SD images.
- Evaluation: FID scores disagreed with human realism judgments, while the opposite slider direction improved FID despite users preferring the realism-enhancing direction.This result highlights a mismatch between the automated metric and perceptual assessment in this experiment.
7. Ablations
Ablations examine disentanglement and low-rank adaptation as the two key components of Concept Sliders, finding that both support precise, low-interference editing.
- Disentanglement: The disentanglement objective reduces unintended gender or race changes while isolating the requested age edit.Table 3 measures interference as the percentage of samples with gender or race changes relative to the original images.
- Low-rank adaptation: LoRA adaptors improve edit precision in structure and edit direction while maintaining similar ∆CLIP concept strength across ablations.The ablation reports improvements in LPIPS and Interference for LoRA-based sliders.
8. Limitations
The disentanglement formulation reduces unwanted interference, but residual effects remain and require better selection of preserved directions. SDEdit improves structural coherence at the cost of edit intensity.
- Residual interference: Residual interference remains despite disentanglement, motivating more careful or automated selection of directions to preserve.Further study is needed to minimize interference while retaining edit fidelity.
- Disentanglement: The disentanglement objective helps avoid unintended race or gender changes when editing age.The low-rank constraint supports a more precise edit.
- SDEdit trade-off: SDEdit preserves image structure but can reduce edit intensity compared with inference-time editing.The authors characterize this as a trade-off between edit strength and fidelity to the original image.
9. Conclusion
Concept Sliders provide scalable, interpretable control through precise semantic directions, while supporting composition across many textual, visual, and GAN-defined concepts. Experiments demonstrate their versatility across models, concept types, latent transfer, and image-quality correction.
- Conclusion: Concept Sliders enable continuous, single-pass semantic control and simultaneous overlaying of many sliders for complex image manipulation.Their modular design is presented as a flexible alternative to text-driven, image-specific editing.
- Conclusion: Experiments on Stable Diffusion XL and Stable Diffusion 1.4 demonstrate textual, visual, and StyleGAN-latent sliders, including hand-distortion correction.A human study verifies the high quality of sliders that enhance and correct hand distortions.
- Availability: The source code, trained sliders, and datasets are available online for reproducing the results.The implementation is provided through the project website and GitHub repository.
- Disentanglement: Preserving multiple attribute directions helps find a disentangled edit direction rather than allowing single-prompt training to entangle unintended attributes.The formulation explicitly defines preservation directions during training.
- Inference trade-off: Increasing SDEdit timesteps preserves more spatial structure but lowers CLIP-based edit strength, with timesteps 750 - 850 reported as a favorable trade-off.This conclusion comes from an ablation over 2,500 images.
12. Textual Concepts Sliders
Textual Concept Sliders are evaluated for edit strength, spatial stability, attribute isolation, and stylistic variation across slider types and scales. Compared with prompt-based alternatives, they provide finer control with less interference, while retaining diverse outputs across customization settings.
- Evaluation: CLIP score change measures concept modification strength, while LPIPS distributions assess precise spatial manipulation across 15 sliders and 12 scales.Tighter LPIPS distributions indicate less spatial distortion across scales.
- SDEdit analysis: Higher SDEdit timesteps strengthen concept attributes per CLIP while increasing LPIPS-based spatial change.The figure compares the same slider scale as SDEdit time increases.
- Attribute isolation: Prompt-to-prompt and inference-time composition can change race or gender when editing age, whereas disentangled sliders show minimal interference.The Interference metric measures the percentage of samples with gender or race changes, and LPIPS indicates finer editing.
- Attribute isolation: Attribute-isolation analysis covers 15 slider types across 12 scales, including symmetric antonym edits and clipped negative ranges for harder-to-negate concepts.Some concepts, such as cluttered rooms, show higher LPIPS in ways described as intuitive.
- Customization: Concept Sliders produce more diverse outputs while learning new concepts, whereas customization methods can learn unintended hair and eye colors.The comparison concerns unintended concept learning during customization.
13. Visual Concept
Visual Concept Sliders learn concepts from image datasets while reducing irrelevant attribute correlations, and their composability supports multidimensional and progressively layered image control. The examples include paired food attributes and stacks of up to 10 sliders.
- Visual concept learning: Paired-image training isolates local attribute changes and avoids spurious correlations that can reduce diversity in customization baselines.On the large-eyes dataset, baseline outputs frequently include correlated blonde hair or blue eyes.
- Slider composition: Composing cooked and fine-dining sliders creates two-dimensional control over Thanksgiving food images.The sliders can be applied in positive and negative directions.
- Progressive composition: Progressively composing sliders produces semantic control over images, culminating in examples that stack 10 sliders.The sequence begins from a Stable Diffusion XL image and adds one slider at each step.
15. Editing Real Images
Concept Sliders can edit real images by first converting them into diffusion-model representations through null inversion, then applying sliders for precise attribute control.
- Editing Real Images: Null inversion finetunes the unconditional text embedding during classifier-free guidance to convert a real image into a diffusion-model-generated representation.This avoids manually engineering a prompt that closely reproduces the real image.
- Editing Real Images: Concept Sliders then enable precise control of attributes in the converted real image.The approach is illustrated in Figure 20.
- Editing Real Images: Slider compositions progressively add multiple sliders to an SDXL image for nuanced, complex control over attributes.Figure 19 shows composition of 10 sliders.
16. Sliders to Improve Image Quality
Concept Sliders improve diffusion-image quality by repairing distortions, especially malformed hands, while also enabling targeted edits to people, styles, weather, vehicles, and furniture.
- Quality Repair: 62% of 150 SDXL images had hand issues, compared with 22% of images generated using the “fix hands” slider.The evaluation used matched seeds and prompts, with majority-user ratings determining hand issues.
- Quality Repair: The “fix hands” slider targets misplaced or distorted fingers and incorrect finger counts in generated images.Users classified hand issues as misplaced or distorted fingers, incorrect finger number, or no issue.
- Quality Repair: The “repair” slider is evaluated in an A/B test comparing original images with same-seed slider outputs for fewer flaws or distortions.The test contains 300 image pairs with randomized left/right placement.
- Attribute Editing: Text-based sliders edit attributes such as curly hair, surprise, chubbiness, glasses, muscles, beards, and long hair while maintaining overall image structure.These examples cover facial, bodily, and hair attributes.
- Attribute Editing: Style sliders modify images toward Pixar, realistic details, clay, or sculpture while preserving the overall structure.The examples demonstrate multiple stylistic directions applied through text-based sliders.
- Attribute Editing: Weather, vehicle, and furniture sliders control concepts including tropical settings, winter snow, futuristic or damaged vehicles, and royal or modern furniture.The weather examples also include delightful and dark conditions.