Source-linked AI summary
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, Jingfeng Zhang
TL;DR
Adapting image diffusion models can be resource-intensive, and prior efficient tuning work has not thoroughly analyzed which factors improve image quality. SCEdit edits U-Net skip connections with lightweight tuners, extending them to conditioned synthesis; the paper reports efficiency and performance advantages across text-to-image and controllable generation.
Problem
Full fine-tuning can be impractical under limited training data and computational resources, while existing efficient tuning methods can still consume substantial resources.
Source: §1 · p. 2Method
SCEdit edits U-Net skip-connection features with lightweight SC-Tuners, extending them as CSC-Tuners to incorporate single or multiple conditions.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
The framework supports single- and multi-condition generation while limiting backward propagation to the decoupled U-Net decoder.
Source: §1 · p. 2, §5 · p. 8Because tuning freezes the backbone, transfer ability depends substantially on the upstream model’s performance.
Source: D. Limitations and societal impacts · p. 14
Abstract
from arXiv · showhide
Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable image synthesis. Recent research has introduced tuning methods that make subtle adjustments to the original models, yielding promising results in specific adaptations of foundational generative diffusion models. Rather than modifying the main backbone of the diffusion model, we delve into the role of skip connection in U-Net and reveal that hierarchical features aggregating long-distance information across encoder and decoder make a significant impact on the content and quality of image generation. Based on the observation, we propose an efficient generative tuning framework, dubbed SCEdit, which integrates and edits Skip Connection using a lightweight tuning module named SC-Tuner. Furthermore, the proposed framework allows for straightforward extension to controllable image synthesis by injecting different conditions with Controllable SC-Tuner, simplifying and unifying the network design for multi-condition inputs. Our SCEdit substantially reduces training parameters, memory usage, and computational expense due to its lightweight tuners, with backward propagation only passing to the decoder blocks. Extensive experiments conducted on text-to-image generation and controllable image synthesis tasks demonstrate the superiority of our method in terms of efficiency and performance. Project page: \url{https://scedit.github.io/}
1. Introduction
Full fine-tuning and some existing efficient tuning methods can be costly, while skip connections materially affect U-Net output detail. SCEdit edits these connections with lightweight tuners, and is reported to improve generation performance while reducing resource use.
1. Introduction
- Full fine-tuning can be impractical with limited data and computational resources, while methods such as LoRA can require gradients across the full U-Net.The authors position skip-connection modules as a way to decouple the encoder from backpropagation and reduce computation and memory requirements.Source: §1 · p. 2
- Removing more skip connections progressively lowers final-decoder output variance and reduces structural detail in decoder feature maps, with further deterioration when all are removed.The reported comparison removes connections from half to all; reduced detail is noted in the 6th and 11th decoder blocks.Source: §1 · p. 2, Figure 3 · p. 2
- Figure 2 compares performance and efficiency across text-to-image and controllable synthesis; marker shape distinguishes task type and area represents relative parameter count.Circular markers denote text-to-image tasks and pentagonal markers denote controllable synthesis.Source: Figure 2 · p. 2
- SCEdit places lightweight SC-Tuners in U-Net skip connections and extends them to CSC-Tuners that accept single or multiple conditions for controllable synthesis.The framework decouples the encoder so backward propagation passes only through decoder blocks.Source: §1 · p. 2
- SCEdit outperforms existing methods on COCO2017 FID and qualitative results, while reducing training memory by 52%; controllable synthesis uses 7.9% of ControlNet’s parameters and 30% less memory.The paper also reports faster transfer and high-quality results in few-shot fine-tuning.Source: §1 · p. 3
2. Related work
Related work covers efficient adaptation of image diffusion models and research on U-Net skip connections. SCEdit builds on these lines by analyzing and editing skip connections for image generation.
2. Related work
- Prior efficient-tuning methods adapt pretrained diffusion models with lightweight adjustments, but few works thoroughly analyze which tuning factors improve generative image quality.Examples include LoRA, U-Tuning, Res-Tuning, and MAM-Adapter.Source: §2 · p. 3
- Prior skip-connection research studies training stability and generation quality; SCEdit analyzes their role in generation and edits them with tuning modules.Some approaches rescale skip connections to address signal-variance doubling and U-Net oscillations.Source: §2 · p. 3
3. Method
SCEdit edits U-Net skip-connection features with lightweight tuners, extending the approach to conditional image synthesis. Its framework routes tuner outputs into decoder blocks while decoupling encoder blocks for efficient training.
3.1. Preliminaries
- The denoising U-Net decodes multiscale encoder features through corresponding skip connections, which complement high-frequency information.The preliminaries also describe diffusion as adding noise over a T-step Markov chain and denoising as estimating that noise with a parameterized network.Source: §3.1 · p. 3
3.2. Tuner modules
- SC-Tuner edits latent features within each skip connection, using a tuner operation and residual connection; its operation uses tunable projection matrices and GELU.The framework can use different tuner types and scales to modify skip-connection features.Source: §3.2 · p. 4
- CSC-Tuner incorporates single or multiple conditions into skip-connection tuning, weighting condition branches before combining them with the original features.Conditions may be trained jointly or combined at inference after single-condition training; examples include edges, depth, segmentation, pose, color, and masked images.Source: §3.2 · p. 4
3.3. SCEdit framework
- SCEdit feeds skip connections and conditions to SC-Tuner or CSC-Tuner, then concatenates their outputs with original feature maps for the corresponding decoder blocks.For controllable synthesis, condition features are encoded through a cascade dense convolution network before entering the skip-connection branch.Source: §3.3 · p. 5
4. Experiments
SCEdit is evaluated on text-to-image generation and controllable synthesis, with reported gains in quality and efficiency across both tasks. The experiments also examine few-shot style transfer, compositional generation, and control generalization.
4.1. Experimental setups
- The evaluation covers COCO2017 and customized-style text-to-image tuning plus LAION-based controllable synthesis, measuring parameters, memory, training speed, FID, and image quality.SC-Tuner is used for text-to-image tasks and CSC-Tuner for controllable synthesis; the setups include several condition types and SD v1.5 or SD v2.1 pretrained models.Source: §4.1 · p. 5
4.2. Text-to-image generation
- 0.33 FID performance gain over full fine-tuning accompanies 2.29% of its parameters and nearly 22% less training time.Against LoRA/r=64, SCEdit also reduces training memory by 52.1%; few-shot transfer on six styles with 30 image-text pairs each more accurately captures style while maintaining text alignment.Source: §4.2 · p. 6
4.3. Controllable image synthesis
- SCEdit uses 7.9% of ControlNet’s parameters, reduces memory consumption by 30%, accelerates training, and achieves a lower FID score.Qualitative comparisons report better fidelity and realism, including lake reflections and car texture; results span diverse conditions, scenes, and styles.Source: Figure 8 · p. 7, §4.3 · p. 8
- SCEdit combines separately trained edge and color models for training-free scene-level translation on unpaired conditions, and interpolates CSC-Tuner embeddings for blended effects.The examples include seasonal transitions and variations balanced across multiple elements.Source: §4.3 · p. 8, Figure 9 · p. 8
- An edge-based model supports sketch-to-image generation and an inpainting-based model supports outpainting, despite no task-specific adaptation for those conditions.The reported examples lack adaptation to hand-drawn sketches and specialized adjustment of outpainting mask patterns.Source: §4.3 · p. 8, Figure 10 · p. 8
5. Conclusion
SCEdit edits U-Net skip connections through lightweight SC-Tuner and extends the approach to conditional generation with CSC-Tuner. It confines backward propagation to the decoupled decoder and supports single- and multi-condition generation.
5. Conclusion
- SCEdit uses SC-Tuner to edit skip connections and CSC-Tuner to handle diverse conditional inputs.The framework is described as lightweight and plug-and-play.Source: §5 · p. 8
- Backward propagation occurs exclusively in the decoupled U-Net decoder, supporting the method’s stated efficiency.Source: §5 · p. 8
- The framework supports arbitrary single- and multi-condition generation and is reported to perform strongly.Source: §5 · p. 8
A.1. Dataset description
The experiments use COCO, customized-style, and LAION datasets for different generation settings; a separate appendix table summarizes model hyperparameters.
A.1. Dataset description
- COCO2017 Captions provides 118,287 training images and 591,753 captions for text-to-image fine-tuning, while the customized-style dataset has 30 training images across styles.LAION is used for controllable image synthesis.Source: §A.1 · p. 11
A.2. Hyperparameters
- The appendix provides an overview of trained-model hyperparameters organized by task in Table 4.Source: §A.2 · p. 11
A.3. Architectures design
SCEdit edits skip connections through SC-Tuner for text-to-image generation and CSC-Tuner for controllable generation. The section describes condition preparation and SC-Tuner design trade-offs.
A.3. Architectures design
- SCEdit edits skip connections using SC-Tuner for text-to-image generation and CSC-Tuner for controllable generation.The architectures are designed to transfer to similarly designed modules.Source: §A.3 · p. 11
A.4. Conditions for generation
- Condition inputs include edge, depth, boundary, segmentation, pose, and color maps, plus inpainting; inpainting uses masks and cutouts together.Conditions follow implementations from ControlNet and T2I-Adapter; pose training uses 162,338 images, and the tasks otherwise share LAION-ART and hyperparameters.Source: §A.4 · pp. 11–12
B.1. SC-Tuner structure
- Reducing SC-Tuner width lowers parameters but does not substantially reduce memory or improve FID; removing skip layers degrades performance.The tuner can also use convolutional or independent residual networks, with linear layers offering some advantages.Source: §B.1 · pp. 12–13
B.2. CSC-Tuner structure
CSC-Tuner ablations examine condition-encoder kernels, skip connections, and tuner structure. Larger kernels and more data improve qualitative detail, while omitted skip connections and reduced channel dimensions have drawbacks.
B.2. CSC-Tuner structure
- Larger condition-encoder kernels reduce FID but add parameters; omitting skip connections raises FID, and a dual convolution with dimension reduction improves FID.Qualitatively, larger kernels yield richer detail, more training data adds realistic effects, and the dual-convolution variant controls conditions poorly; omitting skip connections loses image content.Source: §B.2 · p. 13, Figure 11 · p. 13
C.1. Additional qualitative comparison
- Figure 12 compares controllable generation using canny edges, depth, and semantic segmentation against ControlNet, T2I-Adapter, ControlLoRA, and ControlNet-XS.The boxed regions are enlarged for comparison.Source: §C.1 · p. 14
C.2. Additional qualitative results
Additional qualitative results show controllable generation across varied conditions, including multiple controls derived from one source image.
C.2. Additional qualitative results
- Figures 13–16 show results using different controls extracted from one image and cover edge, depth, boundary, segmentation, pose, color, outpainting, and inpainting.The figures include canny edge and depth examples, boundary and segmentation maps, pose keypoints, color maps, and masked-image tasks.Source: §C.2 · p. 14
D. Limitations and societal impacts
The method freezes the pretrained backbone, so transfer depends substantially on the upstream model's performance. The authors also warn that high-risk data could produce misleading outcomes.
D. Limitations and societal impacts
- Because the backbone is frozen during tuning, transfer ability depends substantially on the pretrained upstream model's performance.This is the stated boundary on transfer ability.Source: D. Limitations and societal impacts · p. 14
- Using high-risk data maliciously could potentially generate misleading outcomes, motivating ethical care in deployment.The authors specifically warn about propagation of harmful bias or false information.Source: D. Limitations and societal impacts · p. 14