Source-linked AI summary
ControlNeXt: Powerful and Efficient Control for Image and Video Generation
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, Jiaya Jia
TL;DR
Existing controllable-generation methods often add costly auxiliary components and face convergence or controllability limitations, particularly for video. ControlNeXt instead uses a compact architecture with selective finetuning and Cross Normalization, and experiments across image and video backbones demonstrate its effectiveness and generality.
Problem
Controllable generation methods commonly require substantial auxiliary computation and can suffer from slow or unstable training and weak control, especially for video.
Method
ControlNeXt uses a lightweight convolution module and selective finetuning of pretrained models, while Cross Normalization replaces Zero Convolution for large-model finetuning.
Results
Experiments across various image and video generation backbones demonstrate ControlNeXt’s effectiveness, generality, and broad compatibility.
Takeaways & Limitations
ControlNeXt provides a lightweight plug-and-play control method that integrates with LoRA weights for style modification without additional training.
Takeaways & Limitations
The architecture assumes pretrained models are sufficiently powerful and general to learn controllability directly from much smaller finetuning datasets.
Abstract
from arXiv · showhide
Diffusion models have demonstrated remarkable and robust abilities in both image and video generation. To achieve greater control over generated results, researchers introduce additional architectures, such as ControlNet, Adapters and ReferenceNet, to integrate conditioning controls. However, current controllable generation methods often require substantial additional computational resources, especially for video generation, and face challenges in training or exhibit weak control. In this paper, we propose ControlNeXt: a powerful and efficient method for controllable image and video generation. We first design a more straightforward and efficient architecture, replacing heavy additional branches with minimal additional cost compared to the base model. Such a concise structure also allows our method to seamlessly integrate with other LoRA weights, enabling style alteration without the need for additional training. As for training, we reduce up to 90% of learnable parameters compared to the alternatives. Furthermore, we propose another method called Cross Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and stable training convergence. We have conducted various experiments with different base models across images and videos, demonstrating the robustness of our method.
1. Introduction
ControlNeXt targets controllable image and video generation with a lightweight architecture that lets the pretrained model learn control directly. Cross Normalization replaces Zero Convolution to improve convergence, while selective finetuning and LoRA compatibility reduce training and deployment costs.
- Motivation: Existing controllable-generation methods add parallel branches or adapters, increasing computation and creating convergence, stability, and controllability challenges, especially for video.T2I-Adapter improves efficiency for images but can compromise controllability on video and fidelity-oriented low-level tasks.
- ControlNeXt: ControlNeXt injects control signals through a lightweight convolution module while selectively finetuning a small subset of pretrained parameters.Most base-model parameters remain frozen, limiting catastrophic forgetting and reducing training costs with minimal latency increase.
- Cross Normalization: Cross Normalization is introduced as an alternative to Zero Convolution, addressing distributional mismatch between control features and pretrained intermediate features.Zero initialization can stabilize training but slows convergence because newly added parameters initially receive poor gradients.
- Experiments and capabilities: ControlNeXt is evaluated across multiple image and video generative backbones, demonstrating broad compatibility and generality.Its lightweight design also supports plug-and-play integration with LoRA weights for style modification without additional training.
2. Related Work
Prior work develops diffusion-based image and video generation and introduces multiple strategies for adding controllable conditions. Related studies also examine latent-space efficiency and distribution alignment as routes to improved generation.
- Image and video diffusion models: Diffusion probability models generate images and videos by restoring data from Gaussian noise through learned noisy-data distributions.Their ability to fit complex data distributions has supported strong performance in image and video generation.
- Image and video diffusion models: Latent diffusion models use variational autoencoders to move diffusion into latent space, accelerating training and inference.Stable Diffusion is identified as a widely adopted latent image-generation model.
- Controllable generation: Controllable generation commonly uses textual conditions or adds controllability through models trained from scratch or adapters finetuned on pretrained models.Adapter-based approaches keep the original model parameters frozen while learning control components.
- Distribution alignment in diffusion models: Recent studies emphasize distribution alignment in diffusion models for addressing generation issues and improving image-to-image translation, inpainting, or sample quality.The cited approaches modify or align noise and intermediate representations with learned or reference distributions.
3. Method
ControlNeXt replaces heavy auxiliary control branches with a lightweight convolution module and selective fine-tuning of the pretrained base model. Cross Normalization aligns control and denoising features to support faster, more stable convergence.
- Motivation: ControlNet and adapter-based methods add computational costs, slow convergence, or weaken controllability, especially for video and complex visual tasks.These approaches typically keep the base model frozen and rely on auxiliary components to capture controls.
- Architecture: ControlNeXt removes heavy auxiliary components and fine-tunes a small subset of pretrained parameters while using compact convolution blocks to extract and align controls.Most pretrained modules remain frozen, reducing the additional architecture and learnable parameter cost.
- Architecture: The method injects controls after the first block through a residual connection rather than directly at the input, avoiding confusion with denoising features.Controls are added after Cross Normalization to further enhance training stability.
- Cross Normalization: Cross Normalization uses the main branch’s channel-wise mean and variance to normalize control features, aligning their distributions before integration.The control feature is produced by the lightweight convolution module, while the normalization statistics come from the denoising features.
- Cross Normalization: Cross Normalization serves as a bridge between the base model and control blocks, accelerating training and enabling the model to capture guiding information from the outset.The cited experiments report that it significantly alleviates sudden convergence.
4. Experiments
Experiments evaluate ControlNeXt across image and video backbones, tasks, conditions, efficiency measures, and training-free LoRA integration. The results indicate broad compatibility, support for multiple controls, faster convergence than ControlNet, and minimal inference overhead.
- Training Convergence: ControlNeXt starts converging after only a few hundred training steps, while ControlNet requires thousands of steps.The comparison is conducted with two types of controls and is reported as alleviating sudden convergence.
- Generality: ControlNeXt is evaluated across Stable Diffusion 1.5, Stable Diffusion XL, Stable Diffusion 3, Stable Video Diffusion, and Open-Sora-Plan backbones.Tasks include image generation, super-resolution, character animation, and video outpainting.
- Efficiency: The efficiency evaluation compares ControlNeXt primarily with ControlNet across backbone settings, with parameter statistics reported for total and learnable parameters.The parameter analysis counts only the UNet, excluding VAE and encoder components.
- Efficiency: ControlNeXt increases inference latency minimally relative to the pretrained base generation model.Inference-time comparisons consider one UNet and ControlNet inference step across various base models.
- Additional Studies: ControlNeXt integrates with multiple backbones and LoRA weights in a training-free manner, altering generated image quality and styles.The reported experiments use SD1.5, AnythingV3, and DreamShaper, including pose-conditioned generation from a simple text prompt.
- Generality: ControlNeXt supports mask, depth, pose, and Canny conditional controls across the reported experiments.The conditions are integrated into the main branch using lightweight control modules.
5. Conclusion
ControlNeXt targets controllable image and video generation with a compact architecture and Cross Normalization, reducing overhead while improving training convergence across backbones.
- ControlNeXt uses a compact architecture to minimize latency overhead and reduce trainable parameters.
- Cross Normalization improves the speed and stability of fine-tuning convergence for large pre-trained models.
- Experiments across image and video generation backbones demonstrate ControlNeXt’s effectiveness and generality.