PaperlayerView exact revision on arXiv

Source-linked AI summary

ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen

arXiv:2404.07987v4cs.CVcs.AIcs.LG

TL;DR

Existing controllable text-to-image methods struggle to align generated images with image-based conditions. ControlNet++ explicitly optimizes pixel-level cycle consistency using discriminative reward models and makes this optimization efficient through noise-perturbed inputs and single-step denoising. Experiments show comprehensive controllability improvements across various conditional controls, while the current method remains limited in supported conditions and does not yet address image quality and aesthetics.

  • Problem

    Existing controllable-generation methods still struggle to achieve precise and fine-grained alignment between generated images and input image conditions.

  • Method

    ControlNet++ uses pre-trained discriminative reward models to extract generated-image conditions and explicitly optimize pixel-level cycle consistency, with diffusion training loss retained alongside reward loss.

  • Results

    ControlNet++ comprehensively outperforms existing methods in controllability under various conditional controls.

  • Takeaways & Limitations

    Single-step denoising after adding noise to input images avoids the substantial time and memory costs of multi-step image sampling for reward fine-tuning.

  • Takeaways & Limitations

    The current method supports six control conditions but leaves expansion to conditions such as Human Pose and Scribbles for future work.

Abstract

from arXiv · show

To enhance the controllability of text-to-image diffusion models, existing efforts like ControlNet incorporated image-based conditional controls. In this paper, we reveal that existing methods still face significant challenges in generating images that align with the image conditional controls. To this end, we propose ControlNet++, a novel approach that improves controllable generation by explicitly optimizing pixel-level cycle consistency between generated images and conditional controls. Specifically, for an input conditional control, we use a pre-trained discriminative reward model to extract the corresponding condition of the generated images, and then optimize the consistency loss between the input conditional control and extracted condition. A straightforward implementation would be generating images from random noises and then calculating the consistency loss, but such an approach requires storing gradients for multiple sampling timesteps, leading to considerable time and memory costs. To address this, we introduce an efficient reward strategy that deliberately disturbs the input images by adding noise, and then uses the single-step denoised images for reward fine-tuning. This avoids the extensive costs associated with image sampling, allowing for more efficient reward fine-tuning. Extensive experiments show that ControlNet++ significantly improves controllability under various conditional controls. For example, it achieves improvements over ControlNet by 11.1% mIoU, 13.4% SSIM, and 7.6% RMSE, respectively, for segmentation mask, line-art edge, and depth conditions. All the code, models, demo and organized data have been open sourced on our Github Repo.

1 Introduction

Existing controllable-generation methods struggle to make generated images precisely match image-based conditions. ControlNet++ addresses this gap by explicitly optimizing pixel-level cycle consistency with discriminative reward models and an efficient single-step fine-tuning strategy.

  • Existing methods such as ControlNet and T2I-Adapter struggle to generate images accurately consistent with input image conditions.
  • ControlNet++ models controllable generation as image translation from conditional controls to generated images and back to extracted conditions.
  • The method uses pre-trained discriminative models to extract conditions from generated images and explicitly optimizes pixel-level cycle consistency with the inputs.
  • Direct reward optimization through multi-step sampling requires storing gradients at every timestep, causing substantial time and GPU-memory costs.
  • ControlNet++ disrupts input-image consistency with noise and uses single-step denoising for efficient reward fine-tuning.
  • ControlNet++ comprehensively outperforms existing methods under various conditional controls.

2 Related Work

Diffusion models enable text-to-image generation but remain limited in representing dense visual details through language alone. Related controllable-generation methods add trainable modules, prompt constraints, or multi-condition mechanisms, while visual reward models offer an objective alternative to subjective human-feedback optimization.

  • 2.1 Diffusion-based Generative Models: Latent Diffusion reduces diffusion-model training demands by mapping pixel-space processes into latent feature space.
  • 2.1 Diffusion-based Generative Models: Text-to-image diffusion models use cross-attention between UNet denoisers and text embeddings to facilitate image generation.
  • 2.1 Diffusion-based Generative Models: Language is a sparse semantic representation that is poorly suited to describing dense, low-semantic images, and detailed prompts remain challenging for existing models.
  • 2.2 Controllable Generation: ControlNet and T2I-Adapter introduce trainable modules for conditional control, while other methods use prompt engineering, cross-attention constraints, or multi-condition generation.
  • 2.3 Linguistic and Visual Reward Models: Reward models evaluate how generative outputs align with human expectations and provide quantified feedback for improving generation.
  • 2.3 Linguistic and Visual Reward Models: Unlike subjective human-preference optimization for global image quality, this work targets fine-grained, objective controllability using cost-effective AI feedback.

3 Method

ControlNet++ models controllable generation as image translation and explicitly optimizes cycle consistency between input conditions and generated-image conditions. It combines reward consistency with diffusion training while replacing costly full sampling with single-step denoising from deliberately noised training images.

  • Reward controllability: ControlNet++ uses pre-trained discriminative reward models to extract conditions from generated images and optimize their consistency with input controls.For segmentation masks, the consistency loss can be a per-pixel cross-entropy loss, with UperNet used as the reward model.
  • Reward controllability: The reward loss compares each input condition cv with the corresponding extracted condition ˆcv from the generated image.The generated image is obtained through the diffusion denoising process, and the metric and reward model depend on the visual condition.
  • Joint optimization: The total objective combines diffusion training loss with weighted reward loss to preserve the original image-generation capability.The two losses have different optimization goals, so the training loss remains part of the final objective.
  • Efficiency challenge: Full reward fine-tuning from random noise requires multiple denoising steps and storing gradients at every timestep, creating substantial time and GPU-memory costs.A 50-step DDIM process is estimated to require approximately 340GB of memory for a single sample, compared with approximately 6.8GB for one denoising step with stored gradients.
  • Efficient reward fine-tuning: The efficient strategy adds small noise to training images, disturbs their condition consistency, and predicts a denoised image in one step for reward fine-tuning.The noised image follows the standard diffusion forward process, and the single-step denoised image is directly used for the reward calculation.
  • Efficient reward fine-tuning: The noise timestep is restricted by t ≤ t_{thre}, because small noise can disturb consistency while large noise causes severe image distortion during direct prediction.The threshold t_{thre} determines whether a noised image is used for reward fine-tuning, avoiding the time and GPU-memory costs of multiple sampling steps.

4 Experiments

ControlNet++ is evaluated across multiple conditional controls for controllability, image quality, text alignment, qualitative consistency, and downstream usefulness. Results show stronger condition alignment while generally preserving image and text generation quality, with ablations clarifying the training-loss trade-off.

  • Controllability: 32.55 mIoU is achieved by ControlNet for segmentation-mask generation, compared with 50.7 mIoU on real data under the same evaluation.The gap illustrates the remaining difficulty of precise controlled generation.
  • Image quality: ControlNet++ generally exhibits superior FID values in most conditional-generation tasks without decreasing image quality relative to existing methods.FID is reported at 512×512 resolution using Clean-FID for fair comparisons.
  • Text alignment: ControlNet++ achieves comparable or superior CLIP-Score outcomes on several datasets while preserving text-to-image generation proficiency.CLIP-Score evaluates similarity between generated images and input text.
  • Downstream evaluation: A 1.19 mIoU gain appears when training DeepLabv3 on ControlNet++ images, while combining real data with generated data yields a +1.76 mIoU improvement.ControlNet-generated data added no improvement when augmenting real ground-truth data: 34.11 v.s. 34.08.
  • Qualitative comparison: Generated images from ControlNet++ maintain better consistency with input segmentation and depth conditions than prior methods in qualitative comparisons.Prior methods produced extraneous wall frames or inaccurate finger depth representations.
  • Ablation study: Pixel-level consistency loss alone causes severe distortion, whereas combining it with diffusion training loss improves controllability without affecting image quality.The original diffusion training process is identified as crucial for preserving quality and controllability.
  • Ablation study: Reward fine-tuning on a small timestep subset generalizes to larger unoptimized timesteps during sampling.The study attributes this to updating all ControlNet parameters despite using reward fine-tuning at only some timesteps.

5 Discussion

ControlNet struggles when text prompts are empty or conflict with image-based controls, while ControlNet++ generates content that better complies with those controls.

  • ControlNet++ generates images that comply with the input segmentation controls across varied text-prompt scenarios.The reported scenarios include no prompt, conflicting prompts, and perfect prompts.
  • Under empty or conflicting text prompts, ControlNet often fails to generate content matching the segmentation map.The failures are highlighted by red boxes in Fig. 9.

6 Conclusion

ControlNet++ targets imprecise conditional control by explicitly optimizing consistency between input conditions and generated images, using an efficient single-step reward strategy. Experiments across multiple conditional controls report improved controllability without compromising image quality or image-text alignment.

  • ControlNet++ explicitly optimizes consistency between input conditions and generated images with a pre-trained discriminative reward model.The method uses a cycle-consistency formulation rather than relying only on latent-space denoising to achieve controllability.
  • The efficient reward strategy adds noise to input images and uses single-step denoising to calculate consistency loss.This avoids the time and GPU-memory costs of sampling from random Gaussian noise across multiple timesteps.
  • Across multiple conditional controls, ControlNet++ significantly improves controllability without compromising image quality or image-text alignment.

1 Overview of Supplementary

The supplementary material covers implementation details, an equation proof, additional experiments and analyses, broader-impact discussion, limitations, and more visualizations.

  • Section 2 provides implementation details for all experiments.
  • Section 3 contains a proof for Eq.(7) in the main paper.
  • More experiments and analysis: Section 4 presents more experiments and analysis, including conditioning-scale effects and human evaluation.The human evaluation covers controllability, text guidance, and image quality.
  • Sections 5 and 6 discuss broader impact and limitation, followed by additional visualization results.

2 Implementation Details

The implementation details describe dataset construction, reward and evaluation model selection, condition-specific loss weights, training settings, and the relationship between timestep noise and predicted-image precision.

  • Dataset construction: The training dataset follows ControlNet’s construction principles and uses accurately labeled conditional datasets where available.The supplied passage specifically introduces this choice for segmentation conditions.
  • Model selection: The authors use weaker reward models and stronger evaluation models to distinguish reward-model alignment from genuine controllability improvements.This practice is feasible for segmentation and depth but challenging for HED, Canny, and LineArt Edge.
  • Training settings: λ is 0.5, 0.5, 1.0, 1.0, and 10 for segmentation mask, depth, HED edge, Canny edge, and LineArt edge conditions, respectively.
  • Training settings: All experiments fine-tune the pre-trained ControlNet with batch size 256 and learning rate 1e-5, followed by 10k reward-fine-tuning iterations using the same settings.
  • Reward estimation: At smaller timesteps, corresponding to smaller noise ϵ_t, the predicted image x′_0 is estimated more precisely.

3 Proof of Equation 7 in the Main Paper

The diffusion forward process constructs noisy inputs by adding Gaussian noise to original data, while algebraic inversion estimates the original image for reward evaluation. This estimate is most accurate at relatively small timesteps and varies substantially across timesteps.

  • The forward diffusion process gradually adds noise to input data x_0 through a Markovian chain q(x_t|x_0).
  • The noisy sample x_t is expressed using the scaled original image and Gaussian noise ε ~ N(0, I).
  • Algebraic manipulation of the noise-prediction equation yields an estimate x′_0 of the original image from a noisy sample and predicted noise.
  • The predicted original image x′_0 can be used as input for the reward consistency loss at any denoising timestep.
  • The approximation has smaller error when timestep t is relatively small, while predicted x′_0 differs significantly across timesteps.

4 More Experiments

Additional experiments examine conditioning strength, visual consistency, and human preferences on ADE20K segmentation data. They show that increasing image-condition weight alone does not resolve controllability-quality trade-offs, while ControlNet++ improves controllability without sacrificing image quality or text guidance.

  • Increasing image-condition weight in ControlNet and T2I-Adapter cannot improve controllability while ensuring image quality.The experiments vary the relative weighting of image-condition and text features.
  • ControlNet++ offers better controllability without sacrificing image quality or text guidance in a 20-user human evaluation.The evaluation uses one condition and compares images according to three criteria.
  • The ADE20K validation experiment reports a win-rate comparison for segmentation-conditioned generation.

5 Broader Impact and Limitation

The paper identifies broader directions beyond its current controllability focus, including expanding supported conditions, improving aesthetics, and jointly optimizing the control network and reward model.

  • ControlNet++ currently improves controllability under six control conditions, while future work targets Human Pose and Scribbles.
  • Future work aims to use human feedback to optimize controllability and aesthetics together.
  • The authors propose joint optimization with more controllable images so the control network and reward model can co-evolve.

6 More Visualization

Supplementary visualizations present ControlNet++ outputs across multiple image conditional controls, including edge, depth, and segmentation conditions.

  • Supplementary visualizations show ControlNet++ results for LineArt Edge and Depth Map conditions.
  • Additional visualizations cover Hed Edge and Canny Edge conditions.
  • The supplementary figures also visualize ControlNet++ generation under Segmentation Mask conditions.
Loading 2404.07987v4…