Source-linked AI summary
LISA: Likelihood Score Alignment for Visual-condition Controllable Generation
Yanghao Wang, Hongxu Chen, Jiazhen Liu, Zhenqi He, Rui Liu, Zhen Wang, Long Chen
TL;DR
The dual-branch paradigm leaves the side network’s likelihood-score role and training efficiency underexplored. LISA explicitly aligns side-network features with an approximated likelihood score, consistently accelerating convergence and improving synthetic quality across visual-condition generation tasks.
Problem
The standard dual-branch objective supervises only final predictions, leaving the side network’s likelihood-score role and training efficiency underexplored.
Method
LISA regularizes intermediate side-network features by aligning them with an approximated likelihood score alongside standard generative training.
Results
Across image and video conditions, architectures, and diffusion or flow models, LISA consistently accelerates convergence and improves perceptual quality and condition fidelity.
Takeaways & Limitations
LISA provides an effective and extensible solution for visual-condition controllable generation, with reported potential for compositional control.
Takeaways & Limitations
LISA is formulated for paired joint data consisting of visual conditions and corresponding clean samples.
Abstract
from arXiv · showhide
The prevalent dual-branch paradigm, i.e., training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score Alignment (LISA), an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder's output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard diffusion loss and our regularization loss. Experiments across various image/video tasks, architectures, and diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network's features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost.
1 INTRODUCTION
The paper reframes dual-branch visual-condition generation as combining an unconditional score from the frozen main network with a likelihood-score-like control signal from the trainable side network. It proposes LISA, which regularizes side-network features toward an approximated likelihood score and consistently improves convergence and synthesis quality across tasks and models.
- Dual-branch paradigm: The dual-branch paradigm freezes a pretrained diffusion or flow-matching main network while training a side network to encode conditions and inject intermediate features for control.The side network receives the condition input and integrates its features into the main network’s forward process.
- Score-based interpretation: The frozen main network provides an unconditional score for perceptual quality, while the side network learns condition-dependent control corresponding implicitly to a likelihood score.This role decomposition motivates explicit regularization of the side network’s intended control signal.
- LISA: LISA explicitly aligns the side network with an approximated likelihood score by regularizing its intermediate output.The method targets the training challenge arising from the side network’s lack of explicit role regularization.
- LISA: A lightweight decoder transforms hooked side-network features into score space, and the distance to the likelihood-score target is added as a regularization loss during generative training.The decoder is usually around 0.1% the size of the side network and contains only a few layers.
- Experiments: Experiments across image and video conditions, diffusion models, architectures, and tasks show consistent gains in training convergence and perceptual and condition-fidelity quality.Evaluated conditions include pose maps, depth maps, low-resolution images, segmentation maps, and pose videos.
2 PRELIMINARIES
Diffusion and flow models learn to transport a prior distribution pT to the target data distribution p0, using forward and reversed stochastic processes or an equivalent ODE. Because the unconditional score ∇x_t log p_t(x_t) is unknown, a parameterized network sθ is trained to predict it for sampling.
- Diffusion and Flow Matching: Diffusion and flow models learn a transport process from a prior distribution pT, such as a Gaussian, to the target data distribution p0.A forward diffusion process maps p0 to pT, while generation reverses this process.
- Stochastic Processes: The forward stochastic differential equation induces marginal distributions p_t(x_t), and its reverse describes generation from pT to p0.The reverse process can be represented by a reversed SDE and converted into an ODE with the same marginal distributions.
- Score Matching: Sampling synthetic data requires solving either the reverse SDE or its equivalent ODE, but both depend on the unknown unconditional score ∇x_t log p_t(x_t).The unconditional score is the gradient of the log marginal density at time t.
- Score Matching: A parameterized network sθ is trained with conditional score-matching targets and then replaces the unknown unconditional score during SDE or ODE sampling.The conditional score ∇x_t log p_t(x_t|x_0) is tractable because the forward SDE has closed-form conditional solutions.
3 LISA: LIKELIHOOD SCORE ALIGNMENT
LISA interprets the dual-branch conditional score as an unconditional score from the frozen main network plus a likelihood-score correction from the trainable side network. It constructs a sample-wise approximated likelihood-score target and aligns selected side features to it through a lightweight decoder during training.
- Approximated Likelihood Score Construction: LISA estimates the likelihood score by subtracting the unconditional main-network score from a denoising target whose expectation equals the conditional score.The main network is additionally forwarded without condition injection to obtain the unconditional score estimate.
- Dual-branch Decomposition: The conditional score decomposes into an unconditional score and a likelihood score, matching the main network’s prior prediction with the side network’s residual correction.The frozen main network approximates the unconditional score, while the side network supplies the correction toward the conditional score.
- Dual-branch Decomposition: Standard final-prediction supervision leaves the side network’s likelihood-score role implicit, motivating explicit feature alignment.The standard objective supervises only the final prediction rather than directly constraining the side-network features.
- Likelihood Score Alignment: A lightweight decoder maps a selected pre-integration side feature into the score latent space, where it is aligned with the approximated likelihood-score target.The decoder uses convolution, activation, and upsampling layers, and stop-gradient is applied in the regularization loss.
- Likelihood Score Alignment: During training, the frozen main network remains fixed while the side network and decoder are jointly optimized with standard diffusion loss and LISA regularization.The auxiliary decoder is discarded during inference, so the alignment adds no inference-time decoder computation.
4 EXPERIMENTS
Experiments show that LISA improves condition following, training efficiency, and visual quality across image, flow/DiT, video, and compositional generation tasks. Ablations and overhead analyses indicate that these gains require little additional computation and support practical deployment.
- Image Generation Results: LISA consistently improves condition following across four conditional image-generation tasks and can achieve better performance with fewer training iterations.In pose-conditioned generation, early training PCK improves from 19.38 to 83.02 relative to ControlNet.
- Image Generation Results: Qualitative comparisons show that LISA follows conditions more accurately while producing natural images with fewer artifacts.In one pose example, ControlNet inverted the person front-to-back, whereas LISA produced the correct pose.
- Ablation Studies: Alignment depth 5 achieves the best pose consistency, with PCK of 89.90%, while depths 2 and 8 improve PCK to 88.03% and 88.06%.These settings maintain comparable FID and CLIP; the alignment weight is fixed at λ = 0.2.
- Ablation Studies: λ = 0.2 provides the best overall balance, whereas λ = 0.1 yields PCK of 86.19% and λ = 0.5 yields FID of 56.34 with PCK of 87.83%.The results indicate that weak alignment is insufficient, while overly strong alignment can hurt structural matching.
- Computational Overhead: LISA adds about 0.1% parameters and 0.2s per training iteration, uses the same 21G GPU memory, and has identical inference cost to ControlNet.Model size increases from 364.2M to 364.6M, while training time increases from 2.1s to 2.3s per iteration; the decoder is dropped during inference.
- Generalization: LISA generalizes beyond U-Net and VP-SDE to DiT and OT-FM, reducing FID from 32.08 to 31.87 and increasing mIoU from 20.81% to 22.64% at 1K iterations.At 5K iterations, LISA further improves FID, CLIP, and mIoU over ControlNet; in video, it reduces FVD from 10.57 to 7.85 and raises PCK from 30.22% to 57.00% at 5K iterations.
- Compositional Generation: LISA also strengthens compositional generation, with explicit role decomposition and likelihood-score alignment producing more disentangled and composable condition representations.Both quantitative and qualitative evidence indicate improved multi-condition controllability over naive ControlNet.
5 RELATED WORK
Prior work enables visual-condition control through multimodal or side-network designs, but training cost can limit adaptation to new condition types. Other studies add regularization or pretrained visual encoders to diffusion and flow-matching objectives to improve training efficiency and final performance.
- Conditional Visual Generation: Composer trains a network from scratch for multimodal conditional control, but its training cost limits efficient extension to new condition types.It naturally supports conditional control across visual modalities.
- Conditional Visual Generation: Visual-modality conditions provide spatial-structure guidance for generation alongside class and text conditions.The passage frames visual conditions as an additional modality for controllable generation.
- Training Diffusion Models with Regularizations: Regularized diffusion and flow-matching methods add auxiliary training losses beyond regression of targets such as noise, score, or velocity to accelerate convergence.The vanilla objectives regress the target as the main loss, while additional regularization is introduced on top.
- Training Diffusion Models with Regularizations: Several studies use pretrained semantic visual encoders to improve diffusion-model efficiency and final performance.The cited studies include Yu et al. (2024), Pernias et al. (2024), and Li et al. (2024b).
6 CONCLUSION
The paper studies dual-branch visual-condition controllable generation from a score-based role decomposition and proposes LISA to align side-network middle features with a constructed likelihood score. This simple additional objective accelerates training and improves perceptual quality and condition fidelity.
- 6 CONCLUSION: LISA aligns the side network’s middle feature with a constructed likelihood score in the dual-branch paradigm.The method is motivated by decomposing the roles of the main and side networks from a score perspective.
- 6 CONCLUSION: The method adds a simple extra realization objective to the training process.
- 6 CONCLUSION: LISA significantly accelerates training and bootstraps better synthetic results in perceptual quality and condition fidelity.