Source-linked AI summary
TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control
Minkyoung Cho, Ruben Ohana, Christian Jacobsen, Adityan Jothi, Min-Hung Chen, Z. Morley Mao, Ethem Can
TL;DR
Fixed conditioning pathways do not vary with the changing demands of denoising, from coarse structure to fine detail. TC-LoRA uses a hypernetwork to generate time- and condition-specific weight adapters; the authors report improved spatial-condition adherence, including lower TransferBench errors than baseline.
Problem
Controllable diffusion models use fixed conditioning pathways even though denoising stages have different needs, from establishing coarse structure to refining fine detail.
Source: §1 · p. 1Method
TC-LoRA uses a hypernetwork to generate low-rank adapters conditioned on diffusion timestep and spatial input for pretrained diffusion weights.
Source: §3 · p. 3Results
11.7% lower NMSE and 3.4% lower si-MSE than baseline on TransferBench; OpenImages also shows markedly lower si-MSE.
Source: §4 · p. 4Takeaways & Limitations
TC-LoRA provides dynamic spatial control by regenerating LoRA weights at each denoising step, tailoring parameters to evolving generation needs.
Source: §4 · p. 5Extending TC-LoRA to text-to-video remains future work, with temporal consistency across frames and per-frame spatial adherence identified as a challenge.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Current controllable diffusion models typically rely on fixed architectures that modify intermediate activations to inject guidance conditioned on a new modality. This approach uses a static conditioning strategy for a dynamic, multi-stage denoising process, limiting the model's ability to adapt its response as the generation evolves from coarse structure to fine detail. We introduce TC-LoRA (Temporally Modulated Conditional LoRA), a new paradigm that enables dynamic, context-aware control by conditioning the model's weights directly. Our framework uses a hypernetwork to generate LoRA adapters on-the-fly, tailoring weight modifications for the frozen backbone at each diffusion step based on time and the user's condition. This mechanism enables the model to learn and execute an explicit, adaptive strategy for applying conditional guidance throughout the entire generation process. Through experiments on various data domains, we demonstrate that this dynamic, parametric control significantly enhances generative fidelity and adherence to spatial conditions compared to static, activation-based methods. TC-LoRA establishes an alternative approach in which the model's conditioning strategy is modified through a deeper functional adaptation of its weights, allowing control to align with the dynamic demands of the task and generative stage.
1 Introduction
The introduction argues that fixed conditioning pathways cannot adapt to the different demands of early and late denoising. TC-LoRA instead dynamically conditions model weights, and the authors report stronger adherence to depth controls than ControlNet-style baselines.
1 Introduction
- Fixed conditioning applies the same operation throughout denoising, although coarse structure matters early and fine detail later.The introduction argues that model weights should vary with diffusion time and the conditioning input.Source: §1 · p. 1
- TC-LoRA uses a hypernetwork to generate weight adapters conditioned on both time and the control signal.The introduction presents this as dynamic weight conditioning rather than static, frozen weights.Source: §1 · p. 1
- For depth-map conditioning, the authors report stronger control-signal adherence than ControlNet-style baselines.The introduction describes depth maps as constraining the generated image to respect depth.Source: §1 · p. 2
2 Related Work
This section reviews diffusion and spatial control, then contrasts fixed-weight conditioning and static LoRA adapters with time-varying adaptation approaches.
2 Related Work
- Diffusion models generate samples by progressively removing noise, while latent diffusion performs denoising on smaller autoencoder representations to reduce operating cost.The reverse process is conditioned on an input such as text or an image.Source: §2 · p. 2
- ControlNet-style models inject spatial features through auxiliary branches but keep network weights fixed throughout generation.The conditioning signals are inputs processed through static computational pathways.Source: §2 · p. 2
- Standard LoRA uses a fixed low-rank matrix update; time-varying variants scale its magnitude while leaving the underlying adapters functionally static.The update is represented as the product of matrices B and A, added to frozen pretrained weights.Source: §2 · p. 2
3 TC-LoRA: Temporally Modulated Conditional LoRA
TC-LoRA addresses the challenge of efficiently learning dynamic weights for pretrained diffusion models. A shared hypernetwork generates timestep- and condition-specific low-rank adapters, enabling weight-based adaptation during denoising.
3 TC-LoRA: Temporally Modulated Conditional LoRA
- Unlike activation injection into a fixed network, dynamic weight adaptation can alter the model's computational mechanisms across generation stages.The section motivates this distinction as allowing qualitatively different processing strategies at different stages.Source: §3 · p. 2
- TC-LoRA adds dynamic low-rank updates to selected pretrained weight groups as a function of timestep and conditioning signal.The updated weights take the form W′i = Wi + B(i, t, y)A(i, t, y).Source: §3 · p. 3
- A single hypernetwork shared across layers generates the complete A and B adapter parameters from the weight-group index, timestep, and condition.The low-rank matrices are generated on-the-fly rather than trained directly.Source: §3 · p. 3
- By modifying processing weights, TC-LoRA reshapes the diffusion model's function so conditioning can constrain generation beyond influencing activations.The figure depicts a text prompt and input condition guiding generation from initial noise, with hypernetwork-generated adapters modifying frozen base weights.Source: §3 · p. 3, Figure 1 · p. 3
4 Experimental Results
TC-LoRA is evaluated for depth-map alignment on OpenImages and TransferBench against a ControlNet-style baseline, with training limited to MS-COCO. It improves reported spatial-condition metrics and qualitative adherence, while extension to video remains future work.
4 Experimental Results
- Evaluation uses the Cosmos-Predict1 base model and compares TC-LoRA with Cosmos-Transfer1, a ControlNet-style baseline using the same frozen base model.The study evaluates depth-map alignment on OpenImages and TransferBench; TC-LoRA uses 251M trainable parameters versus 900M for the baseline.Source: §4 · pp. 3–4
- On TransferBench, TC-LoRA reduces NMSE by 11.7% and si-MSE by 3.4% versus the baseline.The reported generalization follows training exclusively on MS-COCO; OpenImages also shows markedly lower si-MSE, though no value is supplied here.Source: §4 · p. 4, Table 2 · p. 4
- Qualitative comparisons report better depth-condition adherence, including finer dog-pose details and pedestrian silhouettes than the baseline captures.Figure 2 compares Cosmos-Transfer1 with Cosmos-Predict1 + TC-LoRA.Source: §4 · p. 4, Figure 2 · p. 4
- TC-LoRA regenerates LoRA weights at each denoising step to adapt from coarse structure formation to fine-detail refinement; video extension is proposed as future work.The stated video challenge is maintaining temporal consistency while adhering to per-frame spatial conditions.Source: §4 · p. 5
A Hypernetwork Architecture
A single hypernetwork generates layer-specific LoRA adapters from a context vector combining timestep, user condition, layer identity, and layer type. Its multi-scale residual design uses skip connections and zero-initializes the B-matrix output layer.
A Hypernetwork Architecture
- The hypernetwork conditions adapter generation on timestep, user input condition, target-layer identity, and layer type.The context vector combines these four information types for specialized parameters across contexts.Source: A Hypernetwork Architecture · p. 7
- The condition is embedded as a 1024-dimensional vector, time as a 64-dimensional sinusoidal embedding, and layer properties as a 128-dimensional embedding.The layer encoder represents layer depth and type, including attention and query/key/value distinctions.Source: A Hypernetwork Architecture · p. 7
- Multi-scale residual blocks and multi-range skip connections combine contextual features, while zero-initializing the B-matrix output makes initial adaptation have no effect.Adapters attach to linear projections in all self-attention and cross-attention blocks; Figure 3 presents the hypernetwork design.Source: A Hypernetwork Architecture · p. 7, Figure 3 · p. 7
B Training Objective
TC-LoRA uses the standard diffusion noise-prediction objective to train its hypernetwork. The learned adaptations vary with timestep and conditioning while the base weights remain frozen.
B Training Objective
- The training loss is expected squared error between sampled noise and the denoiser’s noise prediction on a noisy latent.The noisy latent is derived from a clean latent, and the model includes the frozen base weights dynamically adapted by the trainable hypernetwork.Source: B Training Objective · p. 7
- The hypernetwork learns weight adaptations conditioned on timestep and spatial input alongside text conditioning, across timestep-condition combinations.The stated objective is to minimize reconstruction error over these combinations.Source: B Training Objective · pp. 7–8
C.1 Benchmark and Dataset Details
The evaluation uses MS-COCO for adapter training and two 600-example benchmarks for testing. OpenImages is sampled across three scene topics, while TransferBench targets robotics, driving, and everyday-life domains.
C.1 Benchmark and Dataset Details
- All adapters are trained on 120k MS-COCO images, with no training on the evaluation benchmark datasets.Source: §C.1 · p. 8
- OpenImages contains 600 diverse samples, balanced across nature, urban, and indoor topics using classifier-guided topic selection and CLIP-based diversity sampling.Prompts were generated for the selected images using Florence 2-L.Source: §C.1 · p. 8
- TransferBench contains 600 examples adapted from a text-to-video benchmark for single-frame generation, spanning robotics, driving, and everyday-life scenes.The evaluation generates the first frame of each example video.Source: §C.1 · p. 8
C.2 Evaluation Metric Details
Generated-image depth is compared with the input depth using si-MSE and NMSE, where lower values indicate better alignment. The comparison uses Cosmos-Transfer1 as a baseline and reports TC-LoRA’s trainable-parameter count.
C.2 Evaluation Metric Details
- Generated-image depth is compared with the input depth using si-MSE and NMSE, and lower values indicate better alignment.si-MSE measures structural and shape-related errors invariant to global scale shifts; NMSE quantifies relative prediction error.Source: §C.2 · p. 8
C.3 Baseline Details
- Cosmos-Transfer1 is the baseline, using ControlNet with Cosmos-Predict1; the comparison attaches TC-LoRA adapters to the same base model.Source: §C.3 · p. 8
- TC-LoRA has 251M trainable hypernetwork parameters versus 900M in Cosmos-Transfer1, demonstrating memory efficiency in post-training and deployment.Source: §C.3 · p. 8
C.4 Detailed Qualitative Analysis
Qualitative comparisons show TC-LoRA preserves depth-conditioned details better than the baseline and progressively aligns outputs with the condition during post-training.
C.4 Detailed Qualitative Analysis
- The qualitative comparisons visually substantiate the quantitative metrics.Source: §C.4 · p. 8
- Across dog and street scenes, TC-LoRA better preserves depth-conditioned pose, texture, and pedestrian placement than the baseline.The examples include a curled tail and surrounding path and grass texture, as well as pedestrian placement and silhouettes.Source: §C.4 · p. 8
- The unconditioned base model produces plausible but random images, while condition-guided structure improves progressively during post-training.Source: §C.4 · p. 9
- At 10k iterations general composition emerges, while at 150k the output is high-fidelity and structurally consistent with the spatial condition.Source: §C.4 · p. 9, Figure 4 · p. 9
D On the Distinction Between Activation Conditioning and Weight Conditioning
The section argues that an input-dependent activation addition cannot be represented by a single static subsequent weight update. TC-LoRA instead generates context-dependent weight modifications before they process the input, enabling a time-varying processing strategy.
D On the Distinction Between Activation Conditioning and Weight Conditioning
- A non-constant, input-dependent activation vector cannot be expressed as a modification of a static subsequent weight matrix.For a general choice of c(x), the required functional equality cannot hold across the entire input space.Source: D On the Distinction Between… · pp. 9–10
- The assumed static update would require matching a function restricted by the first-layer architecture to one that can be arbitrary through c(x).The proof concludes that the functions cannot generally be equal, so the operation cannot be represented by a static weight update.Source: D On the Distinction Between… · pp. 9–10
D.1 TC-LoRA as a True Parametric Adaptation
- TC-LoRA generates input-agnostic weight updates from time t and condition y before applying them to the input, allowing a learned time-varying strategy.The weight modification reconfigures the layer’s processing function rather than adding a bias to its input.Source: §D.1 · p. 10