PaperlayerView exact revision on arXiv

Source-linked AI summary

TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward

Debottam Dutta, Jaehoon Hahm, Jianchong Chen, Romit Roy Choudhury

arXiv:2607.21606v1cs.AI

TL;DR

Complex prompts still challenge text-to-image diffusion models’ compositional faithfulness. TILT addresses this with intrinsic test-time reward alignment and achieves strong or comparable-to-stronger performance across compositional generation benchmarks while preserving image quality.

  • Problem

    Text-to-image diffusion models frequently fail to generate faithful images for complex compositional prompts.

  • Method

    TILT reward-aligns diffusion sampling at inference time using an intrinsic pure-mode reward and derives principled guidance objectives without external supervision or fine-tuning.

  • Results

    TILT’s hybrid guidance achieves comparable or stronger generation quality than prior approaches across multiple compositional generation benchmarks.

  • Takeaways & Limitations

    The framework provides a principled basis for test-time compositional sampling and theoretically grounds CO3 as a special case.

  • Takeaways & Limitations

    Gradient-based guidance adds computational cost and can be unstable, with some categories showing relatively lower BLIP-VQA scores despite strong ImageReward performance.

Abstract

from arXiv · show

Recent advances in powerful text-to-image generation models have made it increasingly important to develop test-time methods that modify the sampling trajectory to produce images more faithful to complex compositional prompts. We present TILT, a training-free framework for compositional text-to-image generation via test-time reward alignment. We interpret compositional failures as overlap modes between joint and single-concept distributions, and define a reward that favors samples where all concepts are jointly present. This reward is intrinsic to the base model and does not require any external supervision or reward models. This yields a KL-constrained objective with a closed-form tilted target distribution and principled guiding steps for diffusion sampling. The interaction of concept distributions together with the above reward naturally leads to two different guidance strategies while a hybrid approach that balances their respective benefits produces stronger performance. Experiments on prompts from T2ICompBench show that our method improves compositional alignment while preserving image quality compared to previous baselines.

1. Introduction

Complex compositional prompts often cause T2I diffusion models to let stronger concepts dominate weaker ones. TILT addresses this by deriving intrinsic test-time reward alignment and complementary guidance updates, including a hybrid evaluated against prior methods.

  • Motivation: T2I diffusion models frequently fail on complex compositional prompts because competing concept affinities can cause one concept to dominate the others.The paper attributes this competition to imperfect composition, data imbalance, and imperfect training.
  • Motivation: Prior concept-contrasting correction methods are empirically effective but heuristic, lacking a principled objective or clearly defined terminal distribution.This motivates a mathematically principled treatment of concept-contrasting generation.
  • TILT framework: TILT formulates pure-mode sampling as an intrinsic test-time reward alignment problem and rigorously derives intermediate diffusion guidance objectives without external supervision or model fine-tuning.The method combines priors learned by the base model to define a suitable terminal reward and guide sampling toward pure modes.
  • Guidance algorithms: TILT produces two complementary updates: efficient TILT-S for early high-noise steps and more accurate TILT-C for later low-noise compositional binding.Their differing strengths motivate switching between them across diffusion time.
  • Contributions: The framework places prior works within a common formulation, and its hybrid guidance achieves comparable or better performance than prior methods on compositional generation benchmarks.The paper reports this benchmark comparison in §4.

2. Background

The background introduces diffusion T2I sampling through Tweedie denoising and deterministic DDIM updates, then reviews classifier-free and compositional guidance methods. It highlights that heuristic score composition can fail for complex prompts because it does not match the forward diffusion distribution.

  • Diffusion Sampling: Tweedie’s formula derives a denoised estimate from the noisy latent x_t using the text-conditioned predicted noise and cumulative noising schedule.The predicted noise is denoted ϵ_θ, and the schedule term is ¯α_t.
  • Diffusion Sampling: Under noise-free sampling, DDIM deterministically evolves the denoised estimate x̂_0 to the previous latent x_{t−1}.DDIM reuses the same predicted noise and therefore avoids the renoisification step used by stochastic samplers such as DDPM.
  • Classifier-Free Guidance: Classifier-free guidance combines conditional and unconditional scores through a convex combination to improve prompt alignment during inference.Standard CFG uses the guided prediction both for the Tweedie estimate and for propagating the noise component.
  • Classifier-Free Guidance: CFG++ reduces off-manifold drift by retaining the guided Tweedie estimate while using unconditional noise prediction for the renoising direction.This makes transport from x_t to x_{t−1} follow the unconditional diffusion manifold.
  • Compositional Guidance: Composable Diffusion composes concept-specific scores for joint conditions, but heuristic linear score combinations cannot accurately generate complex prompts.The limitation arises because no score of the forward distribution ˜p_t(x_t | c_1, . . . , c_K) coincides with the heuristic combination at t > 0.

3. Method

TILT models compositional failures as overlap modes between joint and single-concept distributions, then derives a KL-constrained reward-alignment target that favors pure modes while staying near the pretrained model. It implements this target through Jacobian-based diffusion guidance, including shared, per-concept, and hybrid strategies.

  • Overlap-mode formulation: Compositional failures arise when the joint conditional overlaps a single-concept conditional, producing samples that retain only one concept.For animal–object prompts under SDXL, DINOv2 score densities concentrate near the axes, identifying these overlap modes.
  • Reward-alignment target: TILT reweights the joint distribution by inverse marginal products, suppressing overlap regions while preserving samples supported by all concepts.This constructs a pure-mode sampling target with high joint likelihood and balanced per-concept likelihood.
  • Reward alignment: The optimal distribution maximizes pure-mode reward under a KL constraint to the pretrained joint conditional, preventing drift toward out-of-distribution images.The framework derives a closed-form solution and interprets pure-mode sampling as test-time reward alignment.
  • Diffusion guidance: Intermediate-time guidance replaces an intractable terminal-likelihood optimization with a Jacobian-based surrogate whose Jacobian approximation defines the design space.Directly solving the PF-ODE and backpropagating through it would be extremely computationally extensive.
  • Guidance variants: TILT-S uses one joint-conditional Jacobian and one backward pass, whereas TILT-C uses per-concept paths for tighter approximation at the cost of K+1 backward passes per step.TILT-H combines them by using TILT-S at high noise and TILT-C at low noise, with switching threshold τ ∈(0, T).

4. Experiments

Experiments evaluate TILT as a training-free, inference-only correction method on SDXL using T2ICompBench’s compositional categories and automatic metrics. Results show stronger compositional consistency and a better balance between correctness and visual quality than prior baselines.

  • Implementation details: Experiments use SDXL without additional training or fine-tuning, modifying only inference with 50-step DDIM sampling at 1024×1024 resolution.Prompts are automatically decomposed into concept-level sub-prompts using noun parsers and simple preprocessing.
  • Implementation details: TILT applies score correction during the first few denoising steps, where global composition is typically determined.Unless otherwise specified, the method corrects the first 5 steps with one latent correction step per timestep and 10 initial-timestep correction iterations.
  • Evaluation benchmark and metrics: Evaluation uses T2ICompBench prompts spanning Color, Shape, Texture, and Complex categories, with ImageReward, CLIP, DINO, and BLIP-VQA metrics.The Complex category tests simultaneous multiple objects, relations, and attributes, while BLIP-VQA specifically measures compositional correctness through attribute-based questions.
  • Comparison methods and baselines: TILT is compared with train-free, gradient-free, model-agnostic SDXL baselines including CFG and Composable Diffusion.CFG is the standard classifier-free guidance method, while Composable Diffusion combines multiple concept-specific score functions.
  • Quantitative results: TILT consistently performs strongly on BLIP-VQA and ImageReward, achieving the best ImageReward scores on Shape and Complex categories.The Complex improvement indicates particular effectiveness for prompts requiring simultaneous satisfaction of multiple attributes and object relationships.
  • Qualitative results: Qualitative results show TILT better preserves requested concepts and attributes, improving color, shape, texture, object-count, material, and spatial-relation fidelity while retaining visual quality.Prior baselines often satisfy only part of the prompt, such as the object category without correct attribute binding.

5. Related Works

Related work improves compositional text-to-image generation through score or energy composition, spatial layout control, and language-model augmentation. These approaches include training-free sampling methods, explicit spatial priors, and richer linguistic guidance for frozen diffusion models.

  • Composable generation works: Composable-generation methods algebraically combine conditional diffusion scores or energies, but early score composition struggles with concept mixing and omission.Composed-Diffusion formalizes score composition within classifier-free guidance and demonstrates test-time generalization.
  • Composable generation works: Training-free, model-agnostic samplers improve multi-condition generation by interpolating between frequent- and rare-concept distributions.The passage introduces this family alongside energy-parameterized diffusion and Metropolis/MCMC-corrected samplers.
  • Composable generation works: Kwon & Ye (2025) uses a Tweedie-space composition strategy similar to TILT’s resampler, but to sample initial noise through repeated DDIM forward-backward passes.Unlike TILT’s resampler, their method is described as operating in a different role rather than acting as a corrector.
  • Layout-augmented image generation: Layout-to-image methods impose explicit spatial priors, including bounding boxes, segmentation masks, or region-level text, while training-free variants manipulate cross-attention maps.These methods aim to tighten prompt–image correspondence and place objects within designated regions.
  • Layout-augmented image generation: LLM-augmented methods decompose complex prompts into spatially grounded sub-tasks, infer layouts from text, or provide richer text representations to frozen diffusion backbones.These strategies guide region-wise diffusion, resolve relational constraints before generation, or inject stronger language representations during denoising.

6. Conclusion

TILT frames compositional text-to-image generation as principled test-time intrinsic-reward alignment, yielding a closed-form target distribution and diffusion-derived guidance rules. Its gradient-based guidance introduces computational cost and can be less stable than standard external-reward optimization.

  • 6. Conclusion: TILT presents a principled framework for compositional text-to-image generation based on test-time reward alignment.It formulates pure-mode compositional sampling as intrinsic-reward optimization rather than heuristic diffusion-trajectory manipulation.
  • 6. Conclusion: The formulation yields a closed-form target distribution and naturally derived guidance rules for diffusion sampling.
  • 6. Conclusion: Gradient-based reward updates add computational cost compared with gradient-free correction methods.The cost is more pronounced for the per-concept Jacobian variant, which requires multiple backward passes per denoising step.
  • 6. Conclusion: Model-likelihood gradient optimization can be less stable than optimizing standard external rewards.

B. Pointwise reward & Interpretation … B.3. Per-prompt expected reward

This section interprets TILT’s pointwise reward as a per-prompt information-theoretic quantity and explains how its expected form guides sampling toward images with high pointwise conditional Total Correlation. Aggregating this quantity across prompts recovers population-level conditional Total Correlation.

  • Pointwise reward & Interpretation: The analysis follows an InfoNCE-style template, identifying the per-prompt, per-image quantity acted on by the sampler before relating it to population-level conditional Total Correlation.The aggregation result is established in the subsequent population-level analysis.
  • B.1. Decomposition via Bayes: Bayes’ rule rewrites each conditional likelihood as a corresponding posterior likelihood plus the image marginal minus the prompt marginal.The decomposition is applied to both the joint prompt tuple and each individual concept conditional.
  • B.1. Decomposition via Bayes: The resulting κ(C) term is independent of x, so it does not affect optimization over images for a fixed prompt.Its independence from x makes it irrelevant to image-dependent sampling decisions.
  • B.2. The pointwise objective: pcTC: The pointwise objective is the pointwise conditional Total Correlation at prompt tuple C given image x, analogous to pointwise mutual information.It is described as the per-realization analogue of conditional Total Correlation and mutual information.
  • B.2. The pointwise objective: pcTC: For fixed C, ι_C(x) measures how much more likely the joint concept tuple is under the model posterior than under independent per-concept posterior factors.The comparison is evaluated at the same image x.
  • B.3. Per-prompt expected reward: For small ϵ, the constrained optimum p* remains approximately pθ(x | C), allowing expectations to be taken under the base conditional up to O(ϵ).This approximation characterizes the per-prompt expected-reward analysis.
  • B.3. Per-prompt expected reward: Only the first term in the expected reward steers sampling; the remaining terms are constant with respect to x for fixed C.One remaining term is a C-dependent cross-entropy, while the other is a pure constant.
  • B.3. Per-prompt expected reward: The sampler therefore pushes probability mass toward images where pcTC is high.This is the per-prompt interpretation stated in Proposition 3.1.

B.4. Population-level interpretation: from pcTC to conditional TC

Averaging the per-prompt reward across a benchmark transforms the pointwise objective into a population-level functional equal to conditional Total Correlation up to a constant. This connects local diffusion-time guidance with benchmark-level compositional evaluation.

  • Population-level identity: Averaging equation 24 over the empirical prompt distribution p(C) yields a population-level identity involving conditional Total Correlation of the concept tuple given X.The derivation uses p(x)p(C | x) = p(x, C) = p(C)p(x | C).
  • Population-level identity: The benchmark reward satisfies C[R(x)] = TC(c1; . . . ; cK | X) + const.Thus, the averaged reward is the conditional Total Correlation functional up to an additive constant.
  • Interpretation: The pointwise objective ιC(x) provides a local signal at each diffusion state xt, while benchmarks assess the population summary TC(· | X) that it aggregates to.This establishes the relationship between per-prompt diffusion guidance and population-level compositional evaluation.
  • Interpretation: Conditional Total Correlation vanishes when concepts are conditionally independent given X and is maximized when X jointly determines all concepts.The maximizing regime corresponds to non-trivial compositional reasoning.

C. CO3 as a special case of TILT-S … D.3. Tweedie-Space Composition

The paper situates CO3 as an approximation of TILT-S and presents Tweedie-space composition as a general framework encompassing score-space composition. It identifies the approximations underlying CO3 and their limitations at high noise, while relating CFG and multi-concept composition methods.

  • C. CO3 as a special case of TILT-S: CO3’s corrector composes scaled Tweedie means using weights w0 = 1+β and wk = −β/K.This weighted Tweedie-mean composition forms the correction’s underlying composed estimate.
  • C. CO3 as a special case of TILT-S: CO3’s corrector is recovered from TILT-S equation 16 by imposing an identity-Jacobian approximation and a time-frozen-score approximation.These assumptions simplify the TILT-S guidance term into CO3’s correction direction, up to a scalar absorbed into β.
  • C. CO3 as a special case of TILT-S: CO3 discards diffusion-time geometry and Tweedie-posterior score rerouting, making both approximations highly inaccurate at high noise levels.At high noise, the Tweedie mean is far from xt and its Jacobian deviates strongly from I.
  • D. Composition in Score and Tweedie-space: The paper reviews existing composition methods and characterizes Tweedie-space composition as a strictly more general framework.The comparison spans classifier-free guidance, score-space composition, and Tweedie-space composition.
  • D.1. CFG as a Form of Score Composition: Classifier-free guidance composes conditional and unconditional predicted noise at each timestep to sample from p(x | c), then uses the result to compute the Tweedie mean and next denoised state.CFG is described as a binary composition of conditional and unconditional score estimates.
  • D.2. Score-Space Composition: Liu et al. (2022) extended CFG to multi-concept generation by assuming p(x0 | C) = p(x0) Q i p(ci | x0) and directly summing per-concept scores.The paper notes that this extension is intuitive but is not a valid CFG composition for arbitrary weights.
  • D.3. Tweedie-Space Composition: CO3 instead combines Tweedie means from different conditional predictions, providing a more principled composition framework than directly adding noise predictions.Tweedie-space composition recovers score-space composition as a special case when the composition weights are chosen appropriately.

E. More Implementation Details

The implementation applies reward correction over the first five diffusion timesteps using staged TILT-S and TILT-C guidance, with specified optimization budgets and β values. Prompts are parsed into noun-based concepts after removing articles and adjectives.

  • Reward correction schedule: For Table 1, reward correction runs for the first 5 time steps, using TILT-S initially and TILT-C for the following 4 steps.Each timestep uses 5 optimization steps, except the initial timestep, which uses 10.
  • Reward correction schedule: The guidance strengths are β = 0.05 for TILT-S and β = 0.004 for TILT-C.
  • Prompt parsing: Stanza parses prompts into noun chunks, removes articles and adjectives, and uses the remaining proper nouns as TILT concepts.For “a black cat and a brown dog,” the concepts are “cat” and “dog.”

F. More Qualitative Results

TILT is qualitatively compared with other baselines on T2ICompBench through additional success and failure examples spanning color, shape, texture, and complex prompt categories.

  • Qualitative Results: Additional qualitative comparisons show TILT success and failure examples against other baselines on T2ICompBench.The success examples appear in Figure 4, while failure examples are visualized in Figure 5.
  • Qualitative Results: The evaluated prompts cover color, shape, texture, and complex categories.These categories are represented in the qualitative comparison prompts shown in Figure 4.
Loading 2607.21606v1…