ORCA: Hunting Compositional Failures in Text-to-Image Diffusion

1 Introduction

• Compositional prompts expose persistent conditioning failures: attributes, spatial relations, and object counts can be wrong even when individual objects are well formed.The paper reports that these failures persist across model generations, architectures, and training scales.Source: §1 · p. 1
• Prior work links CLIP’s loss of binding structure to its contrastive objective, while T5 preserves compositional structure but does not itself ground it in visual composition.Class-conditional feature-alignment methods provide visual grounding, but whether they address structured text-image binding remains open.Source: §1 · p. 2
• ORCA uses the T5–CLIP residual to condition a predictor that aligns an intermediate diffusion latent with a low-rank target from self-supervised visual features.The auxiliary alignment is a training signal, leaving the inference pipeline unchanged.Source: §1 · pp. 2–3
• The paper bounds recoverable cross-modal information at a given rank by the visual encoder covariance’s top-component spectral mass, with performance saturating at the spectral elbow.The reported evaluation compares ORCA with MMDiT, REPA, and REG, with the largest gains on attribute binding, spatial relations, and multi-object prompts.Source: §1 · pp. 2–3

2 Preliminaries

• The setup interpolates between a data sample and Gaussian noise, then samples by integrating a probability-flow ODE from noise toward data.For the adopted linear interpolant, the target velocity is noise minus the data sample; the framework also permits other schedules for which the objective is defined.Source: §2 · p. 3
• A neural velocity model conditioned on text is trained with a velocity-matching objective, and the data variable is the frozen autoencoder’s latent representation.The paper adopts the linear interpolant used by MMDiT.Source: §2 · p. 3
• MMDiT uses frozen CLIP and T5 text encoders and processes text and image tokens through separate weights joined by full sequence-level attention.Its diffusion-block hidden state hT is the representation ORCA supervises against an external visual target.Source: §2 · p. 3

3 Method

• ORCA adds a training-time auxiliary objective aligning an intermediate diffusion latent with a low-rank visual target, while leaving inference unchanged.The total objective combines the diffusion loss with a single weighted ORCA loss.Source: §3 · p. 3, §3.5 · p. 5, §3.6 · p. 6

3.1 Residual textual signal

• The learned T5–CLIP residual isolates compositional textual information and supplies a prompt-dependent signal for selecting readout geometry, rather than an orthogonal complement of CLIP.A learnable linear projection maps T5 into CLIP’s embedding space; the projection is trained jointly through the auxiliary objective.Source: §3.1 · p. 4

3.2 Low-rank visual target

• The visual target projects centered features from a frozen self-supervised encoder onto its top principal components, yielding a decorrelated, variance-ordered target with nonzero variance in each dimension.The target construction has no learnable parameters, avoiding target-side representational collapse without variance regularisers or asymmetric stop-gradient schemes.Source: §3.2 · p. 4

3.3 QR map

• A small MLP maps the residual signal to an orthonormal QR basis, so text selects an n-dimensional subspace while the image latent supplies coordinates within it.The predicted target is the projection of the diffusion latent onto that basis; Figure 2 reports that subspaces separate more as prompt changes become semantically larger.Source: §3.3 · p. 4, Figure 2 · p. 5

3.4 The factorisation as an orthogonal decomposition of hT

• The latent decomposes into a rank-n component in the prompt-selected subspace and an orthogonal complement, with the auxiliary loss acting only on the selected component.The decomposition is unique once the QR map fixes the subspace.Source: §3.4 · p. 5

3.5 Auxiliary loss and training objective

• The auxiliary loss is squared error between the predicted and frozen visual targets, and the full objective adds it to diffusion loss with scalar weight λ.Performance is reported to be robust over approximately one order of magnitude of λ.Source: §3.5 · p. 5

3.6 Inference

• At inference, ORCA’s auxiliary parameters and visual target are unused, adding zero parameters, memory, and FLOPs to sampling.Sampling follows the MMDiT baseline.Source: §3.6 · p. 6

3.7 Theoretical analysis

• The recoverable-information bound is the top-n spectral mass, while power-law spectra imply a polynomially decaying tail and predict performance saturation near the spectral elbow.The paper verifies that DINOv2 features on its training corpus exhibit power-law behavior.Source: §3.7 · p. 6

4.1 Setup

• Experiments compare vanilla, REPA, and REG using FID-30K and GenEval as primary metrics, with CLIPScore and PickScore as auxiliary metrics.The experiments use MS-COCO at 256×256 and evaluate DiT-B/2, DiT-L/2, and U-ViT-L; the main results table groups results by backbone and reports checkpoint, FID-30K, and GenEval.Source: §4 · p. 6, §4.1 · pp. 6–7, Table 1 · p. 7

4.2 Main results: backbones and benchmarks

• FID 16.65 and GenEval 0.291 at 200K steps: ORCA outperforms REG at the same budget and beats the 400K Vanilla and REPA runs on both metrics.Against REG, ORCA improves FID by 10.3% and GenEval by 12.8%; it reaches better compositional fidelity than the strongest 400K baseline with half the training budget.Source: Table 1 · p. 7, §4.2 · pp. 7–8
• CLIPScore and PickScore change little across methods and are treated as auxiliary metrics because they are less sensitive to binding errors than GenEval.The main comparison therefore focuses on FID and GenEval.Source: §4.2 · p. 8
• On DiT-L/2 at matched 150K steps, GenEval gains concentrate on Position (2.9×), Color attribution (3.0×), Two objects (1.6×), and Counting (1.4×).Single-object accuracy improves more modestly (1.2×); Table 2 reports the per-task breakdown.Source: §4.2 · p. 8, Table 2 · p. 8

Subspace rank

• FID is minimised at r=64 (16.76), while GenEval is highest at r=128 (0.279); r=64 is the Pareto-efficient default.Relative to r=128, r=64 gives an 8.4% relative FID gain for a 2.5% relative GenEval cost.Source: §5 · p. 8

Alignment-block placement

• At 50K steps, block 8 of 24 outperforms earlier or later alignment, while block 24 collapses to FID 64.16 and GenEval 0.054.The authors associate the collapse with the pre-output representation being too task-specific for the auxiliary signal.Source: §5 · pp. 8–9

Loss coefficient

• The auxiliary-loss weight has a U-shaped FID response, with λ=1.0 the optimum and both smaller and larger values degrading FID.Performance is robust over approximately one order of magnitude around the default; Figure 3 sweeps each hyperparameter while holding the other two at their defaults.Source: §5 · pp. 8–9, Figure 3 · p. 9

6 Conclusion

• Across DiT-B/2, DiT-L/2, and U-ViT-L, ORCA improves FID and GenEval over vanilla training and representation-alignment baselines, with its largest GenEval gains on compositional prompts.On DiT-L/2, it reaches FID 16.65 and GenEval 0.291 at 200K steps, exceeding the strongest 400K baseline at half the training cost; it adds zero parameters and FLOPs at inference.Source: §6 · p. 9
• The evaluation uses MS-COCO at 256×256 rather than web-scale corpora because of GPU constraints.The authors identify characterising ORCA at scale as a next direction.Source: §6 · p. 9

A.1 Probability flow ODE and reverse SDE

• The appendix relates the interpolant’s velocity field to the probability-flow ODE and notes that either ODE or SDE solvers can be used at inference.The reverse SDE is also described as having the same marginal at time t as the probability-flow ODE.Source: §A.1 · p. 12

A.2 Specialisation to rectified flow and DDPM

• ORCA’s auxiliary loss does not depend on the choice of interpolation schedule and applies in both rectified-flow and DDPM settings.The appendix identifies the linear interpolant as rectified flow and the discretised cosine or linear schedule as DDPM.Source: §A.2 · p. 12

A.3 Classifier-free guidance

• ORCA leaves classifier-free guidance unchanged and applies its auxiliary loss to the conditional pass, not the unconditional pass.The unconditional pass replaces text embeddings with a learned null token during training.Source: §A.3 · p. 12

B.1 Proof of Theorem 1

• For the rank-n projection Z of centered visual features, the minimum prediction error from context H is attained by the conditional expectation E[Z | H].The residual error is zero exactly when E[Z | H] = Z almost surely.Source: §B.1 · pp. 12–13

B.2 Proof of Proposition 2

• Under λ_i ≤ Ci^-α for α > 1, the proof uses an integral test to bound the spectral tail.The bound is stated in the proposition's conclusion.Source: §B.2 · p. 13

B.3 Tightness of the bound under linear-Gaussian assumptions

• With jointly Gaussian (V, H) and a linear map plus independent noise, the optimal predictor lies in ORCA’s predictor class.The bound is tight when H contains all linear information about Z.Source: §B.3 · p. 13

C.1 QR map: numerical details

• Column normalisation and 10^-6 Gaussian noise condition the QR input and guard against rank deficiency and near-zero diagonal entries.The candidate matrix is also initialised with small, nonzero values.Source: §C.1 · pp. 13–14
• QR runs in an fp32 island while the diffusion model uses fp16 mixed precision, and the resulting basis is cast back to fp16.The passage says this avoids QR instability under fp16 arithmetic.Source: §C.1 · p. 14
• Orthonormality error stays below 5 × 10^-5 from step zero through 400K training steps, with gradients reported as well-behaved.The authors attribute stability to column normalisation, additive noise, and fp32 QR.Source: §C.1 · p. 14

C.2 ORCA module architecture

• All ORCA module components are removed at inference and contribute zero parameters and FLOPs to the deployed model.Source: §C.2 · p. 14
• The module’s trainable components are the textual projection W and basis MLP gϕ; QR and PCA add no learnable parameters.The basis MLP is a two-layer MLP with LayerNorm and GELU.Source: Table 3 · p. 14, §C.2 · p. 14

C.3 Hyperparameters

• The auxiliary-loss weight λ and rank n are ORCA-specific hyperparameters; other listed settings are inherited from the diffusion backbone.The reported hyperparameters are for DiT-L/2 on MS-COCO at 256 × 256 resolution.Source: §C.3 · p. 14, Table 4 · p. 15

C.4 Compute and runtime

• Each experiment used 4× A100 GPUs; a 400K-iteration MMDiT-L baseline takes about 1–2 days, and ORCA adds less than 7% per-step wall-clock time.Source: §C.4 · p. 15

D Additional Results

• Appendix experiments use DiT-L/2, DiT-B/2, and U-ViT-L on MS-COCO 256×256, with default rank 64, alignment block 8, loss weight 1.0, and frozen DINOv2 PCA targets with MSE loss.Source: D Additional Results · p. 15

D.1 Full per-task GenEval breakdown

• 3.9× Color attribution on U-ViT-L, 3.4× on DiT-B/2, and 1.8× on DiT-L/2 are among the largest gains; Two objects and Position also improve strongly.Compared with the 400K vanilla baseline, single-object accuracy improves more modestly, by 1.18–1.27×.Source: §D.1 · p. 15, Table 5 · p. 16

D.2 Subspace rank

• FID and GenEval improve as rank grows from 16 to 64, plateau around r∈{64, 128}, and FID degrades above that range.The table caption identifies FID’s best rank as 64 and GenEval’s as 128.Source: Table 6 · p. 16, §D.2 · p. 16

D.3 Alignment block placement

• Block 8 of 24 performs best on FID and GenEval, while last-block alignment collapses to FID=64.16 and GenEval=0.054.The ablation fixes r=64 and λ=1.0; the authors use block 8 throughout.Source: §D.3 · pp. 16–17, Table 7 · p. 17

D.4 Auxiliary-loss weight λ

• Performance is robust over roughly one order of magnitude around λ=1.0; λ=0.1 and λ=5.0 degrade FID, while PickScore stays essentially flat.Source: Table 8 · p. 17, §D.4 · p. 17

D.5 Loss function and target construction

• Frozen PCA with MSE is strongest; learnable MSE collapses, VICReg prevents collapse only partially, and NLL underperforms MSE.The frozen target removes target-side collapse as a stationary point, eliminating the need for anti-collapse regularizers.Source: Table 9 · p. 17, §D.5 · p. 17

D.6 Encoder choice

• The additional encoder configuration is only a preliminary sanity check; a systematic encoder-swap study remains future work.Source: §D.6 · p. 17, Table 10 · p. 18

• The energy ratio measures the hidden-state fraction in span K(∆y), while the aligned-component cosine compares that projection with the lifted DINOv2 PCA target.Source: §D.7 · p. 18
• The aligned component captures 80% of ∥ht∥2 in a rank-64 subspace and reaches cosine 0.80 with the DINOv2 PCA target.Across training, the energy ratio and aligned-component cosine rise from approximately 0.58 and 0.52 to approximately 0.80 within approximately 80K steps, then remain stable.Source: Figure 4 · p. 18, §D.7 · p. 18
• Residual-target cosine stays at zero by construction and serves only as an implementation sanity check, not evidence that the orthogonal complement contains no visual information.Source: Figure 4 · p. 18, §D.7 · p. 18

E Bootstrap error bars on main metrics

• Compute constraints prevented multiple-seed retraining, and bootstrap evaluations for DiT-B/2 and U-ViT-L were still pending.Each 400K-step run takes 1 to 2 days on 4× A100 GPUs; the reported bootstrap uncertainties therefore characterize DiT-L/2 configurations.Source: E Bootstrap error bars on main metrics · p. 18, Table 11 · p. 19
• FID uncertainty is estimated by resampling 30K generated features 100 times; GenEval uncertainty uses 2,212 per-image trials resampled 1,000 times.Both standard deviations are unbiased (ddof=1); the real FID reference distribution is held fixed.Source: E Bootstrap error bars on main metrics · p. 19, Table 11 · p. 19

NeurIPS Paper Checklist

• The authors report using public data and releasing code, including training scripts and configurations intended to reproduce the main results.The bootstrap script, seeds, and cached features are also reported as released for exact reproduction of the standard deviations.Source: E Bootstrap error bars on main metrics · p. 19, NeurIPS Paper Checklist · pp. 22, 25
• The authors identify more faithful controllable generation as a potential benefit and misleading or synthetic visual content as a potential harm; they do not release pretrained image-generation checkpoints.Source: NeurIPS Paper Checklist · p. 23