Where to Look Is Not How to Fix: Pre-Denoising Diagnostics and Modality-Dependent Control in Diffusion Composition

1 Introduction

• The study tests the assumption that the pipeline location where compositional defects are most detectable is also where they are most effectively correctable.It compares text-encoder diagnostics and denoiser interventions at distinct pipeline sites.Source: §1 · pp. 1–2
• Text-only CSI separates common from rare compositions with ROC-AUC=1.0 across SD1.5, SDXL, and the SD3 text path.It provides an upstream diagnostic coordinate under the controlled anchor–stress setting.Source: §1 · p. 2
• On a matched six-prompt subset, downstream intervention increases CHR by 0.0272, while embedding intervention decreases it.Residual-minimizing adapters improve representation fit, linking intervention location to distinct representation and image-level outcomes.Source: §1 · pp. 2–3
• D2 has the largest positive signed diagnostic-accessibility mean, while decoder blocks have the largest positive mean CHR response to selective boosting.Broad ablation has a different architecture-dependent profile, including CHR decreases at SDXL U0 and early SD1.5 encoder blocks.Source: §1 · p. 2, Table 1 · p. 3

2 Related Work

• Many cross-attention-steering methods aggregate attention signals across layers and resolutions, treating cross-attention as a homogeneous control surface [4] [8] [20] [29] [36].Layer- or resolution-specific constraints exist, while WiCLP [40] studies embedding repair aligned with the denoiser readout.Source: §2 · p. 3
• Prior work finds CLIP embeddings compose single-object attributes but fail under multi-object binding [3], while probing research shows probe-friendly layers need not be intervention-friendly [1] [2] [9] [25].Prompt-to-Prompt [11] studies cross-attention structure, and FreeU [32] treats UNet encoder–decoder information flow as a control axis.Source: §2 · p. 3
• This paper distinguishes which layers are diagnostic from which layers and modalities respond to intervention, and quantifies a pre-denoising non-factorization signal.Output-side benchmarks [19] [33] measure composition failures; this work also diagnoses where risk appears and how interventions respond.Source: §2 · p. 3

3 Setup: Sites, Diagnostics, and Interventions

• Each controlled pair holds object and context fixed while changing the target attribute from a common anchor composition to a rarer stress composition.The study asks where stress is diagnostically readable and where interventions can improve generated output.Source: §3 · p. 3
• The experiments use 432 prompts in 216 anchor–stress groups, with attribute–object combinations drawn from T2I-CompBench, CLEVR, and ARO lexicons.The examples span common and rare compositional regimes.Source: §3 · pp. 3–4
• Generation and denoiser interventions use SD1.5 and SDXL, while SD3 is used only to extend text-path diagnostics.UNet sites include encoder stages, a middle bottleneck, and decoder stages; D2 denotes the deep encoder.Source: §3 · p. 4
• CSI tests whether a low-capacity composition model fitted on anchor prompts can reconstruct full-prompt embeddings from attribute, object, and context representations.The normalized non-factorization residual rises when prompt-specific interactions are not predictable from those components; CSI also aggregates span-geometry features.Source: §3 · p. 4
• Denoiser diagnostic accessibility measures how well block-level cross-attention patterns distinguish anchor from stress prompts during baseline generation.It uses attribute-binding measures; positive DA means tighter attribute-to-object binding for anchors, and significance against zero does not establish significant differences between sites.Source: §3 · p. 4
• Denoiser edits vary selectivity and sign: BOOST strengthens attribute-token attention, ATTR_ZERO removes it, and ZERO flattens all cross-attention at a block.The broad-additive condition is omitted because uniform attention scaling is canceled by post-softmax normalization; embedding-level repairs also move stress representations toward anchor-like structure.Source: §3 · pp. 4–5
• CHR measures the fraction of pixels inside a fixed target-object mask whose color matches the desired attribute.The mask comes from baseline final-timestep attention and is held fixed; in full per-block runs, only 52 stress prompts per seed have finite CHR.Source: §3 · p. 5

• ROC-AUC=1.0 across SD1.5, SDXL, and SD3, although the controlled protocol and a frequency-only baseline reaching 0.79 limit the claim to a stable pre-denoising stress coordinate.The protocol deliberately constructs semantically rare stress prompts; CSI is not established as a universal detector.Source: §4 · p. 5
• Within stress prompts, CSI correlates with graded severity (Spearman ρ=0.87) and attribute failure rate (ρ=0.68), and predicts attribute and binding failures with ROC-AUC=0.89 and 0.85.Controls support text-side non-factorization as the dominant contributor; the pattern appears across distinct encoder families.Source: §4 · p. 5
• Stress prompts occupy substantially higher CSI values than anchors across all three models, though absolute magnitudes are not directly comparable across models.Figure 1 uses a MAD-normalized symlog scale to show the magnitude of the anchor–stress separation.Source: §4 · p. 5, Figure 1 · p. 6

• On the matched six-prompt subset, downstream intervention increases CHR by +0.0272, whereas token-level embedding repair decreases CHR by −0.0680.Both arms use the same prompts and seeds and have identical baseline CHR; these results do not establish general semantic repair.Source: §5 · p. 6
• Pooled and token adapters reduce representation residuals (∆repr= −0.100 and −0.184), but each has observed ∆CHR=0 on its seven evaluable prompts.The pooled adapter has q=0.0024 and the token adapter q=0.0008 for residual change; both CHR results have q=1.Source: §5 · p. 6
• Rank-sweep results show no significant positive CHR effect for pooled adapters and negative effects for token adapters; the data do not identify the underlying readout mechanisms.Residual improvement is not monotonic in rank, and capacity or objective choice remain possible explanations.Source: §5 · p. 6
• The representation–generation boundary is specific to residual-minimizing objectives and does not rule out embedding repair with binding-aligned supervision such as WiCLP [40].Source: §5 · p. 6
• The largest positive signed diagnostic-accessibility mean is at D2, while the updated selective-boost CE means peak at decoder groups U0/U1.Figure 3 retains historical plotted profiles and intervals; its caption reports updated phase1 means.Source: Figure 3 · p. 7

Diagnostic accessibility

• D2 has the largest positive signed diagnostic-accessibility mean in both models, with SDXL DA(D2)= + 0.0039 (q=0.0174) and SD1.5 DA(D2)= +0.0282 (q=0.00028).These tests establish a positive anchor–stress contrast at D2, not that D2 is significantly more diagnostic than every other block.Source: §6 · p. 7

Selective boost

• Decoder-minus-D2 boost is positive and nominally significant in SDXL (+0.0074, p=0.0028), but only directionally consistent in SD1.5 (+0.0063, p=0.1882).Neither decoder-minus-baseline contrast survives within-model correction; D2 boost has no detected CHR effect in either model, which is not evidence of equivalence.Source: §6 · p. 7

Broad ablation

• SDXL broad zeroing at U0 reduces CHR by −0.1338 (q=0.0010), while SDXL D1 improves and SD1.5 D0/D1 show substantial decreases.D2 ablation does not establish benefit or harmlessness in both models; SD1.5’s positive D2 mean does not survive FDR.Source: §6 · pp. 7–8

Selective boost

• Figure 5 shows decoder boosting increased CHR over baseline in all three prompts, while same-site D2 boosting changed CHR inconsistently, including a slight increase for Red horse.Across baseline→D2→decoder conditions, CHR was 0.125→0.114→0.244 for Gold guitar, 0.239→0.224→0.372 for Blue apple, and 0.148→0.160→0.329 for Red horse.Source: Figure 5 · p. 8

Broad ablation

• Figure 6 retains historical estimates; its caption reports a substantial CHR decrease under broad zeroing at SDXL U0 and cautions that no significant D2 effect does not establish tolerance.Updated prompt-level inference is in Appendix G.Source: Figure 6 · p. 8

7 Discussion

• Diagnostic and intervention-response profiles differ along the pipeline: text-encoder residuals provide a diagnostic coordinate, D2 has the largest positive signed DA mean, and decoder blocks have the largest positive boost CE means.The discussion treats these as distinct empirical profiles rather than assuming the most diagnostic site is the best intervention site.Source: §7 · p. 8
• Figure 7 colors encode the sign of estimated CHR change, not corrected significance or functional necessity; its historical annotations require updated inference in Table 8.The caption reports large negative responses at SDXL U0 and SD1.5 D0/D1.Source: Figure 7 · p. 9
• CSI-stratified analysis shows no consistent positive relation between pre-denoising stress severity and decoder-boost response (SDXL partial ρ= −0.17, p=0.007; SD1.5 ρ= + 0.11, ns).The authors say alignment depends on objective and modality, not merely site.Source: §7 · p. 9
• At D2, neither tested token-level intervention has an FDR-significant CHR effect, and DA–CE anti-correlation varies over denoising time; the defect structure remains unidentified.The pattern is consistent with either a distributed defect across D2’s cross-attention output or a localized defect in a subspace the operators miss; competing-signal override is only a possible interpretation.Source: §7 · p. 9
• The authors propose decoder-only versus encoder-only tests of Attend-and-Excite and SynGen as predictions, not established outcomes.The proposed test asks whether decoder-only updates preserve or improve performance while reducing compute, and whether encoder-only updates degrade performance to baseline or worse.Source: §7 · p. 9

7.1 Scope and Interpretation

• The conclusions are supported for the evaluated attribute–object setting and tested intervention modalities, not relational or role-binding compositions or a universal repair mechanism.The paper does not claim that all upstream interventions fail or that decoder-side intervention universally repairs compositional failures.Source: §7.1 · p. 10

Established and suggestive findings

• The authors classify CSI separation, D2’s positive diagnostic contrast, lack of significant positive CHR effects from selective subtraction, and embedding repair without semantic improvement as established findings.They caution that D2 results are not six independent replications; decoder boost as a preferred selective site remains suggestive because its contrasts do not survive per-block FDR.Source: §8 · p. 10
• Decoder boost preference, D2 defect structure, and generality beyond chromatic binding are explicitly treated as suggestive rather than established.Decoder-minus-D2 is significant only in SDXL, and the D2 defect structure is not uniquely identified by the tested operators.Source: §8 · p. 10

Key caveats

• The evidence is constrained by color-only CHR with 52 valid prompts per seed, two UNet architectures, SD3 text-path evaluation only, and fixed intervention hyperparameters.The authors also note that CSI-based routing and a universal decoder repair interface are unvalidated, and that external-mask and background-color checks limit interpretation to a small chromatic response rather than established object-local semantic repair.Source: §8 · p. 10

9 Conclusion

• Across SD1.5, SDXL, and the SD3 text path, text-only CSI provides an upstream stress coordinate, while the two tested UNets show the largest positive signed DA mean at D2 and the largest positive selective-boost CE mean at decoder blocks.A matched six-prompt localization study links intervention location to output response while residual-minimizing adapters improve representation fit; broad ablation shows architecture-dependent sensitivity.Source: §9 · p. 10

A Extended Related Work

• The work complements output-level evaluation suites by diagnosing where compositional failures originate inside diffusion pipelines.The related-work discussion also connects its representation findings to work on compositional abilities and architecture-level evaluation.Source: A Extended Related Work · p. 15
• The per-block analysis complements representation-learning and sampling approaches by measuring how diagnostic readouts and intervention responses vary within existing diffusion pipelines.The discussion draws parallels to slot-based representations, multimodal reasoning, localized geometric evidence, and energy-based concept combination.Source: A Extended Related Work · p. 15

B CSI Definition and Discovery Details

• CSI ROC-AUC remains 1.0 across four conflict-type strata; permutation yields null ROC-AUC 0.49 ± 0.01 versus true ROC-AUC 0.89–1.00.Covariate controls weaken lexical and tokenization explanations but do not eliminate all language-side priors; span-only transfer is near chance.Source: B CSI Definition and Discovery Details · p. 16

C Embedding-Level Repair Details

• Residual-minimizing pooled and token adapters improve representation fit, but their seven-prompt CHR evaluations show zero paired change and do not establish equivalence.Separate rank-sweep runs find no significant positive CHR effect for pooled adapters and negative CHR effects for token adapters.Source: C Embedding-Level Repair Details · p. 16

D Intervention Modality Matrix

• The tested intervention matrix covers three denoiser cells and two embedding cells; broad additive denoiser scaling is untested because post-softmax renormalization makes it an identity.A proper broad-additive control would require pre-softmax logit manipulation, unsupported by the hook architecture.Source: D Intervention Modality Matrix · p. 16, Table 3 · p. 17

E DA Stability Across Conditions

• D2 has the largest positive signed diagnostic-accessibility mean in all six baseline datasets, with its contrast against zero surviving within-model, within-run FDR.Boost and attribute-zero baseline statistics coincide and are not independent replications.Source: E DA Stability Across Conditions · p. 17, Table 4 · p. 17

F Same-Site vs Relocated Comparison

• Decoder-minus-D2 boost is nominally significant in SDXL (p=0.0028), but not SD1.5 (p=0.1882); neither D2 baseline contrast is significant.The nonsignificant contrasts do not establish equivalence.Source: F Same-Site vs Relocated Comparison · p. 17, Table 5 · p. 17

G.1 CE under selective additive boost (432 prompts × 5 seeds)

• Decoder blocks U0 and U1 have the largest positive mean responses to selective boost in SDXL and SD1.5, respectively, but no block survives FDR correction.Table 6 reports per-block cross-entropy under selective additive boost.Source: §G.1 · p. 17, G Full Per-Block CE Tables · p. 17, Table 6 · p. 18

G.2 CE under selective subtractive attr_zero (432 × 5 seeds, both models full-scale)

• Selective attribute-token zeroing has no significant positive effect in either model; SD1.5 D0 has a significant negative effect.The full per-block comparison uses prompt-level BH-adjusted q-values.Source: G Full Per-Block CE Tables · p. 17, §G.2 · p. 18, Table 7 · p. 18

G.3 CE under broad subtractive zeroing (432 × 3 seeds)

• Broad zeroing increases CHR at SDXL D1 but decreases it at SDXL U0 and SD1.5 D0/D1 after FDR correction.These full per-block results contrast with selective attribute-token zeroing, which has no significant positive effect.Source: G Full Per-Block CE Tables · p. 17, §G.3 · p. 18, Table 8 · p. 18

H ABM Metric and Validation

• ABM averages attribute attention within the top-quintile object-attention mask, computed per cross-attention head and timestep.The object-local CHR endpoint measures target-color agreement within an attention-derived object region.Source: H ABM Metric and Validation · p. 18, I Color Hit Rate Details · p. 19
• Stress-prompt ABM at D2 correlates negatively with stress CHR in SDXL (ρ=−0.235) and SD1.5 (ρ=−0.124), with R2 ≈5%, making it a partial proxy.DA is weakly positively correlated with stress CHR (ρ=+0.17); the passage attributes this arithmetically to stress ABM because anchor ABM is approximately uncorrelated.Source: H ABM Metric and Validation · pp. 18–19

I Color Hit Rate Details

• CHR is finite for 52 of 216 stress prompts in the audited full per-block runs; the other 164 lack a target color recorded by the scorer.Missing target-color records do not verify semantic category labels or indicate invalid masks.Source: I Color Hit Rate Details · p. 19

J Rebuttal Sensitivity Checks

• Held-out SDXL U0–D2 CHR is +0.008149, but comparable inside-mask and background color-score changes mean increased CHR alone does not establish localized semantic repair.With baseline-only shared masks, the U0–baseline interval crossed zero while U0–D2 remained positive.Source: J Rebuttal Sensitivity Checks · p. 19

L Per-Timestep Phase Analysis

• D2’s DA is strongest early (+0.016), while early-phase boost CE is most negative (−0.010); these descriptive results do not establish a significant benefit.Block inventories differ by model, and module normalization alone does not rule out intervention-exposure differences between sites.Source: K Block Inventories · p. 19, Table 9 · p. 19, L Per-Timestep Phase Analysis · p. 20

M Power Analysis for Decoder Boost

• Observed decoder-boost power is 0.685 for SDXL and 0.557 for SD1.5; reaching 80% would require 68 and 91 color-stress prompts, respectively.These conditional power estimates do not establish that an underlying effect is nonzero or explain FDR non-significance.Source: M Power Analysis for Decoder Boost · p. 20, Table 11 · p. 20

N Prompt Examples

• The 432-prompt dataset pairs common and rare attribute–object compositions while holding object and context fixed, across four conflict types.Examples include blue pear, glass cat, striped refrigerator, and dry horse.Source: N Prompt Examples · p. 20, Table 12 · p. 21

O Fractional Attenuation Curve

• Fractional attenuation effects remain below ±0.002 and within noise; U0 is flat, while D2’s monotonic ordering is based on only four levels.The D2 magnitude is two orders smaller than broad SD1.5 D2 zeroing (CE=+0.058); U0 responds to additive boost but not attenuation.Source: O Fractional Attenuation Curve · pp. 20–21, Table 13 · p. 21

P Compute Resources

• The reported total compute is approximately 159 GPU-hours across CSI, embedding repair, and per-block modality decomposition.Experiments used NVIDIA A100 80GB GPUs via SLURM.Source: P Compute Resources · p. 21, Table 14 · p. 21