Source-linked AI summary
The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation
Ashritha Gonuguntla
TL;DR
The paper asks whether compositional failures arise from incorrect plans or unfaithful decoding in reasoning-augmented text-to-image models. It validates geometric measurement, intervenes on GoT-R1 plans before decoding, and finds that the decoder is faithful while planner errors and geometry are the main bottlenecks. Training-free plan repair therefore improves generation, provided plan text remains consistent with its boxes.
Problem
Compositional failures are difficult to attribute because benchmark scores conflate incorrect textual plans with poor execution of correct plans.
Method
The study uses controlled plan verification, repair, replacement, and corruption interventions with detector-based geometric scoring and paired tests.
Results
The decoder realizes 94% of planned layouts, while plan editing improves relation accuracy by up to +13.3 points, with geometry more consequential than prose style.
Takeaways & Limitations
Modular planner-decoder designs are viable when plans use clean geometry and remain internally consistent with their text.
Takeaways & Limitations
The faithful-executor conclusion is limited to GoT-R1 at 1B and 7B and validation subsets of 300 prompts, with broader replication still needed.
Abstract
from arXiv · showhide
Reasoning-augmented text-to-image models such as GoT-R1 emit an explicit textual plan - object names, attributes, and bounding boxes - before generating image tokens. When such a model fails a compositional prompt, is the plan wrong, or is the plan right and the decoder unfaithful? Because the plan is machine-readable it can be edited before decoding, which makes the two separable. We first validate the ruler. Swapping the two bounding boxes inside the model's own chain demonstrably flips the generated layout: detector-based accuracy falls 0.75 -> 0.48 (p<1e-3), while a widely used VQA-based spatial metric rises. A five-rater human study agrees with the detector on 81% of items and with the VQA judge on 57%. All spatial results therefore use geometric scoring. Under sound measurement the decoder is a faithful executor: 94% of generated layouts realize the planned relation, and object-box binding survives reordering of the plan's object segments. The planner is the bottleneck. It writes wrong relations for phrasing-dependent reasons - 98% accuracy on "left" against 54% on "right" for semantically identical layouts, a raster-order bias we isolate with a mention-order control - and cluttered geometry that the decoder faithfully reproduces. Editing the plan therefore fixes the image without retraining: symbolic verification with resampling gives +5.0 points (p<1e-3), minimal in-place repair +6.0 (p=.02), rewriting only box geometry +10.7 (p<1e-4), and replacing the plan outright +13.3 (p=1e-4). Gains are indifferent to the plan's prose style and to its likelihood under the planner, but not to its geometry. Modular planner-decoder designs are therefore viable, provided the plan is internally consistent: box-text contradictions induce object duplication and identity fusion. We release the plan-fidelity evaluation protocol, all plans, and 12k generated images.
1 Introduction
The paper separates planning errors from decoding failures by editing GoT-R1's machine-readable plans and measuring whether images follow them. It finds that the decoder is generally faithful, while planner errors in relations and geometry drive failures and can be repaired at inference time.
- Motivation: GoT-style models emit object identities, attributes, and bounding boxes as an intermediate textual plan before image tokens.The plan is trained with reinforcement-learning rewards that include plan-to-image alignment.
- Motivation: The two-stage design makes failures separable: the plan may be wrong, or a correct plan may be poorly executed.Plans provide machine-checkable constraints that can be compared with both prompts and rendered images.
- Findings: 94% of unperturbed generated layouts realize the planned relation, supporting the decoder as a faithful executor.Object-box binding also survives reordering of plan segments, indicating semantic rather than positional association.
- Findings: 98% versus 54% planning accuracy on semantically identical layouts exposes a phrasing-dependent raster-order bias.The planner also produces cluttered, overlapping boxes that limit execution quality.
- Interventions: Training-free plan edits improve results: verification adds +5.0 points and minimal in-place repair adds +6.0 points.These interventions target the diagnosed planner bottleneck without retraining.
- Implications: Plan replacement fixes images most effectively, but box-text contradictions cause duplication and identity fusion.The practical conclusion is that reasoning-augmented decoders execute plans better than their planners write them.
2 Related Work
Related work motivates explicit plans, perturbation-based faithfulness analysis, and inference-time correction for compositional generation. This paper applies those ideas to GoT-R1 and argues that its decoder executes plans faithfully, with failures concentrated in planner errors and metric artifacts.
- Reasoning-augmented generation: GoT-style systems generate semantic-spatial plans before image tokens, while related models pursue similar reasoning-augmented generation recipes.Prior benchmark studies report unreliable transfer from unified-model reasoning to pixels.
- Faithfulness analysis: Per-prompt interventions make plan correctness and image execution separately testable in reasoning-augmented image generators.The paper ports perturbation-based faithfulness analysis from language-model chain-of-thought to visual generation plans.
- Inference-time correction: Inference-time verification and self-correction usually operate on images, whereas this work checks symbolic constraints one stage earlier on textual plans.Plan resampling costs about 5 seconds versus about 18 seconds for decoding, and the two checks catch different errors.
- Plan transfer: External plans excel for this plan-trained decoder, contrasting with reports that untuned external planners can degrade another base model.The comparison links plan transferability to the decoder's training regime rather than planning in general.
- Layout conditioning: Unlike distribution-shift accounts for layout-conditioned diffusion, this study finds alien layouts execute better and internal box-text contradiction is destabilizing.The documented failure signature is duplication and identity fusion.
- Ordering bias: Mention-order biases in generation motivate the raster-order bias examined directly in the planner's box coordinates.Explicit plans make the bias observable at the planning stage.
3 Experimental Setup
The experiments study GoT-R1-1B on color and spatial compositional subsets using controlled generation, detector-based spatial metrics, and paired comparisons. The setup emphasizes plan fidelity and excludes CLIPScore because it lacks useful spatial signal.
- Model: GoT-R1-1B emits delimited object spans with bounding boxes, then generates 576 image tokens conditioned on the prompt and plan.The output resolution is 384px, with box coordinates in [0, 1000).
- Data: The evaluation uses 300-prompt color and spatial validation subsets from T2I-CompBench++ and emphasizes paired same-rig comparisons.Detector-based scoring is the benchmark's official spatial protocol, while BLIP-VQA is official for attribute binding.
- Controls: All experimental conditions share hardware, seeds, decoding settings, and conditioning paths, with headline deltas tested by paired sign-flip permutation tests.The tests use 104 resamples over prompts.
- Metrics: Spatial metrics detect objects with OWLv2 and score relation correctness, plan-execution fidelity, and agreement between planned and detected layouts.Plan-level checks compare prompt relations against parsed box centers; color uses BLIP-VQA diagnostics.
- Metrics: CLIPScore shows no usable spatial signal: relation scores are 0.3287 versus 0.3239, with r=0.066 and n=272, n.s.Its weak full-split association reflects object presence rather than arrangement and disappears under control.
4 The Planner Errs; Sound Metrics See It
Geometric validation shows that the planner, not the decoder, is the main source of compositional failure. The planner makes phrasing-dependent relation errors, while the decoder usually realizes valid plans faithfully.
- Plan validity: 79.7% of asymmetric spatial prompts are planned consistently with the requested relation, while roughly one prompt in five contains a planner error.Under a stricter verifier, 76.3% of spatial and 77.7% of color chains pass on the first sample.
- Planner asymmetry: 98.0% versus 53.9% planning accuracy for left versus inverted right phrasing reveals a raster-order bias independent of object order.The corresponding original values are 97.8% and 55.1%, with identical layouts expressed through inverted wording.
- Metric validation: Box swapping reduces OWLv2 prompt-relation accuracy from 0.750 to 0.477, a −27.3-point collapse, while BLIP-VQA rises from 0.618 to 0.681.Human raters agree with detector verdicts on 81% of items but with BLIP-VQA on 57%, validating geometric scoring.
- Plan interventions: Replacing the plan performs best in the intervention ladder, but gains require clean, well-separated boxes rather than familiar prose style or box statistics.The table reports OWLv2 relation correctness on spatial_val with n=300 per condition.
- Metric validation: All spatial conclusions use detector-based scoring because VQA-style relation metrics can invert the sign of a known layout corruption.The detector is aligned with the benchmark's prescribed spatial evaluation.
- Decoder fidelity: 93.8% of generated layouts realize the planned relation, with planned-object detection at 95.7%/95.0% and planned-vs-detected box IoU of 0.747.Prompt-level spatial accuracy is 0.750, close to the 79.7% plan-validity rate after detection noise.
5 Intervening on the Plan
Inference-time plan interventions improve generation without retraining, showing that image quality tracks plan geometry and that verification or repair can address planner errors before decoding.
- 5.1 Verify-then-generate: +5.0 points: verification with resampling improves detector relation accuracy from 0.773 to 0.823 (p=.0007).Verification checks plan constraints before decoding and resamples failed chains.
- 5.1 Verify-then-generate: Verification is bounded because it converts invalid plans into valid ones while leaving decoder execution unchanged.This separates correcting planner outputs from improving the decoder itself.
- 5.2 Minimal in-place repair: +6.0 points: minimal in-place repair improves relation accuracy from 0.750 to 0.810 (p=.021).Repair swaps box coordinates or inserts omitted colors while preserving most plan content.
- 5.3 Replacing the plan: the oracle dial: +13.3 points: clean-box injected plans outperform the planner’s own plans, whereas planner-statistics boxes produce no significant gain.Performance is tied to box geometry, not prose style or planner familiarity.
- 5.3 Replacing the plan: the oracle dial: +10.7 points: rewriting only box geometry improves overall relation accuracy at a target margin of 0.2 (p<10^-4).The gain is dominated by correcting originally invalid relations; larger target margins provide no additional benefit.
- 5.3 Replacing the plan: the oracle dial: Contradictory boxes and prose produce duplicated or fused objects, so plan edits must preserve internal consistency.The degradation reflects conflicting conditioning rather than plan novelty.
6 What Does the Decoder Read? Perturbing the Plan
Controlled perturbations show that the decoder follows semantic plan content, especially geometry and attributes, while relying on the chain for object identity; contradictory conditioning causes distinctive failures.
- Decoder inputs: Swapping boxes flips layouts, while reversing object-segment order leaves image metrics at control levels.Object-box binding is semantic rather than positional.
- Decoder inputs: Color swaps in the chain recolor objects, with color match falling from 0.788 to 0.561 while object presence remains unchanged.When chain and prompt disagree, the chain dominates attribute binding.
- Conflicting conditions: Internally contradictory plans produce stacked, duplicated objects and identity fusion, including a man–giraffe hybrid.The failure signature arises when plan components disagree.
- Decoder inputs: Ablating the chain reduces object detection by 16–23 points, showing that the prompt alone no longer reliably specifies which objects to draw.The chain is the primary conditioning channel after plan-conditioned reinforcement-learning training.
- Mechanism probes: Exploratory attention probes find only ≈0.7% of conditioning attention on box tokens, with weak or absent correlation to success.The computation that reads boxes is not identifiable from bulk attention reallocation.
7 Robustness: Seeds and Scale
Additional seeds preserve the intervention ladder, while scaling to 7B preserves the faithful-decoder finding and improves planning quality. As planner errors shrink, plan-repair gains also shrink.
- Seeds: Control 0.751±0.005, minimal repair 0.834±0.024, geometric plan repair 0.849±0.007, and clean-box oracle 0.890±0.009 across three seeds.The ladder’s ordering holds for every individual seed.
- Scale: GoT-R1-7B: At 7B, the decoder remains faithful: followed-plan 0.92 and IoU 0.76.Box swapping reduces detector accuracy from 0.863 to 0.433 (−43 points, p<10^-4).
- Scale: GoT-R1-7B: At 7B, 89.3% of chains pass verification first-try versus 76.3% at 1B, while plan-repair gains shrink to +1.7 to +2.3.Only full replacement remains significant at +6.3; every intervention still improves over control.
8 Revisiting Our Earlier Finding, and Why It Reversed
An earlier result that oracle plans degraded metrics was reversed after auditing stimulus integrity, metric validity, and statistical power. Corrected procedures instead produced a larger positive effect.
- Audit findings: The earlier −4.9-point degradation used internally inconsistent oracle templates, partly layout-insensitive VQA scoring, and an underpowered unpaired design.The legacy result used n=300 and fell below the design’s minimal detectable difference.
- Reversal: With corrected templates, geometric scoring, matched controls, and paired tests, the effect reversed from −4.9 to +13.3 points.The corrected effect was 2.7× larger in magnitude.
- Implication: Apparent “plans don’t transfer” findings should be scrutinized for stimulus integrity and metric validity before invoking co-adaptation.This conclusion follows the reported trajectory from the audited pipeline.
9 Why Explicit Plans in the First Place
Explicit plans are studied because global embeddings may not reliably carry compositional bindings, while plan interfaces expose machine-checkable structure for intervention. The paper argues for repairing plans rather than decoders, with geometry and consistency as priorities.
- Motivation: Global embedding probes across CLIP, SDXL, FLUX, and SD3 found near-zero binding margins and 0% linear recovery of the ordered spatial triple.The paper uses this context to motivate carrying bindings explicitly in a plan.
- Patch the plan, not the decoder: The intervention ladder is monotone: verification gives +5, repair +6, and replacement with clean geometry +13, all without training.Plan checks can precede decoding, while image checks can catch residual execution errors.
- Patch the plan, not the decoder: External planners can drive the decoder effectively if boxes remain clean and well-separated and prose stays consistent with box geometry.The alternative is to omit prose when it cannot be kept consistent.
- Train the planner’s geometry: The decoder faithfully reproduces cluttered boxes, making planner geometry—not decoder capacity—the highest-leverage training target.The reported raster-order bias also motivates order-balanced planning data.
- Measure geometrically: Detector-based plan-fidelity scoring is necessary because VQA spatial metrics can invert the sign of a 27-point effect.The paper releases the geometric evaluator.
11 Limitations
The study’s conclusions are bounded by its model family, validation-scale evaluation, detector and judge properties, and exploratory analyses. Improved compositional control also creates dual-use and bias-audit concerns.
- Scope: The faithful-executor conclusion is tested only on GoT-R1 at 1B and 7B and should be replicated on other reasoning-augmented generators.The paper names T2I-R1 and BAGEL as examples before generalizing across architectures.
- Evaluation boundaries: Validation subsets contain 300 prompts, and full-benchmark evaluation remains future work.Headline conditions use three seeds and paired tests.
- Measurement caveats: Detector-based scoring inherits OWLv2 detection limits, while VQA-based color scoring retains judge biases and tests color presence rather than binding.The planner-mimic condition may also contain donor-specific prose noise.
- Ethics: Improved compositional control is dual-use: it increases usefulness while making targeted synthetic imagery easier to produce.The paper recommends screening harmful objects and relations at the plan stage and auditing biased associations.
- Reproducibility: The released artifacts include scripts, the detector-based evaluator, approximately 12k plans and images, scores, and paired-statistics code.The procedures and decode settings are fully specified.
A Per-Relation Planning Accuracy
Planner accuracy tracks the relation word rather than which object is mentioned first, with semantically identical layouts differing by up to 44 points depending on phrasing.
- Up to 44 points separate planner accuracy for semantically identical layouts depending on phrasing.Accuracy follows the relation word regardless of which object is mentioned first.