PaperlayerView exact revision on arXiv

Source-linked AI summary

CCG Diffusion: Combinatory Categorial Grammar-Driven Compositional and Syntactically Disambiguated Text-to-Image Generation

Ryosuke Yamaki, Daichi Mochihashi, Nobutaka Shimada, Seitaro Shinagawa, Tadahiro Taniguchi

arXiv:2610.29415v1cs.CVcs.CL

TL;DR

Text-to-image models do not explicitly receive prompt syntax, limiting control over modifier binding, spatial relations, counts, and ambiguous interpretations. CCG Diffusion uses CCG derivations for prompt rewriting and attention guidance; it performs strongly on compositional benchmarks and PP ambiguity resolution, with scope limitations in other relations and ambiguity types.

  • Problem

    Text-to-image models lack explicit prompt syntax, while existing syntax-based controls have limited coverage of spatial relations, counts, and specified interpretations of syntactic ambiguity.

    Source: §I · pp. 1–3

  • Method

    CCG Diffusion uses CCG derivations to guide generation through prompt rewriting and attention guidance, including for specified PP-attachment interpretations.

    Source: §I · p. 3

  • Results

    CCG Diffusion achieves the highest attribute-binding and 2D-spatial scores among compared methods on both backbones and significantly outperforms all compared methods on PP ambiguity resolution.

    Source: §I · p. 3

  • Takeaways & Limitations

    A single CCG derivation provides syntactic information for both compositional generation control and PP ambiguity resolution.

    Source: §VII · p. 16

    CCG Diffusion addresses only PP attachment ambiguity and lacks controls for depth relations, actions, and multiple constraints.

    Source: §VII · p. 16

Abstract

from arXiv · show

Text-to-image generation models do not explicitly receive the syntactic structure of the prompt, i.e., which word modifies which and where each phrase attaches. Consequently, they often struggle to bind attributes to the correct objects, to depict spatial relations and counts correctly, and to render syntactically ambiguous prompts in the intended interpretation. Existing syntax-based control methods mainly address attribute binding and have limited ability to control spatial relations and counts or to specify the interpretation of syntactically ambiguous prompts. They rely on phrase-structure or dependency grammars and require additional rules or constraints on the tree structure to extract modification relations between words or to specify a syntactic interpretation. We propose CCG Diffusion, a training-free method that controls image generation using Combinatory Categorial Grammar (CCG). In CCG, the category of each word carries rich syntactic information: it specifies what the word combines with and what phrase it forms. Modification relations between words can therefore be read directly from a CCG parse. Moreover, prepositional phrase (PP) attachment ambiguity, a representative syntactic ambiguity, can be resolved simply by fixing the preposition's category. CCG Diffusion applies the same two CCG-based mechanisms, prompt rewriting and attention guidance, both to controlling attribute binding, spatial relations, and counts and to resolving PP ambiguity. On two architecturally different backbones, Stable Diffusion and Sana, CCG Diffusion achieves the highest scores among the compared methods in attribute binding and 2D spatial relations and significantly outperforms all of them in PP ambiguity resolution.

I. INTRODUCTION

Text-to-image models receive prompt words without explicit syntactic structure, and may misbind attributes, spatial relations, counts, or ambiguous phrase attachments. CCG Diffusion uses CCG-derived structure to guide generation and reports strong benchmark performance across compositional control and PP ambiguity resolution.

I. INTRODUCTION

  • Models may bind attributes to the wrong objects, depict incorrect spatial relations or counts, or leave PP-attachment interpretation to the model because prompt syntax is not explicit.The introduction illustrates these failures with swapped colors, a chicken placed on rather than right of a suitcase, an incorrect rabbit count, and a telescope assigned only to the boy.Source: §I · pp. 1–2
  • CCG categories specify what each word combines with and what phrase it forms, so modification relations can be read from a derivation and PP attachment can be selected by fixing the preposition’s category.Unlike phrase-structure and dependency approaches described here, CCG handles both through word categories rather than additional extraction rules or tree constraints.Source: §I · p. 3
  • The method conveys CCG-derived syntax through prompt rewriting and attention guidance, controlling attribute binding, spatial relations, and counts as well as the selected PP-attachment interpretation.For PP ambiguity, it paraphrases the prompt to admit only the specified interpretation and guides the PP object toward its holder.Source: §I · p. 3
  • CCG Diffusion requires no additional parser or generation-model training and uses only the prompt and, for ambiguous prompts, the specified interpretation.The evaluation uses Stable Diffusion and Sana, which have different architectures.Source: §I · p. 3
  • CCG Diffusion achieves the highest attribute-binding and 2D-spatial scores among compared methods on Stable Diffusion and Sana, and significantly outperforms all compared methods on PP ambiguity resolution.It also significantly outperforms Predicated Diffusion in numeracy.Source: §I · p. 3

II. RELATED WORK

Prior compositional-generation methods use attention guidance, parsing, or text-conditioning interventions, while ambiguity work has largely characterized ambiguity or used user clarification. These approaches have distinct limits in controlling layout and counts or directly specifying an intended syntactic interpretation.

A. COMPOSITIONAL TEXT-TO-IMAGE GENERATION

  • Attention-guidance methods use cross-attention losses to address failures such as missing, misbound, or fused objects, while Predicated Diffusion represents prompt intent as predicate-logic propositions.Parsing-based methods commonly extract attribute–object pairs or noun phrases using rules tailored to parser outputs.Source: A. COMPOSITIONAL TEXT-TO-IMAGE… · pp. 3–4
  • Many methods controlling object layout or counts require boxes, LLM-generated layouts, counting networks, or trained layout-prediction models, although some guide spatial relations through attention centroids..Source: A. COMPOSITIONAL TEXT-TO-IMAGE… · p. 4
  • Other compositional approaches alter conditioning by combining denoising scores, separately encoding parsed constituents, editing token embeddings, or using LLM-planned regional subprompts.Source: A. COMPOSITIONAL TEXT-TO-IMAGE… · p. 4

B. SYNTACTIC AMBIGUITY RESOLUTION

  • Prior ambiguity research has characterized ambiguity and model behavior, while visual-resolution work has often selected interpretations from images rather than generated them.TAB introduced TIED, which appends a user’s answer or selected visual setup to the original prompt; later agents also elicit and represent user intent.Source: B. SYNTACTIC AMBIGUITY RESOLUTION · p. 4

III. PRELIMINARIES

Predicated Diffusion guides a pretrained diffusion model using losses derived from predicate-logic propositions representing prompt intent. Its proposition types target object presence, modifier correspondence and leakage, and holder–object possession.

A. PREDICATED DIFFUSION

  • Predicated Diffusion treats cross-attention maps as fuzzy predicates and converts proposition satisfaction into differentiable losses that guide generation.Each word’s attention value is interpreted as the degree to which its predicate holds at an image location; the negative logarithm of a proposition’s fuzzy truth value supplies its loss.Source: A. PREDICATED DIFFUSION · p. 4
  • At each sampling step, the method adds the loss gradient as guidance to the reverse-process update before applying a standard denoising step.The state starts from noise and, in latent diffusion models such as Stable Diffusion, is a VAE-derived latent.Source: A. PREDICATED DIFFUSION · p. 4
  • Its four proposition types represent existence, correspondence, leakage, and possession, targeting object presence, correct modifier–noun binding, attribute separation from other nouns, and holder–object co-presence.Targets are instantiated from the nouns, modifier–noun pairs, modifier–nonmodified-noun pairs, and holder–object pairs in the prompt.Source: A. PREDICATED DIFFUSION · pp. 4–5, Table 1 · p. 5
  • Predicated Diffusion averages proposition losses within each type, assigns zero loss when a type has no propositions, and combines the type losses with weights.Source: A. PREDICATED DIFFUSION · p. 5

B. COMBINATORY CATEGORIAL GRAMMAR

CCG categories encode what words combine with, allowing modification relations and selected syntactic structures to be read from a derivation. CCG Diffusion uses that information for guidance targets and applies prompt rewriting and attention guidance to improve compositional generation.

B. COMBINATORY CATEGORIAL GRAMMAR

  • CCG categories encode argument structure, so word-to-word modification relations and PP attachment can be read from the derivation rather than recovered with extra tree rules.Fixing one word’s supertag can constrain the sentence structure, including which phrase a PP modifies.Source: B. COMBINATORY CATEGORIAL GRAMMAR · p. 5

IV. METHOD: CCG DIFFUSION

  • CCG Diffusion extends Predicated Diffusion with CCG parsing and controls generation through prompt rewriting and attention guidance, without additional image-model training.The method uses the derivation for compositional generation and interpretation-specific PP ambiguity resolution.Source: §IV · p. 6

A. GUIDANCE TARGETS FROM CCG DERIVATION

  • The guidance targets read from the derivation include nouns, modifier–noun pairs, other nouns a modifier does not modify, and selected verb subject–object pairs.The method also extracts coordinated noun phrases, relation phrases, cardinal numbers, and PPs from the same derivation.Source: A. GUIDANCE TARGETS FROM CCG DERIVATION · p. 6

B. COMPOSITIONAL TEXT-TO-IMAGE GENERATION

  • For compositional prompts, CCG Diffusion addresses attribute leakage or object fusion, incorrect spatial relations, and incorrect object counts by extending Predicated Diffusion with derivation information.The method applies a stronger guidance strength selectively when a CCG-bound noun modifier denotes color, shape, size, material, or surface.Source: B. COMPOSITIONAL TEXT-TO-IMAGE… · p. 6, 1) Selective Guidance Boost · p. 6

1) Selective Guidance Boost

  • Selective guidance boosts the update strength only for inputs whose noun modifiers include a color, shape, size, material, or surface word.Larger updates tend to improve attribute binding, but overly large updates can corrupt the image.Source: 1) Selective Guidance Boost · p. 6

2) Positional Prefix and Region Assignment Loss

  • The Positional Prefix places each coordinated noun phrase after a left/right cue, while Region Assignment Loss directs its noun and modifier attention to the corresponding side.Together, they encourage separate object regions and mitigate attribute swapping and object fusion; the loss assigns left/right sides even when the prompt specifies no positions.Source: 2) Positional Prefix and Region… · pp. 6, 8

3) Spatial Relation Loss

CCG Diffusion detects specified spatial relations from the derivation and guides object positions using noun attention centroids. Its loss enforces directional separation or proximity while penalizing overlap.

3) Spatial Relation Loss

  • For prompts with a specified 2D relation, CCG Diffusion detects relation phrases linking subject and object nouns from the CCG derivation and guides their attention centroids.The listed phrases include on the left/right/top/bottom of, next to, near, and on side of.Source: 3) Spatial Relation Loss · p. 8
  • The spatial loss uses signed centroid differences along and across the relation’s axis to enforce directional margins or proximity and penalize overlap.Because centroids lie on the unit square, the margin and overlap parameters are fractions of image size and independent of resolution.Source: 3) Spatial Relation Loss · p. 8

4) Count Loss

The Count Loss guides noun attention toward as many separated regions as the specified count, while PP ambiguity resolution uses an intended interpretation to obtain a corresponding CCG derivation. These mechanisms extend Predicated Diffusion using information from the parse.

4) Count Loss

  • The Count Loss targets as many mutually separated attention regions as the specified object count, addressing the existence proposition’s inability to enforce a number of objects.The target distribution is built once from highly attended, sufficiently separated locations before the latent update, then held fixed; symmetric KL penalizes missing target peaks and attention away from peaks.Source: 4) Count Loss · pp. 8–9
  • Count constraints pair each noun modified by a cardinal number from one to ten with that number, and the noun’s attention is pulled toward a target with that many peaks.A weight prevents weakly attended nouns from being pushed too strongly.Source: 4) Count Loss · pp. 8–9

C. PP AMBIGUITY RESOLUTION

  • For PP ambiguity resolution, CCG Diffusion takes the intended NP or VP attachment, obtains its derivation, and uses that syntax to control generation toward the selected interpretation.Figure 4 illustrates the alternative derivations obtained by fixing the supertag of “with.”Source: C. PP AMBIGUITY RESOLUTION · p. 9, Figure 4 · p. 9

1) Supertag-Constrained Parsing

Fixing a preposition’s supertag lets CCG parsing produce the derivation for a selected PP attachment. CCG Diffusion then rewrites the prompt and assigns holder–object regions to reflect that interpretation; evaluation uses T2I-CompBench++ and two distinct backbones.

1) Supertag-Constrained Parsing

  • Fixing the preposition’s supertag to the one associated with NP or VP attachment lets the CCG parser derive the selected interpretation.For “with,” (NP\NP)/NP selects NP attachment, while ((S\NP)\(S\NP))/NP selects VP attachment.Source: 1) Supertag-Constrained Parsing · p. 9

2) Disambiguating Paraphrase and Holder–Object Region Assignment

  • For the selected attachment, the prompt is rewritten as an unambiguous sentence and holder–object guidance assigns the holder and PP object to the same region.The holder changes with the interpretation: it is the modified noun for NP attachment and the subject for VP attachment; the unselected holder goes to the other region.Source: 2) Disambiguating Paraphrase and… · pp. 9–10

1) Dataset

  • T2I-CompBench++ evaluates attribute binding, object relationships, numeracy, and complex compositions across 8,000 prompts in four categories and eight subcategories.Each subcategory has 1,000 prompts; the official test split has 300 prompts, with ten seeds per prompt.Source: 1) Dataset · p. 10

2) Backbone Image Generation Models and CCG Parser

  • The experiments use Stable Diffusion v1.4 and Sana 0.6B, whose U-Net/CLIP and DiT/Gemma 2 architectures differ.Using Sana tests whether the method is tailored to a particular backbone model.Source: 2) Backbone Image Generation Models… · p. 10

3) Compared Methods

The section compares CCG Diffusion with prompt-only training-free alternatives and reports gains on its targeted compositional challenges. The contribution of individual components varies between Stable Diffusion and Sana.

3) Compared Methods

  • The comparison includes five prompt-only training-free methods spanning conditioning and attention guidance, with and without syntactic parsing.The set comprises the four methods compared in the Predicated Diffusion paper plus Predicated Diffusion itself.Source: 3) Compared Methods · pp. 10–11
  • The baselines cover denoiser-condition composition, constituency-based phrase encoding, noun-attention boosting, dependency-based modifier alignment, and dependency-derived predicate guidance.Predicated Diffusion is the base method, so its comparison isolates the effect of CCG-based extensions.Source: 3) Compared Methods · p. 11
  • The authors adapted each compared method to Sana, using the same inputs and generation settings as for CCG Diffusion.None of the compared methods provided a Sana implementation.Source: 3) Compared Methods · p. 11

B. RESULTS

  • CCG Diffusion significantly outperforms Predicated Diffusion on both backbones across targeted attribute-binding, 2D-spatial, and numeracy challenges.Attribute binding leads among compared methods on both backbones; 2D-spatial scores significantly exceed all other methods, while numeracy gains over Predicated Diffusion are small but significant.Source: B. RESULTS · p. 11
  • In attribute-binding, 2D-spatial, and numeracy categories, samples show compared methods failing to depict attribute binding, spatial relations between objects, or object counts, while our method succeeds.Source: B. RESULTS · p. 12

C. ABLATION STUDY

The ablation study finds that component contributions differ across backbones. Removing targeted components reduces performance, with the size and significance of the effects varying by task and backbone.

1) Attribute Binding

  • Disabling any one of the three attribute-binding components lowers scores in every category on both backbones; removing Selective Guidance Boost or Positional Prefix significantly lowers them.The three components are Selective Guidance Boost, Positional Prefix, and Region Assignment Loss.Source: 1) Attribute Binding · p. 12
  • Attention-guidance components tend to contribute more on Stable Diffusion, whereas Positional Prefix tends to contribute more on Sana.The authors suggest text-encoder representational capacity as one possible explanation: Sana’s encoder can readily represent placement instructions and within-phrase attribute binding, while Stable Diffusion’s text encoder may represent these relations poorly.Source: 1) Attribute Binding · p. 12

2) 2D-Spatial and Numeracy

  • Removing Spatial Relation Loss substantially lowers 2D-spatial scores on both backbones, with a larger drop on Stable Diffusion.The drop remains large on Sana; the authors suggest that explicitly constraining object locations is likely effective because depicting relation direction remains difficult even with a strong text encoder.Source: 2) 2D-Spatial and Numeracy · p. 12
  • Removing Count Loss causes a small but significant numeracy drop on Stable Diffusion, but no significant drop on Sana.The authors suggest attention manipulation alone is likely insufficient for count control: dividing a noun’s attention among regions does not guarantee the specified number of objects, and changing object count is a larger structural change than moving objects.Source: 2) 2D-Spatial and Numeracy · pp. 12, 14

VI. EXPERIMENT 2: PP AMBIGUITY RESOLUTION

The evaluation tests whether CCG Diffusion renders specified PP-attachment interpretations as distinct images. It significantly outperforms all compared methods on both backbones under both automatic evaluators, including methods supplied with the gold visual setup.

A. SETUP

  • The evaluation uses 74 ambiguous prompts, five seeds per interpretation, and the proportion of images judged “Yes” by each VQA model as its metric.Two independent LLMs evaluate the images using the TAB intention questions.Source: A. SETUP · p. 14

B. RESULTS

  • Across all trials, the two VQA models achieve Cohen’s κ = 0.72 and 87.7% agreement, with κ = 0.65 on Stable Diffusion and 0.79 on Sana.The authors conclude that their findings are unlikely to depend on the judgments of one particular VQA model.Source: B. RESULTS · p. 14
  • On both Stable Diffusion and Sana, CCG Diffusion significantly outperforms every compared method, including gold-visual-setup methods, in both LLM evaluations.The authors attribute the result to combining the Disambiguating Paraphrase and Region Assignment Loss derived from the specified interpretation.Source: B. RESULTS · p. 14, Table 5 · p. 15
  • In the illustrated examples, compared methods fail to match the relation in each visual setup, while CCG Diffusion produces a matching image for each setup.Additional examples are provided in the appendix.Source: B. RESULTS · p. 15, Figure 6 · p. 15

C. ABLATION STUDY

The ablation study finds lower Yes rates when either ambiguity-resolution component is disabled, though statistical significance varies by backbone and component. The conclusion also identifies limits in supported prompt structures and generation speed.

C. ABLATION STUDY

  • On both backbones, disabling either component lowers the Yes rate; significance varies, with both components disabled significant on Stable Diffusion and the paraphrase or both disabled significant on Sana.The ablation varies the Disambiguating Paraphrase and Region Assignment Loss; Table 6 reports Yes-rate drops relative to the configuration with both enabled.Source: C. ABLATION STUDY · p. 15, Table 6 · p. 16
  • The authors suggest encoder capacity may explain backbone differences: the paraphrases differ mainly in word order and syntax, which Gemma captures more readily than CLIP.They suggest the Disambiguating Paraphrase alone may suffice on Sana, while both components may contribute on Stable Diffusion; disabling Region Assignment Loss is not significant on either backbone.Source: C. ABLATION STUDY · p. 16

VII. CONCLUSION

  • The conclusion presents CCG Diffusion as a training-free method using CCG derivations for prompt rewriting and attention guidance across compositional generation and syntactic ambiguity resolution.It reports significant gains over baselines on benchmark datasets except in some categories, and effectiveness across two backbones.Source: §VII · p. 16
  • The method has no mechanism for controlling prompts involving depth relations, actions, or multiple constraints; extensions using depth cues and verb predicate–argument structure are future work.Source: §VII · p. 16
  • The addressed ambiguity type is limited to PP attachment; other types such as coordination scope require additional derivation constraints and ambiguity-specific paraphrases and guidance.Gradient-based attention guidance also makes generation 2.2–2.4 times as long as vanilla generation.Source: §VII · p. 16

APPENDIX A IMPLEMENTATION DETAILS

The appendix describes generation hyperparameters, conditions for applying components, and experimental hardware. It also specifies count-loss construction and update-size controls.

A. GENERATION SETTINGS AND HYPERPARAMETERS

  • The update step size decreases from s to s/2 over denoising, and an update is scaled down if its L2 norm exceeds δ times the latent’s L2 norm.The guidance strength is the pair (s, δ); prompts qualifying for Selective Guidance Boost use the boosted pair.Source: A. GENERATION SETTINGS AND… · p. 16
  • The Count Loss uses a normalized noun-attention map and a target formed from equally weighted Gaussian peaks; it is added only when counted nouns’ attention is neither vanishing nor flat.The implementation selects peak centers from strongly attended pixels and keeps the peaks nonoverlapping.Source: A. GENERATION SETTINGS AND… · pp. 16–17

B. CONDITIONS FOR APPLYING THE COMPONENTS

  • Components are applied only when their corresponding structures can be read from the derivation; prompts with cardinal numbers, 2D relations, or depth words use standard guidance strength.The Positional Prefix and Region Assignment Loss apply to specified pairs of nonoverlapping noun phrases joined by “and,” when neither Count Loss nor Spatial Relation Loss applies.Source: B. CONDITIONS FOR APPLYING THE… · p. 17

C. HARDWARE

  • Compositional-generation experiments ran on a DGX Station A100 with four A100 GPUs, while ambiguity-resolution image generation ran on a machine with one RTX 4090 GPU.Automatic Qwen evaluation used two DGX Spark systems.Source: C. HARDWARE · p. 17

D. COMPUTATIONAL COST

CCG Diffusion increases generation time relative to vanilla backbones and slightly relative to Predicated Diffusion. The appendix also points to further ablation results and qualitative examples across task categories and backbones.

D. COMPUTATIONAL COST

  • Generation takes about 2.2 times as long as vanilla on Stable Diffusion and 2.4 times on Sana, but only about 4% and 1% longer than Predicated Diffusion, respectively.Most of the extra time over vanilla comes from gradient computation and latent updates; parsing adds about 1–3% of total pipeline runtime.Source: D. COMPUTATIONAL COST · p. 17

APPENDIX B FULL RESULTS OF THE ABLATION STUDY ON COMPOSITIONAL TEXT-TO-IMAGE GENERATION

  • Table 9 reports score changes when two or more components are disabled in the attribute-binding ablation on T2I-CompBench++.The appendix describes the score drops relative to the configuration with all three components enabled.Source: Appendix B · p. 17

APPENDIX C ADDITIONAL QUALITATIVE EXAMPLES

  • Additional Stable Diffusion and Sana examples cover color, shape, texture, 2D-spatial, numeracy, 3D-spatial, non-spatial, complex, and PP-ambiguity cases.The appendix identifies Figs. 7–10 for compositional categories and Fig. 11 for PP ambiguity.Source: Appendix C · p. 17, Figure 7 · p. 18, Figure 8 · p. 19

Related papers

Loading 2610.29415v1…