PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic Scene Program Diffusion Transformer

Chika Maduabuchi

arXiv:2609.31780v1cs.CVcs.LG

TL;DR

Text-to-image models still struggle to preserve compositional structure such as instance identity, attribute ownership, counts, and relations. PSP-DiT jointly denoises image and panoptic scene-program latents, with grounding and cycle-consistency objectives. Its reported gains are concentrated in counting, attribute binding, and role-sensitive relations.

  • Problem

    Text-to-image systems still struggle with compositional faithfulness on prompts requiring instance identity, attribute ownership, counting, spatial ordering, and role-sensitive relations.

    Source: §1 · p. 1

  • Method

    PSP-DiT jointly denoises image and instance-indexed panoptic scene-program latents, using grounding and cycle consistency to tie structure to visual support.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The largest improvements appear on counting, attribute binding, and role-sensitive relations.

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    Across matched prompts, seeds, and inference budgets, examples show better preservation of counts, attribute ownership, roles, spatial ordering, and visual support than flat text conditioning alone.

    Source: §4.9 · p. 8

    This work focuses on text-to-image generation; extending panoptic scene-program latents to temporally consistent text-to-video generation is left to future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Modern text-to-image models produce high-fidelity images but still struggle with compositional prompts that require instance identity, attribute ownership, counting, spatial ordering, and role-sensitive relations. We introduce Panoptic Scene Program Diffusion Transformer (PSP-DiT), a diffusion-transformer architecture that treats a panoptic scene program as a first-class latent variable rather than an external control signal or post-hoc parse. PSP-DiT jointly denoises image latents and scene-program latents through coupled transformer streams, while panoptic grounding and cycle-consistency objectives tie object instances, attributes, relations, and counts to visual support in the generated image. Under matched training and inference settings, PSP-DiT improves over a strong flat-text baseline across GenEval 2, SANEval-Simple, PSG-Score, and DetailMaster, with the largest gains on counting, attribute binding, role-sensitive relations, and long structured prompts. The method preserves image quality, adds modest inference overhead, and remains robust to imperfect scene programs.

1 Introduction

Text-to-image systems still struggle with compositional faithfulness when prompts require tracking instance identity, attribute ownership, ordering, and interactions. PSP-DiT addresses the gap by making an instance-indexed panoptic scene program a jointly denoised latent alongside the image.

1 Introduction

  • Compositional failures persist on multi-entity, detail-heavy prompts that require tracking identity, ownership, ordering, and interaction structure.The remaining bottleneck is not only stronger text encoders or more data, but the absence of an explicit scene state inside generation.Source: §1 · p. 1
  • Prior approaches use structure as external control, separate graph generation from image generation, or study panoptic scene graphs primarily for recognition.These approaches do not make panoptic scene graphs first-class latent variables in modern text-to-image generators.Source: §1 · pp. 1–2
  • PSP-DiT jointly denoises an instance-indexed panoptic scene program with image latents, representing objects, attributes, relations, counts, and visual ownership during generation.Its coupled transformer streams and panoptic grounding and cycle-consistency objectives connect structural faithfulness to visual support.Source: §1 · p. 2

2 Related Works

Prior work diagnoses compositional errors, steers pretrained models, or uses structure as conditioning, preprocessing, or supervision. PSP-DiT instead makes a grounded scene program part of the jointly denoised generative state, addressing a bottleneck that stronger backbones alone leave open.

2 Related Works

  • Recent benchmarks and analyses diagnose persistent failures in counting, attribute binding, positioning, relations, and higher-order spatial composition after generation.The paper frames its complementary question as how to represent identity, ownership, count, and relations during denoising.Source: §2 · p. 2
  • Inference-time guidance, editing, grounded generation, and alignment methods steer pretrained diffusion models or fine-tune them for compositionality.These approaches include region-wise guidance, VLM feedback, and preference optimization or reinforcement learning.Source: §2 · p. 2
  • Unlike approaches that patch compositional errors externally, PSP-DiT jointly denoises image and panoptic scene-program latents to represent composition explicitly.Its scene program is maintained through denoising and tied back to the image through scene decoding and cycle consistency.Source: §2 · p. 3
  • Structured-generation work uses scene graphs, layouts, layered decompositions, or panoptic supervision, typically as conditioning, preprocessing, supervision, or annotation.PSP-DiT instead samples the panoptic scene program jointly with the image and grounds it through scene decoding and cycle consistency.Source: §2 · p. 3
  • Scaling image quality and compositional reasoning are not the same problem; PSP-DiT presents a jointly denoised scene-program latent as an orthogonal generative primitive.The method is instantiated on a modern DiT+RAE stack, while its core contribution targets the structured reasoning bottleneck left open by backbone advances.Source: §2 · p. 3

3 Method

PSP-DiT makes a panoptic scene program a jointly denoised latent alongside the image, representing instance-level structure explicitly and grounding it in visual ownership. Coupled transformer streams refine both states, while grounding objectives connect program variables to visual support.

3.1 Problem Setup and Design Goal

  • PSP-DiT jointly models an image latent and structured scene latent, addressing the implicit instance identity, attribute ownership, relations, counts, and visual support in flat text conditioning.The scene latent is refined with the image rather than supplied only as a fixed control or inferred after generation.Source: §3.1 · pp. 3–4

3.2 Panoptic Scene Programs as Latent Variables

  • The panoptic scene program represents object instances, instance-attached attributes, ordered relations, cardinalities, and scene context.Repeated categories remain distinct, with attributes and relations resolved at the instance level.Source: §3.2 · p. 4
  • The scene program is a generated latent state grounded spatially in the image, not a fixed control graph.The overview depicts coupled image/program streams with bidirectional cross-attention and grounding and cycle-consistency heads tied to visual ownership and recoverable structure.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3 PSP-DiT: Joint Program–Image Denoising

  • The coupled denoiser updates image and scene-program tokens using within-stream self-attention and bidirectional cross-stream attention.Image tokens use the current scene state, and scene tokens use the current visual state; the structured state is sampled and refined during denoising.Source: §3.3 · pp. 4–5

3.4 Panoptic Grounding and Scene Ownership

  • Panoptic grounding predicts instance ownership fields and coarse depth or occlusion ordering from the coupled image–program state.These variables constrain repeated-object separation, attribute ownership, and agreement between relations and induced geometry.Source: §3.4 · p. 5

Appendix A.9 gives the explicit training procedure. We optimize

Training optimizes image and scene-program denoising together with grounding, alignment, and cycle-consistency objectives. At inference, PSP-DiT jointly samples image and program latents from a prompt, without test-time search, reranking, or external optimization.

Appendix A.9 gives the explicit training procedure. We optimize

  • The training terms make the image denoisable, the scene program generative and spatially grounded, and the generated image recoverable as the intended scene.The losses include ownership and ordering predictions, instance-to-image evidence alignment, and a frozen recognizer/parser cycle objective.Source: Appendix A.9 · p. 5

3.6 Inference and Relation to Prior Conditioning Methods

  • At inference, PSP-DiT initializes noisy image and program latents from a prompt, then jointly samples them across denoising steps and decodes the final image and scene explanation.The program latent is seeded by a prompt-to-program prior, but is sampled rather than fixed.Source: §3.6 · p. 5
  • Unlike flat-text conditioning, fixed graph or layout conditioning, and external trajectory guidance, PSP-DiT internalizes scene structure and uses a direct forward sampler without test-time search or optimization.It models pθ(z, s | y), rather than only pθ(z | y).Source: §3.6 · p. 5

4 Experiments

The experiments evaluate whether jointly modeled scene-program latents improve compositional faithfulness while preserving image quality, and assess skill-level gains, component contributions, scaling, efficiency, and robustness. Under matched comparisons, PSP-DiT improves across the main compositional benchmarks, with gains concentrated in key compositional skills and supported by qualitative examples.

4.2 Main Compositional Benchmark Comparison

  • PSP-DiT improves over the matched flat-text baseline on GenEval 2, SANEval-Simple, PSG-Score, and DetailMaster.Table 1(a) reports the main compositional benchmark comparison; the experiments use matched backbones, training data, and inference settings where intended to isolate the scene-program latent.Source: §4.2 · pp. 6–7, Table 1 · p. 6, §4.1 · p. 6

4.5 Ablation of the Scene-Program Latent

  • Appending scene-program tokens yields only modest gains; the full improvement requires instance indexing, joint denoising, panoptic grounding, and cycle consistency.The ablation supports elevating scene structure to a first-class generative state rather than using it as extra conditioning.Source: Table 2 · p. 7, §4.5 · p. 7

4.3 Image Quality and Prompt Alignment

  • PSP-DiT matches or improves fidelity, improves prompt-alignment metrics, and achieves the strongest human preference without a compositionality–quality trade-off.The comparison uses a unified evaluation protocol.Source: §4.3 · p. 7

4.4 Breakdown by Compositional Skill

  • The largest gains occur in counting, attribute binding, and role-sensitive relations, where current text-to-image systems remain most fragile.Table 1(c) aggregates benchmark-native submetrics into skill scores.Source: §4.4 · p. 7

4.6 Scaling and Resolution Robustness

  • Across 0.5B–3B backbones and 512px–1024px generation, PSP-DiT improves compositional fidelity while maintaining strong alignment quality.The results indicate that scene-program latents remain useful as model capacity grows.Source: §4.6 · p. 7

4.7 Efficiency and Overhead

  • In the matched 1B H200 setup, scene-program latents add only modest memory and latency overhead while yielding a large compositional-fidelity improvement.The paper limits this systems comparison to a matched setup because measurements depend strongly on hardware and software configuration.Source: §4.7 · p. 8

4.8 Robustness to Program Noise

  • PSP-DiT degrades gracefully under parsing variation, node or edge dropout, and relation corruption, remaining stronger than the baseline under moderate perturbations.The program-noise tests are performed at inference time.Source: §4.8 · p. 8

4.9 Qualitative Results

  • Matched qualitative examples show better preservation of object counts, attribute ownership, subject–object roles, spatial ordering, and grounded visual support.The comparisons use matched prompts, random seeds, and inference budgets.Source: §4.9 · p. 8

5 Conclusion

PSP-DiT jointly denoises image and panoptic scene-program latents, improving structured faithfulness while preserving image quality with modest inference overhead. Its text-to-video extension, which would need temporally consistent scene state, remains future work.

5 Conclusion

  • Across benchmarks, ablations, scaling tests, and robustness analyses, PSP-DiT improves structured faithfulness while preserving image quality and adding modest inference overhead.The conclusion presents explicit scene-state generation as a promising direction for addressing remaining compositional failures.Source: §5 · p. 8
  • Extending panoptic scene-program latents to text-to-video, where scene state must remain temporally consistent across frames, is left to future work.Source: §5 · p. 8
  • Matched qualitative stress cases show PSP-DiT more consistently preserves instance identity, attribute ownership, ordered relations, and visual support than the baselines.The cases test counting, attribute binding, role-sensitive spatial reasoning, and long structured composition.Source: Figure 2 · p. 9

A Supplementary Material

The supplement is organized as an implementation- and reproduction-oriented resource for PSP-DiT.

A Supplementary Material

Appendix entry Page

The appendix contents list directs readers to supplementary methods, evaluation materials, and stress-case results.

Appendix entry Page

  • Sections A.1–A.5 cover training data and scene-program construction, architecture, optimization, compute, and evaluation protocol details.Source: Appendix entry Page · p. 17
  • Sections A.6–A.9 list the human preference study, parser and program-noise protocol, reproducibility checklist, and algorithmic details.The algorithmic details include PSP-DiT overview, training, and inference.Source: Appendix entry Page · p. 17
  • Sections A.10–A.12 provide evaluation references and systems context, skill aggregation details, and qualitative prompts with additional stress cases.Source: Appendix entry Page · p. 17

A.1 Training Data and Scene-Program Construction

PSP-DiT’s matched baseline uses the same image-caption pool, filtering, resolution schedule, and optimization budget, while PSP-DiT additionally receives a panoptic scene-program latent. Scene programs are parsed from prompts, grounded against images, and reconciled into training targets.

A.1 Training Data and Scene-Program Construction

  • For matched comparisons, Ours-FlatText and PSP-DiT share the image-caption pool, filtering, resolution schedule, and optimization budget; only PSP-DiT receives the scene-program latent.Source: §A.1 · p. 17
  • A frozen text-to-program parser extracts candidate instances, attributes, relations, cardinalities, and scene context from prompts.Source: §A.1 · p. 17
  • Frozen panoptic segmentation and open-vocabulary grounding produce ownership maps and ordering targets, which are reconciled with text structure by removing unsupported objects and duplicate mentions.Relations are retained only when both endpoints are grounded.Source: §A.1 · p. 17
  • Training examples are discarded when parsing yields no foreground instance or grounding assigns no object support, and foreground instances are capped at 16.The scene representation also keeps a fixed set of register/background tokens for context.Source: §A.1 · p. 17

A.2 Architecture and Hyperparameters

PSP-DiT uses coupled image and scene-program streams with a shared program-token interface across model sizes. Its scene-program tokens represent instances, relations, cardinality groups, and background/context.

A.2 Architecture and Hyperparameters

  • The 1B backbone uses width 2048, 24 transformer blocks, 16 attention heads, and MLP expansion ratio 4; 0.5B and 3B variants scale width and depth while keeping the program-token interface and losses fixed.All models use the same RAE latent interface as the matched Ours-FlatText baseline.Source: §A.2 · p. 18
  • The program stream allocates one token per object instance, relation edge, and cardinality group, plus a fixed bank of background/context tokens.The respective maximums are 16 object-instance, 32 relation, 8 cardinality, and 8 scene-context tokens; unused slots are masked from program and grounding losses.Source: §A.2 · p. 18
  • The final image and program denoising heads are linear projection heads applied to the coupled-stream outputs.Source: §A.2 · p. 18
  • Grounding and ordering heads predict ownership logits over the RAE latent grid and pairwise or coarse ordinal depth scores among instances.Both heads are trained only through structural losses and used at inference to decode the final scene explanation.Source: §A.2 · p. 18

A.3 Optimization, Loss Weights, and Training Schedule

The training setup fixes the optimization schedule and objective weights across PSP-DiT variants. Training duration and resolution differ by model size, while the main benchmark tables use a shared 50-step inference recipe.

A.3 Optimization, Loss Weights, and Training Schedule

  • Training uses AdamW with weight decay 0.03, bf16 mixed precision, gradient clipping at 1.0, and EMA decay 0.9999; the 1.0 × 10−4 learning rate warms up for 10k steps before cosine decay.Source: §A.3 · p. 18
  • The objective weights are λprog = 1.0, λpan = 1.0, λalign = 0.25, λcycle = 0.5, and λD = 0.1, fixed across variants and not tuned per benchmark.Ablations remove architectural components while keeping the remaining loss weights and optimization schedule unchanged.Source: §A.3 · p. 18
  • The 1B model trains for 1.2M steps at 512px, the 0.5B model for 800k steps at 512px, and the 3B model for 1.5M steps with a final 1024px phase.Main benchmark tables use the same 50-step inference recipe unless otherwise stated.Source: §A.3 · p. 18

A.4 Compute, Hardware, and Runtime Accounting

This appendix documents the compute and evaluation conditions, study protocols, implementation details, and related procedures used to reproduce and contextualize PSP-DiT’s experiments. It reports training-resource accounting and matched-setup inference measurements to support reproducibility.

A.4 Compute, Hardware, and Runtime Accounting

  • The main 1B training run used 64 H200 GPUs for 1.2M steps; the full project used about 1.1M H200 GPU-hours, versus 0.72M for main-paper experiments.Efficiency timing used one H200, batch size 1, 1024×1024 images, 50 denoising steps, FlashAttention, torch.compile, and VAE decode, averaged over 100 runs after 10 warmups. The 1B Ours-FlatText and PSP-DiT pipelines contain 12.8B and 13.4B total parameters, respectively.Source: §A.4 · pp. 18–19

A.5 Evaluation Protocol and Metric Details

  • Evaluation uses a unified pipeline, matched comparison settings, and a flat-text baseline sharing PSP-DiT’s backbone, data, schedule, and inference recipe.GenEval 2 uses four fixed seeds and four images per rewritten prompt without best-of-N or reranking; SANEval-Simple and DetailMaster Avg. are reported on their specified scales, alongside PSG-Score, FID-30K, HPS, ImageReward, and human preference.Source: §A.5 · p. 19

A.6 Human Preference Study

  • The blinded human-preference study compares models on 200 held-out PartiPrompts prompts using three independent raters per pair and majority-vote wins.This yields 600 judgments per comparison; ties are disallowed, model identities and left/right order are randomized, and confidence intervals use 10,000 prompt-level bootstrap resamples.Source: §A.6 · p. 19

A.7 Scene-Program Parser and Cycle Recognizer

  • A frozen prompt-to-program parser initializes scene latents and training targets, while a frozen image-to-scene recognizer supplies cycle loss only during training.The parser output is not a final oracle scene; inference jointly updates the program and image latents and needs no external cycle recognizer, reranker, or search loop. Robustness tests perturb programs through node/edge dropout, relation corruption, or paraphrase noise.Source: §A.7 · pp. 19–20

A.8 Reproducibility Checklist

  • The reproducibility checklist specifies a shared RAE interface, 50 inference steps, bf16 precision, FlashAttention, and model-specific resolutions.Main comparisons use 512px for 0.5B and 1B models and 1024px for the 3B model; the checklist also records benchmark and timing configurations.Source: §A.8 · p. 20

A.9 Algorithmic Details

  • The algorithms jointly denoise image and scene-program latents, predict ownership and ordering, and apply denoising, grounding, alignment, and cycle-consistency losses.At inference, the program prior initializes the noisy scene latent; both latents are iteratively updated before decoding the image and scene outputs.Source: §A.9 · p. 21

A.10 Public baseline positioning and systems anchor

  • The appendix’s public-baseline tables provide literature-based positioning, while the FLUX.1-DEV systems anchor is explicitly not directly comparable to H200 measurements.The external anchor reports 49.7 s/image and 32.8 GB peak VRAM on RTX A6000 for 1024×1024 generation with 50 steps; Table 4 scopes public open-baseline and frontier-reference subsets.Source: §A.10.1 · p. 21, §A.10.2 · p. 22, Table 5 · p. 22

A.11 Skill Aggregation Details

  • Skill scores map to benchmark-native subsets, are linearly rescaled to [0, 100], and contribute equally to the unweighted five-skill average.Prompt counts for SANEval-Simple and the GenEval 2 role-sensitive subset come from the released evaluation splits; the appendix also gives public prompt-count context for GenEval 2 and DetailMaster.Source: §A.11 · p. 22, Table 6 · p. 22

A.4 Compute, Hardware, and Runtime Accounting

  • Appendix A.4 reports hardware, precision, runtime setup, GPU counts and optimization steps for main training runs, total project GPU-hours, and experiment GPU-hours; Table 1 gives latency and VRAM under matched H200 inference.Source: NeurIPS Paper Checklist · p. 29
Loading 2609.31780v1…