PaperlayerView exact revision on arXiv

Source-linked AI summary

OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation

Ziye Li, Henghui Ding

arXiv:2605.21343v1cs.CV

TL;DR

Layout-to-image models lack explicit Z-order modeling, leaving overlapping objects ambiguous and often producing inconsistent layering. OcclusionFormer addresses this with an occlusion-aware framework and SA-Z dataset, achieving state-of-the-art occlusion control and visual fidelity.

  • Problem

    Existing layout-to-image methods lack explicit Z-order modeling, making overlapping instances ambiguous and undermining compositional intent.

  • Method

    The paper introduces SA-Z and OcclusionFormer, which decouples instances, models Z-order through volumetric rendering, and uses queried alignment for spatial precision.

  • Results

    OcclusionFormer establishes a new state-of-the-art, outperforming existing baselines in complex occlusion resolution and semantic integrity.

  • Takeaways & Limitations

    The method significantly improves occlusion accuracy and visual fidelity across the evaluated benchmarks.

Abstract

from arXiv · show

Recent layout-to-image models have achieved remarkable progress in spatial controllability. However, they still struggle with inter-object occlusion. When bounding boxes overlap, most existing methods lack explicit occlusion information, which makes the generation in intersection regions inherently ambiguous and hinders the determination of complex occlusion relationships. As a result, they often produce entangled textures or physically inconsistent layering in the overlapped areas. To address this issue, we first construct SA-Z, a large-scale dataset enriched with explicit occlusion ordering and pixel-level annotations. Building upon our proposed dataset, we introduce OcclusionFormer, a novel occlusion-aware Diffusion Transformer framework that explicitly models Z-order priority by decoupling instances and compositing them via volume rendering. Furthermore, to ensure fine-grained spatial precision, we introduce a queried alignment loss that explicitly supervises individual instances and enhances semantic consistency. The proposed method effectively reduces ambiguity in overlapping regions, enforces correct occlusion dependencies, and preserves structural integrity, leading to substantial accuracy gains across diverse scenes.

1. Introduction

Layout-to-image generation offers precise spatial control but remains ambiguous when objects overlap because existing methods lack explicit Z-order modeling. The paper addresses this gap with the SA-Z dataset, OcclusionFormer’s instance-decoupled volumetric rendering, and a real-world occlusion benchmark.

  • Motivation: Layout-to-image methods use bounding boxes or image signals to control object locations and scales with high spatial precision.This supports applications requiring strong structural fidelity, including complex scene composition.
  • Motivation: Most existing methods overlook inter-object occlusion, treating overlapping boxes as feature mixtures without explicit depth priorities.Unlike graphics pipelines using a Z-buffer, they lack an explicit Z-order and struggle with ambiguous intersections.
  • Motivation: Users provide amodal bounding boxes and expect intended Z-order interactions, but existing models may interpret overlaps as conflicting conditions and shrink objects.This mismatch reflects the gap between the user workflow and models without explicit Z-order modeling.
  • Contributions: SA-Z provides pixel-level captions, explicit Z-order annotations, and amodal annotations derived through SAM-3D 3D reconstruction.The dataset is designed to provide data-driven explicit supervision for occlusion-aware generation.
  • Contributions: OcclusionFormer decouples instances and uses volumetric rendering to model occlusion dependencies, with queried alignment loss supervising individual instances.Transmittance calculation resolves depth dependencies, while the alignment loss enhances instance-level semantic consistency.
  • Contributions: The authors curate a challenging real-world occlusion benchmark from SA-Z to address OverLayBench’s reliance on synthetic images.The benchmark serves as a rigorous testbed for complex occlusion and reduces the synthetic-to-real domain gap.

2. Related Works

Related work spans training-free and training-based layout-to-image methods, as well as datasets for spatial control. Existing methods and datasets improve layout fidelity but remain limited by unstable inference, flattened 2D representations, and insufficient modeling of Z-axis occlusion and invisible object parts.

  • Training-free Methods: Training-free methods enforce spatial constraints at inference by manipulating attention maps, but their heuristic gradients or latent edits make them unstable and hyperparameter-sensitive.LaRender additionally applies volumetric rendering principles to simulate occlusion control.
  • Training-based Methods: Training-based methods add trainable modules to diffusion backbones, typically improving fidelity and stability over training-free baselines by fusing box coordinates with visual features.Examples include U-Net-based GLIGEN and DiT-based Eligen and Creatilayout.
  • Layout-to-Image Datasets: SA-Z provides high-resolution, open-vocabulary, and 3D-aware annotations with rich geometric constraints compared with prior datasets.The dataset comparison defines high resolution as an image long edge greater than 1000px.
  • Layout-to-Image Datasets: Recent layout datasets expand data volume and annotation richness, yet remain in the 2D plane and overlook Z-axis occlusion and invisible object parts needed for dense layouts.COCOA and InstaOrder provide Z-orders or amodal masks, but are specialized datasets.

3. Method

OcclusionFormer constructs occlusion-aware generation by decoupling instances, arranging them in explicit Z-order through volumetric rendering, and enforcing spatial alignment with queried mask supervision. Its training combines rectified flow matching with alignment control.

  • Dataset Curation: The curation pipeline augments SACap-1M with instance-specific captions, occluder annotations, and global prompts to support semantic precision and resolve occlusion ambiguity.Captions are generated strictly from each mask area, while Z-order and captions are annotated through InstaOrder and DescribeAnything.
  • Instance Decoupling: Instance decoupling replaces indiscriminate global attention with local bounding-box attention between each instance’s visual features and its text embedding.The model freezes the pretrained backbone and optimizes only LoRA layers, so each instance is modulated exclusively by its semantic description before composition.
  • Z-Order Rendering: Volumetric rendering composes independently controlled instances along rays ordered by occluders, using transmittance to suppress features from objects covered by dense foreground instances.Instance opacity is restricted to its bounding box and adaptively adjusted across diffusion stages; unconstrained overlaps use normalized weighting or simple averaging.
  • Queried Alignment: Queried alignment derives dynamic instance queries, predicts spatial masks from visual similarity maps, and applies cross-entropy supervision using SA-Z masks.This supervision encourages features to remain semantically consistent with object regions and aligned with spatial geometry.
  • Training Objectives: The overall objective combines rectified flow matching with spatial alignment control using a balancing coefficient λ = 0.5.The coefficient is selected to enforce geometry constraints without compromising the pretrained backbone’s visual quality.

4. Experiment

Experiments evaluate OcclusionFormer on synthetic OverLayBench and realistic SA-Z Eval using spatial, occlusion, and depth-order metrics. Results show that explicit Z-order modeling, instance decoupling, queried alignment, and amodal supervision improve overlap handling and layout fidelity.

  • Experimental Setup: Evaluation uses OverLayBench and a held-out SA-Z Eval of 1,000 difficult images with high instance counts and complex occlusion patterns.Metrics cover spatial precision through mIoU and O-mIoU, occlusion through F1, and depth order through WHDR.
  • Qualitative Analysis: Qualitative comparisons show that OcclusionFormer preserves distinct instances and correct occlusion dependencies, while baselines produce object fusion or incorrect Z-order in dense overlaps.The comparison covers OverLayBench and SA-Z Eval.
  • Quantitative Results: OcclusionFormer establishes state-of-the-art O-mIoU, Occ., and Dep. performance on both OverLayBench and SA-Z Eval.The method explicitly models Z-order through volumetric rendering and suppresses background features in overlapping regions using occluder transmittance.
  • Ablation Study: Reverting to global attention causes the most severe performance collapse across both subsets, confirming instance decoupling prevents feature entanglement.The ablation consistently lowers mIoU and Occ. metrics on both evaluation settings.
  • Ablation Study: Removing occlusion conditioning lowers occlusion accuracy, while removing queried alignment or amodal data harms overlap fidelity and geometric learning across datasets.A naive attention-map BCE loss performs worse than the queried loss, and full amodal shapes provide geometric signals for occlusion learning.

5. Conclusion

The paper addresses inter-object occlusion in layout-to-image generation by introducing SA-Z and OcclusionFormer, which explicitly models Z-order and improves spatial precision.

  • SA-Z is a large-scale dataset enriched with explicit Z-order annotations for inter-object occlusion in layout-to-image generation.
  • OcclusionFormer models Z-order through volumetric rendering to resolve ambiguities along the Z-axis.
  • Queried alignment is introduced to ensure spatial precision in the occlusion-aware framework.
  • Extensive benchmark evaluations demonstrate that OcclusionFormer establishes a new state-of-the-art.

Impact Statement · Appendix for: OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation

OcclusionFormer improves layout-to-image generation through explicit Z-order control and enhances structural fidelity in dense overlapped scenes. The appendix illustrates predicted-mask progression during denoising across 28 timesteps.

  • Impact Statement: OcclusionFormer improves layout-to-image generation through explicit Z-order control.This capability is presented as the paper’s central contribution.
  • Impact Statement: The model significantly enhances structural fidelity in dense overlapped scenes.The stated benefit specifically concerns scenes with substantial object overlap.
  • Impact Statement: OcclusionFormer offers practical benefits for creative design workflows.The impact statement identifies creative design as one application area.
  • Impact Statement: The method supports generation of high-quality synthetic training data.This is identified as a practical benefit alongside creative design workflows.
  • Impact Statement: The paper acknowledges standard generative-AI risks, including inherited biases from the pre-trained backbone.The impact statement also notes potential misuse as a general risk.
  • Appendix for: OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation: Figure 9 shows the progression of predicted masks during the denoising process.The figure uses a total of 28 timesteps.

A. More Implementation Details

The implementation derives instance-specific control parameters with adaptive projections, refines spatial similarity maps using a lightweight CNN, and applies time-dependent mask supervision with staged denoising guidance.

  • Conditioning Projections and Softplus Activation: An adaptive projection module processes time-dependent text embeddings with SiLU and parallel Linear layers for semantic alignment features and instance density prediction.One branch projects the semantic query vector q_i for cosine-similarity-based spatial alignment, while the other predicts the raw instance density.
  • Mask Predictor Architecture: The mask predictor is a lightweight CNN that refines a single-channel similarity map into foreground-background probability logits.Its layers use 3 × 3 convolutions with 1 →32 and 32 →16 channels, followed by GELU activations, then a 1 × 1 convolution producing 2-channel logits.
  • Training and Inference Strategy: Training uses amodal masks at high noise levels, such as t ∈[700, 1000], and modal masks for remaining steps, such as t < 700.This curriculum first establishes global structure for occlusion learning, then refines precise visible boundaries.
  • Training and Inference Strategy: Inference uses a 28-step denoising schedule with layout guidance activated exclusively during the initial 30% of denoising.The strategy follows foundation work by Li et al. (2023).

B. Investigation of Predicted Masks

The predicted foreground masks evolve during denoising from coarse, amodal instance footprints toward sharper masks aligned with fine-grained object boundaries. Figure 9 visualizes this progression at the first single-stream block.

  • Mask evolution: Figure 9 tracks predicted foreground probability maps throughout the denoising process.The maps are examined at the first single-stream block.
  • Early stages: At early denoising stages, the mask predictor captures coarse, amodal instance footprints that roughly fill each bounding box.The masks focus on the overall spatial extent rather than precise object contours.
  • Final stages: As denoising approaches its final steps, the predicted masks become sharper and conform to fine-grained object boundaries.This refinement progressively improves boundary precision over the denoising trajectory.

C. More Qualitative Comparison

Additional visual comparisons evaluate OcclusionFormer on OverLayBench and SA-Z Eval, highlighting its handling of dense overlaps and preservation of instance boundaries relative to previous methods.

  • OverLayBench: Figure 10 shows OcclusionFormer handling dense overlaps and preserving instance boundaries on OverLayBench compared with previous methods.The comparison is presented as evidence of the method’s qualitative superiority over existing state-of-the-art methods.
  • SA-Z Eval: Figure 11 presents additional OcclusionFormer results on the SA-Z Eval benchmark.The passage identifies SA-Z Eval as the benchmark used for this qualitative comparison but does not state a specific outcome.

D. More Ablation Results · E. User Study

The ablations show that instance decoupling, explicit Z-order modeling, alignment components, and amodal supervision each support spatial accuracy and occlusion consistency, especially in complex scenes. A user study found that OcclusionFormer received the highest ratings across all four evaluated dimensions.

  • D. More Ablation Results: Table 4 evaluates the full ablation study on OverLayBench and SA-Z Eval, using Flux.1-dev as a lower bound for spatial metrics.Flux has high CLIP-G and low synthesized-dataset FID on OverLayBench, but its FID degrades on real-world SA-Z Eval.
  • D. More Ablation Results: Removing instance decoupling causes the most severe degradation, with Complex mIoU falling from 0.6037 to 0.5177 and Occ. from 0.7797 to 0.6109.The results support decoupling as foundational for preventing feature entanglement and preserving distinct instance identities.
  • D. More Ablation Results: Explicit Z-order modeling matters most under dense overlap: Complex Occ. drops roughly 5.3% without occlusion conditioning, while Simple mIoU remains 0.7385 versus 0.7405.The ablation indicates that implicit learning is sufficient for simple layouts, whereas explicit volumetric rendering resolves intricate occlusion relationships.
  • D. More Ablation Results: Removing Learned Sigma decreases O-mIoU, while removing Queried Loss reduces SA-Z Eval SRE from 0.8158 to 0.8024.Learned density enables dynamic opacity modulation, and queried alignment loss improves boundary precision.
  • D. More Ablation Results: Without amodal annotations, performance degrades in O-mIoU, Occ., and Dep. across Complex subsets and SA-Z Eval despite competitive FID scores.Amodal supervision provides geometric signals for learning correct occlusion dependencies and maintaining occluded-object structure.
  • E. User Study: In a study of 15 participants and 300 samples, evaluators ranked seven methods on Occ., Layout Align, Local Fidelity, and Global Align using normalized scores from 1/7 to 1.The samples came from the OverLayBench Complex subset and SA-Z Eval benchmark.
  • E. User Study: OcclusionFormer achieved the highest user-study ratings across Occ., Layout Align, Local Fidelity, and Global Align.The result confirms the method’s superiority in human evaluation across all four reported dimensions.
  • E. User Study: Inference speed scales linearly with the number of objects on an NVIDIA A800 GPU, supporting efficiency in dense scenes.The efficiency analysis varies the number of objects during inference.

F. Efficiency Analysis … I. Examples and Statistics of the SA-Z Eval

The paper analyzes efficiency, documents caption noise in SACap-1M, and presents SA-Z’s detailed occlusion annotations. It also introduces SA-Z Eval as a semantically broad benchmark spanning 749 categories and varying scene complexity.

  • F. Efficiency Analysis: F. Efficiency Analysis: Generation speed has a linear relationship with object count, while inference time increases as scenes become more cluttered.The efficiency evaluation uses a single NVIDIA A800 GPU, and computational cost is correlated with scene complexity because local features are processed through instance decoupling.
  • G. Illustration of Noise in SACap-1M: G. Illustration of Noise in SACap-1M: Box-based prompting can misattribute background or neighboring-instance features because rectangular boxes poorly align with irregular object shapes.SACap-1M generates regional captions with Qwen2-VL-72B using bounding box coordinates.
  • G. Illustration of Noise in SACap-1M: G. Illustration of Noise in SACap-1M: Mask-level annotation with DescribeAnything filters context-induced noise and yields cleaner, more detailed captions tied to specific instances.The method constrains visual analysis to segmented regions, with noise highlighted by blue marks in Figure 13.
  • H. Examples of SA-Z: H. Examples of SA-Z: SA-Z provides detailed mask-area annotations and pairwise instance occlusion relationships, including amodal annotations for occluded instances extracted with SAM-3D.Figure 14 shows examples sampled from the training dataset.
  • I. Examples and Statistics of the SA-Z Eval: I. Examples and Statistics of the SA-Z Eval: SA-Z Eval contains 749 distinct categories and varying instance densities per image, providing semantic breadth and scene complexity.Figure 16 summarizes instance distributions per image and semantic diversity across categories.
  • I. Examples and Statistics of the SA-Z Eval: I. Examples and Statistics of the SA-Z Eval: To align with OverLayBench, the benchmark uses Qwen-VL-32B to generate semantic labels and filters nonsalient objects from SA-1B.The examples are presented in Figure 15.
Loading 2605.21343v1…