PaperlayerView exact revision on arXiv

Source-linked AI summary

OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rank

Wenyang Hong, Yuan Wang, Yanbin Hao, Lanqing Xue, Ke Wang, Xiang Wang, Kuien Liu, Richang Hong

arXiv:2608.20932v1cs.CV

TL;DR

Bounding-box layouts control locations but omit occlusion order, leaving occlusion-aware generation vulnerable to incorrect layering and missing or confused instances. OccluRank adds one ordinal rank per instance and uses rank-conditioned interaction before aggregation, supported by geometry-derived data and comprehensive evaluation. It achieves the strongest results on most structured metrics while retaining competitive attribute consistency and image quality.

  • Problem

    Bounding-box layouts specify spatial extent but not relative occlusion order, limiting control in scenes with overlapping instances.

  • Method

    OccluRank adds one ordinal rank to each bounding box and uses rank conditioning with Order-aware Instance Interaction before feature aggregation.

  • Results

    OccluRank achieves the strongest results on most structured metrics, including Presence, Box mIoU, Color, Strict Pair, and Strict Image under three evaluators, while maintaining competitive FID and attribute consistency.

  • Takeaways & Limitations

    A single ordinal rank provides explicit visibility control without additional geometric inputs or specialized inference, while preserving a competitive balance across controllability and image quality.

Abstract

from arXiv · show

Layout-to-image generation enables explicit spatial control through bounding-box layouts, yet bounding boxes specify only instance locations and cannot represent their occlusion order. Existing methods may rely on additional geometric conditions, employ complex inference procedures, or aggregate independently constructed instance representations without explicitly modeling their occlusion-dependent interactions. We propose OccluRank, a simple and controllable occlusion-aware layout-to-image framework that augments each bounding box with only one ordinal rank. OccluRank encodes the user-specified occlusion order through lightweight rank-based conditioning and introduces an Order-aware Instance Interaction (OII) module to jointly update rank-conditioned instance representations before aggregation. This allows the specified order to guide information exchange among occluding instances without additional geometric inputs or specialized inference-time optimization. We further construct OccluLayout, a synthetic training dataset whose occlusion order and amodal annotations are derived directly from known scene geometry rather than estimated from partially occluded images using auxiliary prediction models. For comprehensive evaluation, we introduce OccluLayout-Bench, which uses multiple multimodal large language model evaluators to assess instance presence, spatial layout, attributes, and occlusion order, together with FID for overall image quality. Experiments show that OccluRank more reliably preserves target instances, follows specified layouts, and realizes desired occlusion relationships while maintaining comparable attribute consistency and overall image quality.

1 Introduction

OccluRank addresses the missing occlusion-order information in bounding-box layouts with ordinal rank conditioning and order-aware instance interaction. The paper also introduces geometry-derived supervision and a multi-axis benchmark for evaluating controllable occlusion-aware generation.

  • Bounding boxes specify instance locations but not relative occlusion order, causing incorrect layering, merged objects, or attribute confusion.
  • Existing approaches may isolate instance semantics, require complex geometric conditions, or use involved inference procedures without explicitly modeling occlusion-dependent interactions.
  • OccluRank adds one ordinal rank per bounding box and combines rank conditioning with an Order-aware Instance Interaction module for user-specified occlusion control.
  • OccluLayout derives occlusion order and amodal annotations from controllable 3D scene geometry, alongside fine-grained descriptions for each instance.
  • OccluLayout-Bench jointly evaluates instance presence, spatial layout, attributes, occlusion order, and image quality using multiple multimodal evaluators and FID.
  • Experiments report better preservation of target instances, layout adherence, and desired occlusion order with comparable attribute consistency and image quality.

2 Related Work

Related work spans layout-to-image conditioning, occlusion-aware generation, and datasets or evaluation protocols for visibility and amodal supervision. OccluLayout-Bench broadens evaluation beyond occlusion order to jointly assess generation quality and controllability.

  • Layout-to-Image Generation: Layout-to-image methods inject bounding-box layouts, instance features, regional descriptions, or attention constraints to control object placement and semantics.
  • Occlusion-Aware Layout-to-Image Generation: Occlusion-aware methods incorporate visibility order through denoising modifications, ordered branches, depth arbitration, or other training and inference mechanisms.
  • Occlusion-Aware Dataset and Evaluation: Existing occlusion-aware datasets provide amodal regions or pairwise visibility relations but may retain limited resolution, closed vocabularies, or insufficient instance descriptions.
  • Occlusion-Aware Dataset and Evaluation: OccluLayout-Bench jointly evaluates instance presence, layout, attributes, occlusion order, and image quality.

3 Method

OccluRank preserves separate spatially aligned instance features, injects ordinal ranks, and performs location-wise interaction before aggregation into an Instance Semantic Map. Region-focused denoising further emphasizes instance structures and occlusion boundaries during training.

  • Overview and Problem Formulation: OccluRank extends pretrained SDXL conditioning with a single ordinal rank specifying each instance’s intended front-to-back position.
  • Per-Instance Feature Construction: The method independently constructs spatially aligned instance feature maps before retaining their instance dimension for order-aware interaction.
  • Order-aware Instance Interaction: Rank embeddings are broadcast across valid locations within each instance’s bounding box, while features outside the box are set to zero.
  • Order-aware Instance Interaction: OII applies self-attention along the instance dimension at each latent location so representations adapt to competing objects and ordinal roles.
  • Order-aware Instance Interaction: Unlike reweighting-based aggregation, OII updates high-dimensional instance representations before fusion, allowing prescribed order to affect feature content rather than only aggregation weights.
  • Interaction-aware Aggregation and Injection: Updated features are aggregated into an ISM and residually injected into selected SDXL layers within the union of layout-conditioned regions.
  • Training Objective and Inference: The region-focused denoising objective normalizes error over the box-union region to strengthen supervision for instance structures and occlusion boundaries.

4 Dataset and Benchmark Construction

OccluLayout is constructed from controllable 3D scenes to provide geometry-derived occlusion supervision, while OccluLayout-Bench evaluates generation across presence, layout, attributes, occlusion order, and image quality.

  • OccluLayout Dataset: OccluLayout generates 1024 × 1024 scenes with two to five objects whose positions, scales, orientations, and viewpoints vary to create diverse occlusion patterns.Physically intersecting scenes are rejected, and amodal masks and boxes are obtained from complete scene geometry.
  • OccluLayout Dataset: The dataset derives amodal boxes and front-to-back order from complete projected geometry and camera-coordinate object positions.This provides annotations without relying on partially observed object regions.
  • OccluLayout Dataset: The resulting collection uses edited scenes and fine-grained instance descriptions while excluding inter-object and visibility relations from those descriptions.This prevents textual leakage of the target occlusion order.
  • OccluLayout-Bench: OccluLayout-Bench evaluates 1,000 held-out images containing 3,386 instances across presence, spatial adherence, attributes, occlusion order, and image quality.The benchmark addresses the limitation of evaluating occlusion-aware generation only through pairwise visibility order.
  • OccluLayout-Bench: Presence, Box mIoU, Color, Texture, Strict Pair, Strict Image, and FID measure complementary semantic, spatial, attribute, ordering, and quality outcomes.Multiple MLLM evaluators independently assess the semantic, spatial, attribute, and order metrics.

5 Experiments

Experiments evaluate OccluRank quantitatively and qualitatively on OccluLayout-Bench, showing stronger instance, layout, and occlusion-order control than comparison methods and ablations.

  • 5.1 Quantitative Comparison: OccluRank achieves the best Presence, Box mIoU, Color, Strict Pair, and Strict Image scores under all three MLLM evaluators.Its advantage is most pronounced on the strict occlusion metrics.
  • 5.1 Quantitative Comparison: 7.38%-10.61% higher Strict Pair and 7.77%-11.40% higher Strict Image scores than IFAdapter demonstrate stronger ordered occlusion control.OccluRank also consistently improves Presence and Box mIoU in this controlled comparison.
  • 5.1 Quantitative Comparison: OccluRank maintains a competitive FID of 62.746 while achieving the strongest results on most structured metrics.OcclusionFormer obtains the best FID of 57.124 but performs substantially worse on other metrics.
  • 5.2 Qualitative Analysis: Under identical overlapping layouts, OccluRank preserves requested objects, aligns them with designated regions, and realizes coherent visibility boundaries following the prescribed order.Competing methods often omit, merge, misplace, or incorrectly layer instances.
  • 5.3 Ablation Study: Removing rank embeddings makes interaction insensitive to front and back roles, whereas removing the local objective weakens object structure within conditioned regions.The full model preserves all four instances and clearly places the zebra in front of the refrigerator in the ablation example.

6 Conclusion

OccluRank enables explicit visibility control by adding ordinal ranks and updating rank-conditioned instance features before aggregation, supported by geometry-derived data and controlled evaluation.

  • 6 Conclusion: OccluRank adds an ordinal rank to each instance, while its OII module lets overlapping rank-conditioned features interact before aggregation.This provides visibility control without additional geometric inputs or specialized inference.
  • 6 Conclusion: OccluLayout supplies geometry-derived occlusion and amodal annotations, and OccluLayout-Bench provides synthetic controlled evaluation.The conclusion presents these resources alongside the model framework.

A.1 Comparison with Existing Datasets

OccluLayout prioritizes complete, geometrically consistent supervision for occlusion-aware generation, combining high-resolution images, detailed instance descriptions, and explicit occlusion annotations.

  • A.1 Comparison with Existing Datasets: Unlike the compared datasets, OccluLayout jointly provides high-resolution images, phrase-level descriptions, boxes, masks, geometry-derived Z-order, and amodal annotations.Its 71-category vocabulary supports attribute-controllable generation beyond category-level supervision.
  • A.1 Comparison with Existing Datasets: Although smaller than large-scale SA-1B-derived datasets, OccluLayout is designed to provide complete and reliable supervision for occlusion-aware layout-to-image generation.Deriving order, boxes, and masks from one 3D scene maintains their geometric consistency without manual labeling or auxiliary prediction.

A.2 Detailed Dataset Construction

OccluLayout is built from varied Blender scenes with geometry-derived amodal annotations and front-to-back order. Filtering and image editing improve recognizability and appearance diversity while preserving instance alignment.

  • Scene Generation: OccluLayout uses Blender scenes with two to five public 3D assets placed on a shared ground plane across varied viewpoints and object configurations.The collection spans 71 instance categories, including furniture, appliances, vehicles, instruments, animals, and everyday objects.
  • Annotation and Order Extraction: Amodal masks and bounding boxes are rendered for each instance, while front-to-back order is derived from relative positions in camera coordinates.The RGB image and annotations therefore originate from the same complete scene geometry rather than retrospective image inference.
  • Quality Filtering: Scenes are filtered by a minimum 1% instance-box area and a maximum 0.70 occlusion ratio to retain recognizable instances under meaningful occlusion.The filters exclude extremely small or almost entirely invisible objects.
  • Appearance Diversification: FLUX.2 diversifies backgrounds and selected appearances using 13,728 combinations spanning scene types, seasons, weather, and times of day.Each edited image is subsequently checked for object drift, disappearance, or category changes.
  • Text Annotation: Qwen3-VL generates global and instance descriptions focused on intrinsic properties while excluding inter-object relations that could reveal target order.Retained samples include an RGB image, captions, instance descriptions, amodal boxes, and an ordered instance list.

A.3 Dataset Statistics

OccluLayout provides a large training split and a held-out benchmark with closely matched instance-count distributions and substantial overlap. The benchmark evaluates presence, placement, attributes, occlusion order, and image quality through complementary metrics.

  • Dataset Scale: OccluLayout contains 34,496 images and 117,433 instances, with 33,496 training images and 1,000 benchmark images.The benchmark includes 3,386 instances, while training uses 114,047.
  • Split Statistics: Training and benchmark images average 3.40 and 3.39 instances, respectively, with overlapping boxes in 78.8% and 77.2% of images.Both splits contain scenes with two to five objects.
  • Occlusion Coverage: OccluLayout-Bench contains 772 images with overlapping boxes, including 1,762 overlapping pairs and 503 images with at least two overlapping pairs.This covers both isolated pairwise occlusion and multi-instance visibility relationships.
  • Evaluation Protocol: The benchmark measures instance presence, Box mIoU, Color, Texture, Strict Pair, Strict Image, and FID across complementary evaluation dimensions.These metrics jointly assess whether requested objects are generated, placed, visually specified, correctly ordered, and image-quality consistent.
  • Instance and Layout Metrics: Presence counts recognizable target instances, while Box mIoU uses one-to-one category matching and assigns zero IoU to unmatched ground-truth instances.Because target boxes are amodal but predictions rely on visible evidence, Box mIoU is conservative under occlusion.
  • Attribute and Order Metrics: Color and Texture use 0–2 attribute scores, whereas Strict Pair and Strict Image require recognizable instances and correct visibility relations for overlapping boxes.Strict Image requires every overlapping pair in an image to satisfy the Strict Pair criterion.
  • Overall Quality: FID independently compares generated images with benchmark references to provide a complementary distribution-level image-quality measure.The order and semantic metrics are designed to distinguish omission, placement, attribute, and visibility failures.

B.1 Complementary Occlusion and Depth Evaluation

A complementary protocol evaluates OccluRank with dedicated order-prediction networks rather than MLLMs. OccluRank achieves the strongest reported occlusion-order and depth-order results against representative baselines.

  • Evaluation Design: The specialized protocol uses order-prediction networks to evaluate pairwise relationships, complementing the MLLM-based benchmark.Prescribed relations come directly from the front-to-back annotation order, with earlier instance indices expected to appear in front.
  • Occlusion Order: Occlusion-order performance is reported with pairwise F1, where higher values indicate better adherence to the requested order.The statistics aggregate predicted and prescribed directions over all overlapping-box pairs.
  • Depth Order: Depth-order performance is reported with WHDR using uniform pair weights, where lower values indicate better depth-order consistency.The predicted and prescribed depth relations are compared over evaluated pairs.
  • Results: 0.8577 occlusion-order F1 and 0.1844 depth-order WHDR are achieved by OccluRank, the best performance on both complementary metrics.The model consistently outperforms the compared layout-to-image and occlusion-aware generation methods.
  • Training Comparison: All trainable methods use the OccluLayout training split, while OcclusionFormer also receives an official-checkpoint evaluation because its training code is unavailable.The additional checkpoint is denoted OcclusionFormer†.

C.2 Implementation Details

OccluRank is trained by optimizing rank-aware instance components and OII-related parameters while keeping the diffusion backbone frozen. Inference uses fixed 1024 × 1024 generation settings.

  • Trainable Components: OccluRank optimizes per-instance feature construction, ordinal rank embeddings, the OII block, and residual injection parameters from pretrained SDXL and IFAdapter checkpoints.The diffusion backbone remains frozen.
  • Network Integration: The OII-enhanced signal is injected into selected SDXL cross-attention layers, with adapter scale s = 1.0 and residual gates initialized to zero.Injection occurs in Transformer sub-blocks 0–3 of the middle and lowest-resolution upsampling blocks.
  • Optimization: Training uses 1024 × 1024 resolution on one NVIDIA RTX PRO 6000 GPU with AdamW, a 1×10^-4 learning rate, and effective batch size 160.These settings define the reported training configuration.
  • Inference: Inference generates 1024 × 1024 images with 30 denoising steps, classifier-free guidance scale 7.5, and random seed 42.One image is generated for each OccluLayout-Bench layout.

D Additional Ablation Analysis

Ablations show that explicit rank conditioning, order-aware interaction, and localized supervision jointly improve occlusion-order control. Controlled reversals and unseen-category tests further indicate that OccluRank responds to prescribed order and generalizes beyond training categories.

  • Quantitative Ablations: The full model achieves the highest Occ. score of 0.8577 and the lowest Dep. value of 0.1844.These results support the joint contribution of explicit rank conditioning, order-aware instance interaction, and localized denoising supervision.
  • Quantitative Ablations: Scalar Reweight improves Strict Pair and Strict Image over IFAdapter by 4.25–5.34 and 3.63–6.99 percentage points, respectively.The full model further improves these metrics by 2.61–7.38 and 0.78–6.86 points, supporting interaction before aggregation.
  • Quantitative Ablations: Removing the local objective reduces Strict Pair by 1.48–5.56 points and Strict Image by 1.55–8.03 points, despite little effect on Box mIoU.The local objective primarily supports recognizable instance structures, coherent boundaries, and visually distinguishable occlusion relationships.
  • Quantitative Ablations: Removing rank embeddings decreases Strict Pair and Strict Image by 2.33–3.86 and 2.20–5.95 points, respectively, while leaving presence and spatial alignment largely unchanged.Without rank embeddings, instance interaction lacks an explicit asymmetric signal for distinguishing prescribed front and back roles.
  • Responsiveness to Reversed Occlusion Orders: Under reversed orders, OccluRank reverses the realized covering relationship, whereas the variant without rank embeddings produces nearly unchanged occlusion patterns.The controlled comparison varies only front-to-back order while preserving prompts, instance descriptions, and bounding boxes.
  • Generalization to Unseen Categories: On categories absent from the 71-category training set, OccluRank preserves requested instances and realizes specified occlusion relationships.The results indicate generalization of order-aware interaction to occlusion relationships between unseen categories.
Loading 2608.20932v1…