PaperlayerView exact revision on arXiv

Source-linked AI summary

SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation

Vaibhav Agrawal, Rishubh Parihar, Pradhaan Bhat, Ravi Kiran Sarvadevabhatla, R. Venkatesh Babu

arXiv:2602.23359v1cs.CVcs.AI

TL;DR

3D layout-conditioned generation methods have largely overlooked precise inter-object occlusions, limiting control over partially hidden objects with depth-consistent geometry and scale. SeeThrough3D uses translucent 3D boxes to represent occlusions and condition generation; it faithfully models heavy occlusion, generalizes despite limited synthetic training data, and outperforms existing baselines.

  • Problem

    Existing 3D layout-based methods largely overlook significant inter-object occlusions, despite their importance for depth-consistent generation of partially hidden objects.

    Source: §1 · p. 2

  • Method

    SeeThrough3D conditions image generation on translucent 3D bounding-box scenes that encode object arrangement and viewpoint, using attention masking to bind object descriptions to boxes.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    SeeThrough3D faithfully models heavy occlusion, preserves the base model's text-to-image prior, generalizes despite limited synthetic data, and outperforms existing baselines.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results indicate that occlusion-aware 3D layout control can be achieved while preserving a strong text-to-image prior.

    Source: §5 · p. 8

Abstract

from arXiv · show

We identify occlusion reasoning as a fundamental yet overlooked aspect for 3D layout-conditioned generation. It is essential for synthesizing partially occluded objects with depth-consistent geometry and scale. While existing methods can generate realistic scenes that follow input layouts, they often fail to model precise inter-object occlusions. We propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly models occlusions. We introduce an occlusion-aware 3D scene representation (OSCR), where objects are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired camera viewpoint. The transparency encodes hidden object regions, enabling the model to reason about occlusions, while the rendered viewpoint provides explicit camera control during generation. We condition a pretrained flow based text-to-image image generation model by introducing a set of visual tokens derived from our rendered 3D representation. Furthermore, we apply masked self-attention to accurately bind each object bounding box to its corresponding textual description, enabling accurate generation of multiple objects without object attribute mixing. To train the model, we construct a synthetic dataset with diverse multi-object scenes with strong inter-object occlusions. SeeThrough3D generalizes effectively to unseen object categories and enables precise 3D layout control with realistic occlusions and consistent camera control.

1. Introduction

SeeThrough3D conditions text-to-image generation on a 3D scene representation that exposes occluded regions and specifies camera viewpoint. It uses object-to-box attention masking and synthetic scenes with strong occlusions to support 3D layout control and generalization.

Introduction

  • Precise 3D layout control requires specifying object size, orientation, placement, and viewpoint, while generating partially hidden objects with depth-consistent scale and perspective.The introduction identifies occlusion reasoning as a capability that 2D controls cannot provide.Source: §1 · p. 2
  • Existing 3D-layout methods handle simple scenes with little occlusion but fail on substantial inter-object occlusions, while 2D object layers flatten scene geometry and perspective [4] [7] [9] [43].Earlier text-to-image controls largely use 2D boxes or segmentation maps, limiting control over 3D arrangement and viewpoint [23] [44] [61] [62] [79].Source: §1 · p. 2
  • SeeThrough3D represents each object as a translucent 3D box whose transparency reveals occluded regions, whose color-coded faces encode orientation, and whose rendered view specifies the camera viewpoint.The Occlusion-Aware 3D Scene Representation jointly encodes object arrangement and camera viewpoint.Source: §1 · p. 2
  • SeeThrough3D conditions FLUX [35] on tokens from the rendered OSCR representation and masks attention to bind each textual object description to its corresponding box.The framework also supports personalized-object 3D control by binding an input object image to a specific OSCR box.Source: §1 · p. 2
  • The authors create synthetic scenes from diverse 3D assets, controlling object placement and camera parameters to induce strong inter-object occlusions.They report qualitative and metric evaluation, as well as a user study, and describe generalization to unseen objects, backgrounds, and complex scene layouts.Source: §1 · p. 2

2. Related work

Prior work offers varied forms of 3D control, but existing layout-generation approaches leave occlusion-aware, geometrically consistent control insufficiently addressed.

3D control in text-to-image generation

3D layout conditioned generation

  • Layout-generation approaches offer partial control but face limits in complex scenes, coherence, placement, viewpoint, or domain generality [4] [19] [9] [43] [41] [70].LooseControl [4] uses depth maps of 3D boxes but struggles with complex, diverse scenes; Build-A-Scene [19] adds objects through repeated generation-inversion cycles, causing inversion artifacts and incoherence. Adapter-based box-set methods [41] [70] are domain-limited and less effective than spatial conditioning.Source: §2 · pp. 2–3

Occlusion awareness

  • Inter-object occlusion matters in perception and generation [32] [36] [42] [39] [69], yet receives little attention in prior work [4] [19].Flat 2D object-layer approaches [14] lack 3D awareness and produce geometrically inconsistent occlusions; SeeThrough3D is proposed for generalized occlusion-aware 3D layout control.Source: §2 · p. 3

3. Method

SeeThrough3D conditions a pretrained flow-based text-to-image model on rendered OSCR layouts, using attention masking to associate layout regions with object semantics.

3.1. OSCR

  • Translucent 3D boxes with color-coded faces encode occluded regions, object orientation, and camera viewpoint, addressing limitations of depth-map and 2D-layer representations.The boxes are rendered from a specified camera view to provide the OSCR condition.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2. SeeThrough3D

  • OSCR images are VAE-encoded into tokens concatenated with text and noisy-image tokens; LoRA adapts attention projections for OSCR while preserving the base model prior.OSCR tokens share positional encodings with noisy-image tokens, and their attention to image tokens is blocked.Source: §3.2 · pp. 3–4, Figure 4 · p. 4

3.3. Object binding with attention masking

  • Bounding-box OSCR tokens attend to their corresponding object nouns using rendered segmentation masks; overlapping regions attend to multiple objects, yet heavy-occlusion outputs preserve precise boundaries.Attention maps retain van attention in empty regions of a bicycle structure, indicating distinct object-specific features in the latent space.Source: §3.3 · p. 4, Figure 5 · p. 4, Figure 6 · p. 5

3.4. Personalization

  • For personalized generation, reference-image appearance tokens are concatenated with text, image, and OSCR tokens, then bound to a target box through the attention mask.Separate appearance-token sets allow the approach to extend to multiple reference objects.Source: §3.4 · p. 5

3.5. Dataset

  • The synthetic dataset pairs Blender-rendered scenes and OSCR layouts across viewpoints, filters for meaningful visibility and overlap, and contains 25K renders plus 25K augmentations.Depth-to-image augmentations preserve layout, with object-level CLIP filtering to remove misaligned placements.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate SeeThrough3D’s 3D layout and occlusion control on a new benchmark, with qualitative and comparative results, a user study, personalization, and design ablations. The results show control and generalization across complex scenes, while the ablations identify components that support layout adherence and orientation accuracy.

4.1. Experimental setup

  • Across diverse layouts and camera viewpoints, 3DOcBench provides 500 paired 3D bounding-box layouts, rendered images, and scene prompts featuring challenging occlusions.The benchmark was procedurally constructed in Blender, varying object arrangements and camera poses while preserving a minimum visible area for each object.Source: §4.1 · pp. 5, 7
  • Evaluation measures layout adherence, visibility order, orientation accuracy, text-image alignment, and image quality using objectness, depth ordering, angular error, CLIP similarity, and KID.The evaluation estimates object masks and depth, compares pairwise relative depth order with ground truth, and measures orientation error against ground truth.Source: §4.1 · p. 7, Table 1 · p. 7

4.2. Results

  • SeeThrough3D generates intricate overlaps, controls viewpoints, and generalizes beyond four-object training layouts to unseen categories, diverse poses, and natural object interactions.Reported unseen categories include musical instruments, electronic devices, transparent objects, and books; examples include sitting and cycling poses and people interacting with objects.Source: Figure 8 · p. 6, §4.2 · pp. 7–8
  • SeeThrough3D produces coherent scenes with precise 3D layout and orientation control, while comparison methods show failures in occlusion handling, object placement, or image quality.LooseControl [4] struggles with complex occlusions and object binding; Build-A-Scene [19] improves layout and occlusion consistency but has inversion artifacts and worse KID. LaRender and VODiff can misplace objects and fail to render heavily occluded objects accurately.Source: Table 1 · p. 7, Figure 9 · p. 7, §4.2 · p. 8
  • In an A/B study with 60 participants, SeeThrough3D was preferred over a randomly selected baseline for realism, layout adherence, and text-prompt alignment.The supplied passages report preference across all three categories but do not provide the preference percentages.Source: §4.2 · p. 8

4.3. Personalization

  • Personalization extends the method to reference-image-based 3D control by texturing 3D assets used in the training data.Figure 11 presents personalization results using a reference image of an object.Source: §4.3 · p. 8, Figure 11 · p. 8

4.4. Ablations

  • Ablations show that box transparency supports reasoning about occlusion and relative depth, face color-coding reduces angular error, attention-based binding improves layout adherence, and filtering simplistic layouts helps performance.Opaque boxes yield the best orientation accuracy because they provide a clearer color signal.Source: §4.4 · p. 8, Table 2 · p. 8

5. Conclusion · A. Overview

SeeThrough3D introduces OSCR for occlusion-aware 3D layout control and reports faithful handling of heavy occlusion, strong generalization, and better performance than existing baselines. The method does not preserve image consistency under layout changes, which the authors identify as a future editing direction.

5. Conclusion

  • SeeThrough3D outperforms existing baselines in evaluations of occlusion-aware 3D layout control.The authors also report ablations of key design choices.Source: §5 · p. 8
  • OSCR is the proposed occlusion-aware 3D scene representation for SeeThrough3D.The approach is designed for 3D layout control.Source: §5 · p. 8
  • SeeThrough3D faithfully models heavy occlusion scenarios while preserving the model’s strong text-to-image prior.Source: §5 · p. 8
  • Despite training on limited synthetic data, SeeThrough3D exhibits strong generalization capabilities.Source: §5 · p. 8
  • SeeThrough3D does not preserve image consistency under layout changes; the authors identify editing as a future direction.Source: §5 · pp. 8–9
  • Figure 12 covers ablations of the OSCR representation, binding mechanism, and data preparation strategy.Source: Figure 12 · p. 9

A. Overview

  • The appendix overview describes additional dataset and implementation details, experimental discussion, analysis, and extended qualitative results and comparisons.It advises readers to use the figures and captions to skim this material.Source: A. Overview · p. 13

B. Dataset · B.1. The rendering pipeline · B.2. Augmentations

The dataset pipeline combines aligned, scale-adjusted 3D assets and constrained scene rendering with realistic, layout-preserving augmentations filtered for object-level adherence. Its rendering filters retain scenes with substantial but not excessive occlusion and suitably sized objects.

B.1. The rendering pipeline

  • The authors collect 39 Objaverse and SketchFab assets, manually align their canonical fronts to +Y, and scale them to match relative real-world dimensions.Scale values were obtained using Gemini 2.5 Pro.Source: §B.1 · p. 13
  • Scenes contain up to four aligned assets placed within a fixed-radius hemisphere, with a virtual camera on its surface pointing toward the origin.The camera renders each scene from a given viewpoint.Source: §B.1 · p. 13
  • Procedural scene generation samples camera and object placements uniformly under predefined constraints, then filters poor-quality scenes to avoid collisions and ensure heavy occlusion.The pipeline defines visibility ratio x as visible object area divided by total area; scenes are discarded if all objects have x > 0.7 or any object has x < 0.3.Source: §B.1 · p. 13
  • Rendered scenes are filtered so each object's 2D bounding-box largest side measures between 0.125 and 0.750 of the image size.Source: §B.1 · p. 13

B.2. Augmentations

  • Augmentations use FLUX.1-Depth-dev to turn rendered-image depth into realistic images that preserve the same spatial layout, addressing limited realism and background diversity in Blender renders.The authors describe augmentation as a scalable alternative to creating highly varied 3D scenes.Source: §B.2 · p. 13
  • Object-level CLIP filtering retains an augmentation only when every cropped object segment scores at least 0.25 against its corresponding text description.The crops are obtained using Blender segmentation masks; augmentations below threshold for any object are discarded.Source: §B.2 · p. 13

B.3. Statistics

Training data and 3DOcBench are biased toward heavily occluded scenes, while retaining roughly uniform object orientations; both also favor smaller objects and low camera elevations.

B.3. Statistics

  • Object-level CLIP filtering removes augmentations with any object similarity score below 0.25 to preserve adherence to the original layout.The pipeline segments and crops augmented objects, then compares each crop with its corresponding text description; depth-to-image generation can misplace objects.Source: Figure 13 · p. 14
  • Training scenes favor low minimum visibility, roughly uniform orientations, smaller objects, and low camera elevations associated with stronger occlusions.Smaller objects allow more objects to fit while remaining visible; the pipeline favors low-camera renders because data selection favors high-occlusion scenarios.Source: §B.3 · pp. 13, 15, Figure 14 · p. 14
  • 3DOcBench contains heavy occlusions, roughly uniform orientations, more small objects, and many low-elevation camera views with high inter-object occlusion.Source: Figure 15 · p. 15

C. 3DOcBench benchmark details · D. Overlapping regions

3DOcBench is procedurally generated to include challenging occlusion and camera configurations. In overlapping scenes, object attention remains separated and distinguishes foreground from background, with spatial alignment strongest in selected DiT layers and denoising steps.

C. 3DOcBench benchmark details

  • 3DOcBench layouts are procedurally generated in Blender with randomly placed 3D assets and cameras, then filtered for occlusion and object-size constraints.The benchmark uses the same procedural generation as the training dataset.Source: C. 3DOcBench benchmark details · p. 15
  • 3DOcBench contains heavy occlusion scenarios and a roughly uniform distribution of object orientations.These benchmark statistics are presented in Fig. 15.Source: C. 3DOcBench benchmark details · p. 15
  • The benchmark has more smaller objects by 2D bounding-box dimension and many low-elevation camera cases with high inter-object occlusion.These statistics are presented in Fig. 15.Source: C. 3DOcBench benchmark details · p. 15

D. Overlapping regions

  • Overlapping object tokens yield sharp occlusion boundaries without attribute mixing, and attention maps distinguish foreground from background in eight heavily occluded two-object layouts.The analysis attributes this separation to object features being segregated in the model’s latent space.Source: D. Overlapping regions · p. 15
  • Image-to-object-token attention aligns spatially most strongly in DiT layers 11–23, with alignment emerging around the fifth of 25 denoising steps.Alignment was assessed by correlation with manually filtered Segment Anything object-region masks.Source: D. Overlapping regions · p. 15
  • For attention visualization, the method selects the top 50 layer–timestep combinations by spatial-alignment correlation and averages image-to-text attention for each object.The resulting attention maps are visualized for the overlapping layouts in Fig. 17.Source: D. Overlapping regions · p. 15

E. Implementation details · F. Taking control with SeeThrough3D · G. Additional baseline comparisons

SeeThrough3D’s implementation combines LoRA-based fine-tuning with a staged resolution schedule, while its OSCR enables joint control of 3D scene attributes and camera viewpoint. The supplementary sections report control demonstrations and qualitative comparisons with orientation-control baselines.

E. Implementation details

  • The model uses rank-128 LoRA adapters on attention query, key, and value projections, with LoRA scale set to 0 for text and image tokens.It is trained from FLUX.1-dev [35] for 30K steps using AdamW, learning rate 10^-4, and effective batch size 2.Source: E. Implementation details · p. 15
  • Training switches from 512 to 1024 resolution for the final 5K of 30K steps, a staged schedule reported to improve realism.Training takes around 9 hours on 2 NVIDIA H100 GPUs, one image per GPU.Source: E. Implementation details · pp. 15–16
  • Training scenes use controlled Blender object arrangements and camera viewpoints to create strong inter-object occlusions while keeping each object sufficiently visible.The dataset pairs images with color-coded translucent 3D bounding-box OSCR renderings and uses depth-to-image augmentation for realistic images following the same layout.Source: Figure 16 · p. 16
  • Personalization adds a rank-128 subject LoRA on reference-image tokens, with both LoRAs fine-tuned on the personalization dataset for 7.5K iterations.Source: E. Implementation details · p. 16

F. Taking control with SeeThrough3D

  • OSCR jointly controls object orientation, size, location, camera viewpoint, and occluded regions, while prompts control object and background appearance.The base text-to-image prior supports diverse visual appearances.Source: F. Taking control with SeeThrough3D · p. 16
  • Same-seed demonstrations show consistent occlusion handling under heavy overlap and low camera elevation, illustrating precise control for design and architecture.The examples include low-elevation cases d1 in Figures 19 and 20 and b4 in Figure 19.Source: F. Taking control with SeeThrough3D · p. 16

G. Additional baseline comparisons

  • ORIGEN controls object orientation but not object locations, whereas Compass Control supports orientation with 2D layouts; ORIGEN is excluded from quantitative layout-adherence evaluation.The comparisons are qualitative for ORIGEN because it lacks 2D layout control.Source: G. Additional baseline comparisons · pp. 16–17
  • SeeThrough3D models complex occlusions without attribute mixing, while Compass Control struggles with heavy overlaps and ORIGEN omits objects and produces artifacts.ORIGEN’s one-step generation also limits image fidelity.Source: G. Additional baseline comparisons · p. 17

H. More on angular error evaluation · I. Personalization · J. Additional qualitative comparisons

Across these sections, the paper reports color-coded OSCR orientation control, personalization through a separate subject LoRA, and realistic layout-following generations that maintain occlusion consistency.

H. More on angular error evaluation

  • Color-coded OSCR encodes front-facing orientation and outperforms baselines on angular error; layout-depth methods can produce 180° flips.The relaxed angular-error measure does not penalize 180° flips, which layout-depth methods may produce because they do not encode front-facing direction.Source: H. More on angular error evaluation · pp. 17–18, Table 3 · p. 18

I. Personalization

  • A separate subject LoRA fuses appearance from personalized object images, enabling occlusion-aware 3D control without test-time tuning, including unseen categories such as bottles and glasses.The paper also demonstrates combining a text-specified dog with an image-specified royal chair.Source: §I · p. 18
  • Personalization training textures a randomly selected rendered object, alters the reference object's orientation, then uses the textured object in the original image as the reference-conditioned target.The textures are generated with FLUX prompts for high-frequency details such as text and sharp patterns.Source: §I · pp. 18, 21
  • Compass Control struggles with complex occlusions and mixes object attributes under heavy overlap, whereas SeeThrough3D handles complex occlusions without attribute mixing..Source: Figure 21 · p. 21
  • ORIGEN sometimes omits scene objects and produces noise-optimization artifacts; its one-step generation also limits image fidelity.SeeThrough3D is shown modeling complex occlusions without attribute mixing.Source: Figure 21 · p. 21

J. Additional qualitative comparisons

  • Additional qualitative comparisons report that SeeThrough3D generates realistic images following precise 3D layouts while maintaining occlusion consistency, outperforming the baselines..Source: J. Additional qualitative comparisons · p. 21

K. Additional qualitative results · L. User interface

Additional qualitative results pair OSCR layouts with generated images, while the user interface supports constructing 3D layouts and controlling camera viewpoints.

K. Additional qualitative results

  • Additional examples show OSCR layouts alongside their generated images, with box-to-object correspondence omitted for clarity.Source: K. Additional qualitative results · p. 21
  • A separate subject LoRA enables personalized, occlusion-aware 3D control without test-time tuning.It fuses appearance attributes from a personalized object image into generation.Source: Figure 22 · p. 21
  • SeeThrough3D composes objects across modalities, such as a text-described dog and an image-conditioned royal chair.Source: Figure 22 · p. 21
  • Personalization generalizes to object categories unseen during training, including bottles and glasses.Source: Figure 22 · p. 21

L. User interface

  • The web interface lets users add object boxes, edit their dimensions and 3D placement, and enter a description for each object.Source: L. User interface · p. 21
  • Users can control camera viewpoints and draw on predefined template dimensions for common objects such as cars and animals.Source: L. User interface · p. 21

M. Limitations

SeeThrough3D inherits limitations from its pretrained FLUX prior, including failures on some out-of-distribution occlusion cases. Personalization also requires all reference-image tokens in the transformer context, increasing VRAM use, especially for multiple subjects.

M. Limitations

  • The FLUX [35] prior sometimes fails on out-of-distribution occlusion cases, and SeeThrough3D consequently struggles with those cases too.The paper illustrates this limitation with parrot-and-cage examples.Source: M. Limitations · pp. 21–22, Figure 25 · p. 22
  • Personalization requires all reference-image tokens in the transformer context, raising VRAM requirements, especially for multi-subject personalization.Source: M. Limitations · p. 22

Related papers

Loading 2602.23359v1…