Source-linked AI summary
SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation
Vaibhav Agrawal, Rishubh Parihar, Pradhaan Bhat, Ravi Kiran Sarvadevabhatla, R. Venkatesh Babu
TL;DR
SeeThrough3D targets the limited ability of existing 3D layout-conditioned generators to model precise inter-object occlusions. It uses translucent-box OSCR conditioning and masked attention to bind objects to boxes, and reports realistic occlusion handling, strong layout control, and generalization despite synthetic training data. The method does not preserve image consistency under layout changes.
Problem
Existing 3D layout-conditioned methods often fail to model precise inter-object occlusions needed for depth-consistent partially hidden objects.
Method
SeeThrough3D renders translucent, orientation-coded 3D boxes as OSCR tokens, conditions a pretrained text-to-image model on them, and masks attention to bind boxes to object descriptions.
Results
SeeThrough3D faithfully models heavy occlusion, outperforms existing baselines, and generalizes strongly despite limited synthetic training data.
Takeaways & Limitations
The approach provides occlusion-aware 3D layout control while preserving the base text-to-image model’s prior.
Abstract
from arXiv · showhide
We identify occlusion reasoning as a fundamental yet overlooked aspect for 3D layout-conditioned generation. It is essential for synthesizing partially occluded objects with depth-consistent geometry and scale. While existing methods can generate realistic scenes that follow input layouts, they often fail to model precise inter-object occlusions. We propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly models occlusions. We introduce an occlusion-aware 3D scene representation (OSCR), where objects are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired camera viewpoint. The transparency encodes hidden object regions, enabling the model to reason about occlusions, while the rendered viewpoint provides explicit camera control during generation. We condition a pretrained flow based text-to-image image generation model by introducing a set of visual tokens derived from our rendered 3D representation. Furthermore, we apply masked self-attention to accurately bind each object bounding box to its corresponding textual description, enabling accurate generation of multiple objects without object attribute mixing. To train the model, we construct a synthetic dataset with diverse multi-object scenes with strong inter-object occlusions. SeeThrough3D generalizes effectively to unseen object categories and enables precise 3D layout control with realistic occlusions and consistent camera control.
1. Introduction
SeeThrough3D addresses the difficulty of generating multi-object scenes with precise, depth-consistent occlusions and 3D layout control. It introduces OSCR, binds object descriptions to boxes, and trains on synthetic scenes designed for strong occlusions.
- Motivation: Existing 2D controls provide limited control over 3D arrangement, object size, orientation, placement, and camera viewpoint.Precise 3D-aware generation also requires reasoning about partially hidden objects with consistent scale and perspective.
- Motivation: Depth-map methods and 2D object layers struggle with significant inter-object occlusions in multi-object layouts.Depth maps fail to represent occluded objects, while layers flatten 3D structure and can violate geometry and perspective.
- Approach: SeeThrough3D uses translucent 3D bounding boxes to jointly encode object arrangement and camera viewpoint.Transparency reveals hidden regions, while the rendered representation provides the conditioning signal for occlusion-aware generation.
- Approach: Attention masking binds each textual object description to its corresponding box for accurate individual-object placement.The model also supports personalized objects by binding a reference appearance to a specified OSCR box.
- Training: A synthetic dataset uses controlled assets, placements, and camera parameters to create scenes with strong inter-object occlusions.The trained model generalizes to unseen objects, backgrounds, and complex layouts despite synthetic training data.
2. Related work
Related work provides 3D control through specialized representations, editing methods, and layout-conditioned generation, but existing approaches retain important limits in occlusion, coherence, placement, or viewpoint control.
- 3D control in text-to-image generation: Prior 3D-control methods use specialized generative models conditioned on varied 3D representations.Other directions leverage diffusion-model 3D understanding for editing, including depth, radiance-field, and Gaussian-splat representations.
- 3D control in text-to-image generation: Single-object editing methods and object-property controllers do not provide comprehensive multi-object placement and camera-viewpoint control.The related-work discussion distinguishes editing and partial property control from full 3D layout-conditioned generation.
- 3D layout conditioned generation: LooseControl conditions generation with depth maps, while other layout methods use sequential generation or learnable box-set adapters.Sequential addition can introduce inversion artifacts and incoherence, while some property-specific methods remain limited in placement or viewpoint control.
3. Method
SeeThrough3D represents 3D layouts with rendered translucent boxes, conditions a pretrained DiT-based generator on OSCR tokens, and masks attention to bind boxes to object descriptions. Its synthetic training data targets severe occlusion while retaining appearance diversity.
- 3.1. OSCR: OSCR represents each object with a translucent 3D box whose colored faces encode orientation and whose rendering captures occlusion and camera view.The representation is rendered from a specified camera view, preserving orientation cues even when occlusion alters apparent face colors.
- 3.2. SeeThrough3D: Rendered OSCR maps are encoded into tokens and concatenated with text and noisy-image tokens inside FLUX’s multimodal DiT.LoRA is trained on projections associated with the added OSCR tokens to preserve the base text-to-image prior.
- 3.3. Object binding with attention masking: Masked attention associates OSCR tokens inside each box with the corresponding object tokens using rendered segmentation extents.Overlapping regions attend to multiple objects rather than being assigned to only one box.
- 3.3. Object binding with attention masking: Under heavy occlusion, attention maps retain distinct bicycle and van features and reveal the van behind the bicycle.The resulting outputs show precise occlusion boundaries, supporting distinct object-specific representations in latent space.
- 3.4. Personalized objects: The method extends layout-conditioned generation to personalized objects by binding reference-image appearance tokens to a selected OSCR box.The conditioning combines reference appearance, text, target-image, and OSCR tokens.
- 3.5. Dataset: The synthetic dataset places diverse 3D assets under controlled configurations and camera views to induce strong occlusions.Depth-to-image augmentation produces realistic appearance diversity while following the rendered layout.
4. Experiments
Experiments evaluate SeeThrough3D on 3D layout control, occlusion consistency, generalization, user preference, personalization, and key design choices. The results indicate strong qualitative performance, favorable user preferences, and distinct contributions from OSCR transparency, orientation coding, attention-based binding, and data filtering.
- Experimental setup: The evaluation uses 3DOcBench, which contains paired 3D bounding-box layouts, rendered images, and scene text prompts, with metrics covering layout adherence, alignment, and image quality.The reported metrics include depth ordering, CLIP objectness, angular error, CLIP image-text similarity, and KID.
- Qualitative results: SeeThrough3D generates realistic scenes with intricate overlaps, viewpoint control, text rendering, transparent-object handling, and natural interactions.These capabilities are reported qualitatively in Figure 8.
- Qualitative results: Despite training on layouts with up to four objects and rigid canonical poses, the model generalizes to complex scenes, diverse poses, natural interactions, and out-of-domain objects.The reported examples include a dog riding a bicycle, a person playing guitar, and categories absent from the training data.
- Baseline comparisons: Compared with LooseControl, Build-A-Scene, LaRender, and VODiff, SeeThrough3D provides more coherent 3D layout, occlusion, orientation, and object-placement control.The baselines are described as limited by depth-map representation, sequential-generation artifacts, or 2D layouts.
- User study: In an A/B study with 60 participants, outputs from SeeThrough3D were preferred over randomly chosen baselines for image realism, layout adherence, and text-prompt alignment.Figure 10 reports preference percentages for each category, but the supplied passage does not provide the numerical percentages.
- Ablations: Ablations show that transparency supports occlusion and relative-depth reasoning, color-coded faces reduce angular error, attention-based binding improves layout adherence, and data filtering improves performance.Opaque boxes instead yield the best orientation accuracy because they provide a clearer color signal.
5. Conclusion
SeeThrough3D is presented as an occlusion-aware model for 3D layout control that preserves text-to-image priors and generalizes from limited synthetic data. The paper also identifies image consistency under layout changes as an unresolved limitation and suggests editing as future work.
- Conclusion: SeeThrough3D introduces OSCR for modeling heavy occlusions in 3D layout-conditioned image generation.The conclusion frames OSCR as the paper’s occlusion-aware 3D scene representation.
- Conclusion: The approach preserves strong text-to-image priors while exhibiting strong generalization despite training on limited synthetic data.The conclusion also reports evaluations against existing baselines and ablations of key design choices.
- Conclusion: The method does not preserve image consistency under layout changes, and the paper identifies editing as a future direction.This is the stated scope boundary in the conclusion.
B. Dataset
The dataset pipeline places manually aligned 3D assets and cameras procedurally, filters scenes for heavy occlusion and valid composition, and augments renders with realistic depth-to-image synthesis.
- B.1. The rendering pipeline: The pipeline uses 39 assets from Objaverse and SketchFab, manually aligned to a canonical +Y front direction and scaled to approximate relative real-world dimensions.Scenes contain up to four objects placed in a Blender environment with a camera on a hemisphere aimed at the origin.
- B.1. The rendering pipeline: Procedural sampling and filtering remove unnatural compositions and retain scenes with strong inter-object occlusion.The visibility ratio x is defined as visible object area v divided by total object area a, and scenes where every object has x > 0.7 are filtered out.
- B.2. Augmentations: Depth-to-image augmentation increases appearance and background diversity while preserving the rendered spatial layout.Object-level CLIP filtering removes augmentations that misalign objects with their intended depth regions.
B.3. Statistics
The training and evaluation data emphasize heavy occlusion, with roughly uniform orientations and distributions shaped by object size and camera elevation. Augmentation filtering addresses occasional depth-to-image placement errors.
- B.3. Statistics: The training dataset is biased toward low minimum visibility ratios because its filtering strategy favors heavy occlusion scenarios.Orientation values are roughly uniformly distributed.
- B.3. Statistics: Examples with large 2D bounding-box dimensions become less frequent because smaller objects allow more objects to fit while remaining visible.This trend is reported as a consequence of the scene-construction constraints.
- B.2. Augmentations: The depth-to-image augmentation pipeline can misplace objects in intended depth regions, so object-level CLIP filtering is used to retain layout-consistent augmentations.The supplied example describes a pigeon being generated instead of a crow.
- B.3. Statistics: Low camera elevations are favored because they produce stronger occlusions, whereas high camera shots tend to produce weaker occlusion scenarios.The evaluation benchmark likewise contains many low-elevation, high-occlusion cases.
C. 3DOcBench benchmark details
3DOcBench is built procedurally in Blender with randomized assets and cameras, filtering layouts for occlusion and object-size constraints. Its analysis also examines whether object-to-image attention aligns with segmented objects in heavily occluded scenes.
- Benchmark construction: 3DOcBench uses Blender scenes with randomly placed 3D assets and cameras, retaining layouts that satisfy occlusion and object-size constraints.The benchmark follows the training dataset’s procedural-generation process.
- Attention analysis: Attention analysis uses eight complex two-object layouts containing heavy occlusion.The layouts are used to visualize attention maps and assess foreground-background separation.
- Attention analysis: The analysis compares image-to-object attention maps with object segmentation masks using correlation coefficient.Segment Anything produces the object-level masks used for the alignment measurement.
E. Implementation details
The implementation trains SeeThrough3D with LoRA adapters on synthetic OSCR-conditioned data while preserving the base text-to-image prior. The method supports controllable 3D attributes, camera viewpoint, occlusions, personalization, and comparisons against orientation-focused baselines.
- Training: Training uses 128-rank LoRA adapters with AdamW for 30K steps and staged 512-resolution training.The first 25K steps use 512 resolution, followed by 5K additional steps; training takes about 9 hours on two NVIDIA H100 GPUs.
- Data: The synthetic dataset renders diverse 3D assets from controlled camera views designed to induce strong inter-object occlusions.The model is evaluated on unseen objects, backgrounds, and complex layouts using qualitative metrics and a user study.
- Personalization: Personalization adds a separate rank-128 subject LoRA trained for 7.5K iterations on reference-image tokens.The subject LoRA fuses appearance attributes from personalized object images into generation.
- 3D control: OSCR jointly represents object orientation, size, location, camera viewpoint, and occluded regions while preserving diverse text-driven appearance.The representation uses rendered translucent, color-coded 3D boxes, and figures demonstrate control under heavy overlaps with a fixed random seed.
- Baselines: Comparisons include orientation-focused methods Compass Control and ORIGEN, alongside broader 3D scene-control baselines.ORIGEN is incompatible with quantitative layout-adherence evaluation because it lacks 2D layout control.
H. More on angular error evaluation
Angular-error evaluation shows why depth-only layout conditioning is insufficient for front-facing orientation, while OSCR explicitly encodes orientation and performs favorably against existing methods. Additional results cover personalization, control demonstrations, baseline comparisons, and textured-object training.
- Angular error: Depth-map baselines produce many 180° orientation flips because bounding-box depth omits the object’s front-facing direction.A relaxed angular error is therefore reported without penalizing 180° flips.
- Angular error: OSCR color-codes orientation in image space, enabling precise orientation control and favorable angular-error performance versus existing methods.Compass Control performs better than other baselines on angular error, while OSCR performs favorably compared with all existing methods.
- Personalization: Personalized-object control is achieved by finetuning a subject LoRA, without test-time tuning, including categories unseen during training.The method can compose text-described and image-personalized objects.
- Personalization training: Personalization training applies textures to rendered assets and uses separately rendered textured objects as reference-image conditions.The reference object’s orientation is slightly altered to encourage reasoning about 3D placement rather than pixel copying.
- Qualitative comparison: The model generates realistic images that follow precise 3D layouts while maintaining occlusion consistency and outperforming the compared baselines qualitatively.The comparisons assess layout adherence and occlusion consistency.
K. Additional qualitative results
Additional results pair OSCR layouts with generated images and provide an interface for constructing scenes, editing 3D boxes, placing objects, and controlling the camera viewpoint.
- Additional results: Additional examples show OSCR layouts alongside generated images, with box-to-object correspondence omitted for clarity.The results provide further qualitative demonstrations of the method.
- Personalization: Personalization results demonstrate occlusion-aware 3D control for personalized objects using a separate subject LoRA.The approach supports composing objects from text and image modalities without test-time tuning.
- Interface: SeeThrough3D’s web interface lets users add object boxes, edit their dimensions and 3D placement, assign text descriptions, and control the camera viewpoint.The interface is designed to support precise scene construction for creative applications.
M. Limitations
SeeThrough3D remains constrained by the capabilities of its FLUX base model and has additional resource demands for multi-subject personalization.
- The model struggles with out-of-distribution cases that FLUX fails to generate realistically, such as a parrot outside a cage.This limitation is inherited because SeeThrough3D conditions the pretrained FLUX model.
- Multi-subject personalization requires retaining all reference-image tokens in the transformer context, increasing VRAM requirements.
- Qualitative comparisons include failures to generate all objects, incoherent scenes, and inaccurate layout or orientation control.One example specifically places a dog incorrectly relative to a chair.