PaperlayerView exact revision on arXiv

Source-linked AI summary

Tetris3D: 3D Scene Generation With Objects That Fit Together

Jaeyeong Kim, Jinhyuk Jang, Jongmin Lee, Kyehong Park, Seungryong Kim

arXiv:2610.10539v1cs.CV

TL;DR

Recovering local compatibility between interacting objects is challenging when neighboring objects occlude the regions where they meet, and existing approaches offer limited explicit guidance. Tetris3D conditions object generation on neighboring geometry and physical relationships, and introduces a simulation-based dataset of annotated interactions. Experiments report improved reconstruction quality and physical consistency on synthetic and real-world scenes, including under occlusion.

  • Problem

    Neighboring objects can occlude interaction regions, while existing approaches provide limited explicit guidance for the local spatial compatibility needed to recover depicted interactions.

    Source: §1 · p. 2

  • Method

    Tetris3D conditions each object's shape and pose on neighboring geometry and physical relationships, and introduces ComOb, a simulation-based dataset of 1.2M scenes with geometry and interaction annotations.

    Source: §1 · p. 2

  • Results

    Experiments on synthetic and real-world scenes show improved reconstruction quality and physical consistency, preserving object interactions even when their regions are occluded.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Physical interactions can serve as cues for recovering object shapes and poses, rather than only as criteria for evaluating completed scenes.

    Source: E DISCUSSION AND LIMITATIONS · p. 22

    The interaction conditions are learned guidance rather than hard constraints, so they do not guarantee physically valid configurations in every case.

    Source: E DISCUSSION AND LIMITATIONS · p. 23

Abstract

from arXiv · show

We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibility between neighboring objects that interact with one another. To address this, we explicitly condition the generation of each object on the geometry of surrounding objects and their physical relationships, guiding its shape and pose to remain geometrically and physically plausible within the scene. Moreover, we introduce ComOb, a physics simulation-based dataset of 1.2M scenes featuring physical interactions across diverse object categories, with per-object meshes and pairwise physical relation annotations. Comprehensive experiments on synthetic and realworld scenes show that Tetris3D recovers coherent object shapes and poses even when interacting regions are occluded, and achieves state-of-the-art performance in both generation quality and physical stability.

1 INTRODUCTION

Recovering compatibility between interacting objects is difficult because neighboring objects can occlude the regions where they meet. Tetris3D conditions each object's generation on neighboring geometry and physical relationships, and the paper introduces ComOb to support training on diverse interactions.

1 INTRODUCTION

  • Existing scene methods assemble independently generated objects or jointly generate objects or poses through implicit feature aggregation, providing limited explicit guidance for local spatial compatibility.That compatibility is needed to recover depicted interactions between neighboring objects.Source: §1 · p. 2
  • Tetris3D generates each object in relation to its neighbors, conditioning shape and pose on neighboring geometry and physical relationships, including when interaction regions are occluded.The framework uses physical relationships to guide how an object should rest on or fit within its neighbors.Source: §1 · p. 2
  • Tetris3D generates objects autoregressively in an order derived by topologically sorting a VLM-inferred physical dependency hierarchy.Previously generated neighbors provide interaction context; for a cup resting on a table, the table is generated first.Source: §1 · p. 2
  • ComOb provides 1.2M simulation-based scenes with per-object geometry, physical-relation annotations, and naturally arising occlusions from diverse interactions.Objects are settled into physically stable configurations, including arrangements such as stacking and containment.Source: §1 · p. 2
  • Experiments on synthetic and real-world scenes show plausible reconstructions that preserve interactions under occlusion, with superior reconstruction and generation quality and physical stability.The introduction reports these outcomes without specifying metric values.Source: §1 · p. 2

2 PRELIMINARIES: TRELLIS.2

TRELLIS.2 represents objects with a two-stage voxel-generation pipeline: it first predicts coarse occupancy structure, then generates structured latents on active voxels. Its stages use diffusion transformers conditioned on image features.

2 PRELIMINARIES: TRELLIS.2

  • TRELLIS.2 first generates sparse structure and then structured latents attached to active voxels, which encode geometry and texture.The sparse structure is a binary occupancy grid whose active voxels outline coarse geometry.Source: §2 · p. 2
  • Figure 2 places object generation in an autoregressive, topologically ordered scene pipeline with same-grid condition injection and visible-to-invisible attention for occluded regions.The figure describes sparse-structure and structured-latent stages for each object.Source: Figure 2 · p. 3
  • The O-Voxel representation jointly encodes geometry and appearance as per-voxel feature tuples.The supplied passage identifies SLAT as the latent encoding associated with this representation.Source: §2 · pp. 2–3
  • Each stage uses a diffusion transformer with flow matching; the first denoises a dense coarse-grid latent decoded into occupancy, while the second operates on active voxels as coordinate-embedded tokens.Both stages are conditioned on image features from a frozen DINOv3 encoder.Source: §2 · p. 3

3 METHOD

Tetris3D reconstructs single-image scenes by generating objects with pose-aligned geometry and explicit conditioning on neighboring geometry and physical relations. It adds visible-to-invisible attention, a simulation-based interaction dataset, and an inference order based on physical dependencies.

3.1 OVERVIEW

  • Tetris3D conditions object generation on neighboring geometry and physical relations to guide geometrically compatible shapes and poses and avoid invalid configurations such as penetration.Source: §3.1 · p. 3

3.2.1 POSE-ALIGNED OBJECT GENERATION IN THE SCENE

  • Objects are generated in the input camera’s shared scene space, using depth-lifted target points and back-projected image features to align shape generation with the image.Source: §3.2.1 · p. 4, Figure 3 · p. 4

3.2.2 OBJECT INTERACTION-AWARE GENERATION

  • Interaction conditioning combines neighboring-surface distance, direction, and normal with a learned physical-relation embedding for each voxel.Distance conditioning is weighted by proximity, while voxels at or beyond truncation distance τ receive no condition.Source: §3.2.2 · pp. 4–5

3.2.3 VOXEL-ALIGNED CONDITION INJECTION

  • Voxel-aligned image, depth, and interaction conditions are projected and injected into DiT tokens, with zero-initialized learnable gates controlling depth and interaction updates.The update applies to all dense-latent tokens in sparse-structure generation and active-voxel tokens in structured-latent generation.Source: §3.2.3 · p. 5

3.3 V2I ATTENTION: AMODAL COMPLETION VIA VISIBLE-TO-INVISIBLE ATTENTION

  • Visible-to-invisible cross-attention updates occluded tokens using visible tokens as keys and values at each DiT layer.Occlusion is estimated from other objects’ occupied pixels and camera rays through those pixels.Source: §3.3 · p. 5

3.4 COMOB: SIMULATION-BASED COMPOSITED OBJECT SCENE DATASET

  • ComOb contains 1.2M simulated scenes and 3.4M annotated samples spanning stack, lean, contain, pile, and touch interactions, with meshes and physical-relation labels.Scenes are settled into physically stable configurations using a physics engine.Source: §3.4 · p. 5

3.5 INFERENCE PIPELINE

  • At inference, a VLM infers a directed physical-dependency hierarchy, and topological sorting generates supporting objects before the objects that depend on them.For example, the table is generated before a cup that rests on it.Source: §3.5 · p. 6

4 EXPERIMENTS

Tetris3D is reported to outperform scene-generation and amodal-generation baselines across reconstruction, generation, and physical stability, including on real-world benchmarks. Ablations associate interaction conditioning and visible-to-invisible attention with better reconstruction and generation outcomes.

4 EXPERIMENTS

  • Evaluation uses disjoint-source Toys4K scenes with ground-truth depth, plus Picasso and MessyKitchens real-world scenes with estimated depth and VLM-inferred relations.The reported measures cover scene- and object-level reconstruction, generation quality, penetration, settling error, and simulation stability.Source: §4 · p. 7

4.1 COMPARISONS

  • Tetris3D outperforms scene-generation and amodal-generation baselines across reconstruction quality, generative quality, and physical stability, and maintains superior performance on real-world benchmarks.The reported stability improvements include penetration depth and mean displacement.Source: §4.1 · p. 7
  • Qualitative results show plausible shapes and poses under occlusion, preserved interactions without penetration, and stable reconstructions after simulation on real scene images.The real-scene results use autoregressive generation with estimated depth and VLM-based physical-relation reasoning.Source: §4.1 · p. 7

4.2 ABLATION STUDY

  • The ablation study examines interaction conditioning and visible-to-invisible attention, with additional training details reported in Appendix B.Figure 4 presents scene-space reconstructions above individual objects from a common viewpoint; Figure 5 shows reconstructions and post-simulation states.Source: §4.2 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8
  • Without interaction conditioning, shape-generation quality and physical stability decline, with substantially greater penetration depth.The ablation compares the full model with a variant lacking the interaction condition.Source: §4.2 · p. 9, Table 3
  • Removing visible-to-invisible attention reduces generative and reconstruction quality; Amodal3R attention improves on its removal but remains below the proposed design.The comparison evaluates alternative amodal-generation architectures on Toys4K.Source: §4.2 · p. 9, Table 3

5 CONCLUSION

The conclusion presents Tetris3D and ComOb as contributions toward coherent single-image scene reconstruction. The authors also specify that generative AI assisted with manuscript polishing and result visualizations, but not research ideation, design, or analysis.

5 CONCLUSION

  • Tetris3D conditions object generation on neighboring geometry and physical relations, while ComOb provides simulated scenes with diverse interactions and annotations.Experiments on synthetic and real-world benchmarks show improved reconstruction quality and physical consistency, including when interacting regions are occluded.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI was used to polish writing and prepare result visualizations, with authors reviewing the assisted work and retaining responsibility for the final content.The authors state that AI was not used for research ideation, model or experimental design, or analysis.Source: AI USE STATEMENT · p. 10

B Implementation Details

The supplied passages identify sections on dataset-generator and inference-pipeline implementation details, without describing their contents.

B Implementation Details

  • The appendix labels separate sections for dataset-generator details and inference-pipeline details.The passages provide headings only.Source: B Implementation Details · p. 15

C Evaluation Details

The supplied passages review related work on 3D generation and scene generation, then provide model implementation details. They identify evaluation and additional-results sections by heading only.

A RELATED WORKS

  • Prior 3D object-generation methods commonly condition canonical-pose generators on image features, while amodal methods use visibility or occlusion cues but complete objects from their own visible regions.Tetris3D instead conditions generation on neighboring spatial context and physical relations, and completes occluded regions by attending from hidden to visible tokens.Source: A RELATED WORKS · p. 15
  • Scene-generation approaches include asset retrieval, domain-specific scene-native models, and open-set object-native pipelines; later methods address physical plausibility through pose refinement, simulation, or interaction-aware generation.The paper distinguishes its explicit conditioning on spatial context and physical relations from approaches where interactions remain implicit or affect only pose.Source: A RELATED WORKS · pp. 15–16

B.1 MODEL DETAILS

  • The framework uses TRELLIS.2 with a 30-block DiT backbone per generation stage and approximately 1.83B and 1.68B parameters for sparse-structure and structured-latent generation, respectively.Each block includes a lightweight encoder for the per-token nearest-surface condition.Source: §B.1 · p. 16
  • Image-patch features are blended with a learnable null embedding according to occlusion ratio, and the same masking is applied to the projected image condition.The V2I branch applies its residual only to invisible tokens through a zero-initialized gate and is skipped when no invisible tokens are present.Source: §B.1 · p. 16

B.2 TRAINING DETAILS

Training starts from a pretrained TRELLIS.2 backbone initialized from Pixal3D and uses a two-stage occlusion curriculum. Depth perturbations are also applied during training to improve robustness to noisy depth estimates at inference.

B.2 TRAINING DETAILS

  • Tetris3D initializes TRELLIS.2 and its back-projected image-feature conditioning components from Pixal3D, then trains on ComOb.Sparse-structure and structured-latent models train for 200K and 100K iterations, respectively, using AdamW at a learning rate of 1 × 10−4.Source: §B.2 · p. 16
  • The occlusion curriculum first trains on complete images, then mixes complete, primitive-occluded, and synthetically occluded images at a 1:2:1 ratio.Stage 1 and Stage 2 use 50K/150K iterations for sparse-structure generation and 30K/70K for structured-latent generation.Source: §B.2 · pp. 16–17
  • ComOb examples show different interaction types alongside target-object images, scene normal maps, and training-view renders.Source: Figure 8 · p. 17
  • For sparse-structure training, depth augmentation perturbs observed points along camera rays with Gaussian noise and randomly drops 10% of observed depth voxels.Noise perturbation is applied with probability 0.3; image features remain unchanged.Source: §B.2 · p. 17

B.3 DATASET GENERATOR DETAILS

ComOb is generated by simulating target objects with interaction-specific primitives and retaining physically stable scenes. The inference pipeline derives object relations and an ordering for interaction-conditioned generation, while evaluation separates scene-level reconstruction from object-level shape quality and tests physical stability.

B.3 DATASET GENERATOR DETAILS

  • ComOb uses approximately 400K target objects to generate 1.2M scenes, each containing one target and one or more fixed neighboring primitives.Scenes are initialized for five predefined interactions and retained after relation and geometry checks.Source: §B.3 · pp. 17–18
  • Twelve parametric primitive types are selected by interaction category, and their sizes are sampled relative to the target object.Containment sizing also accounts for the target diameter relative to the container opening.Source: §B.3 · pp. 17–18
  • Interaction configurations are checked after simulation: stacking requires at least half-footprint overlap, while leaning requires the target to fall when its support is removed.Containment accepts stable fully contained, partially inserted, or rim-supported targets; pile and touch use distinct placement procedures.Source: §B.3 · p. 18
  • Scenes are retained only when target motion stays below specified linear and angular speed thresholds, and accepted target–primitive pairs receive relation labels.Rendered viewpoints are selected to keep target visibility between 0.3 and 0.9.Source: §B.3 · p. 18

B.4 INFERENCE PIPELINE DETAILS

  • At inference, object instances are identified and segmented, while all other objects’ masks are treated as potential occlusions of each target.The pipeline obtains depth and camera parameters with MoGe3, masks with SAM3, and object instances with Qwen3-VL-30B.Source: §B.4 · pp. 18–19
  • A VLM infers confidence-weighted physical relations, and the ordering procedure resolves conflicting dependencies while preserving higher-confidence directed edges where possible.Directed dependencies determine which object should be generated first; undirected touch relations require an interacting neighbor to be available.Source: §B.4 · p. 19

C.1 EVALUATION PROTOCOL DETAILS

  • Evaluation uses shared scene alignment for scene-level metrics and per-object alignment for object-level shape and generative metrics.Physical stability is tested by simulating reconstructed scenes; penetration is measured before simulation.Source: §C.1 · pp. 19–20

C.2 EVALUATION METRIC DETAILS

The evaluation metrics distinguish geometric reconstruction at scene and object scales, pose and interpenetration errors, post-simulation motion, and generative quality or image consistency.

C.2 EVALUATION METRIC DETAILS

  • Chamfer Distance measures symmetric squared nearest-neighbor discrepancy, with CD-S computed on concatenated scene points and CD-O on individual objects.Source: §C.2 · p. 20
  • F1-S and F1-O are scene- and object-level F-scores formed from precision and recall after thresholding Euclidean nearest-neighbor distances at τ = 0.005.Source: §C.2 · p. 20
  • IoU-B measures volumetric overlap between aligned predicted and ground-truth axis-aligned object bounding boxes after a single rigid scene alignment.Source: §C.2 · p. 20
  • ICP-Rot reports the rotation angle of the rigid transformation aligning a predicted object to its ground truth with ICP.Source: §C.2 · p. 20
  • Penetration depth averages the negative signed distances of fixed target-surface samples inside neighboring objects, with non-penetrating samples contributing zero.The metric uses 20,480 area-uniform target-surface samples.Source: §C.2 · p. 20
  • Dmean measures pose change as the mean displacement of the same fixed target-surface samples over the simulation duration.The metric uses 20,480 samples and T = 300 s.Source: §C.2 · p. 20
  • MMD, COV, and P-FID compare generated-object sets with ground-truth sets using Chamfer-based matching or feature-distribution distance.MMD is reported in units of 10−3 and COV as a percentage.Source: §C.2 · p. 21
  • ULIP and Uni3D measure cross-modal consistency through cosine similarity between generated point-cloud embeddings and the corresponding input image-crop embeddings.Source: §C.2 · p. 21

D.1 QUANTITATIVE RESULTS

With estimated depth, Tetris3D retains the best compared performance across reconstruction and physical-stability metrics, while VLM-inferred relations yield results similar to ground-truth relations. Its generative metrics are also strong, though SAM-3D scores higher on Uni3D and ULIP.

D.1 QUANTITATIVE RESULTS

  • Tetris3D retains the best performance among compared methods across reconstruction and physical-stability metrics with estimated depth.Its lower PD and Dmean indicate less interpenetration and greater stability during simulation.Source: §D.1 · p. 21
  • With estimated depth, Tetris3D achieves the best MMD, COV, and P-FID, while SAM-3D obtains higher Uni3D and ULIP scores.Source: §D.1 · p. 21
  • Replacing ground-truth relations with VLM-inferred relations produces similar reconstruction and generative quality, with only marginal changes.The comparison reports modest increases in MMD and P-FID, with comparable coverage and semantic-similarity scores.Source: §D.1 · p. 21

D.2 QUALITATIVE RESULTS

The qualitative comparisons show how Tetris3D handles occluded object parts and scene placement, including in comparison with agentic generation. Toys4K evaluations also compare estimated-depth settings and VLM-inferred physical relations.

D.2 QUALITATIVE RESULTS

  • Toys4K comparisons include estimated-depth inputs shared across depth-requiring methods and evaluate Tetris3D with ground-truth versus VLM-inferred physical relations.The reported caption states that performance remains comparable with VLM-inferred relations; it provides no numerical values here.Source: Table 5 · p. 22, Table 4 · p. 22
  • On Toys4K examples, several baselines omit occluded parts, distort proportions, or recover poses unlike ground truth, while Tetris3D illustrates complete shapes and scene placement.Examples include thin structures and elongated parts partially occluded by surrounding primitives.Source: §D.2 · pp. 21–22

D.3 COMPARISON WITH AGENTIC SCENE GENERATION

  • Tetris3D more faithfully matches the input image and depth than agentic scene generation, while the agentic approach struggles with scene configuration and produces penetration or unsupported floating.The comparison attributes some geometry discrepancies to script-built CAD meshes and notes that image-based spatial reasoning struggles to place objects precisely.Source: §D.3 · p. 22

E DISCUSSION AND LIMITATIONS

The section discusses limitations of Tetris3D’s interaction-based reconstruction guidance and presents prompts for identifying object relations, ordering generation, refining scenes against rendered comparisons, and exporting editable scenes and meshes with provenance.

E DISCUSSION AND LIMITATIONS

  • Physical interactions guide Tetris3D’s object shape and pose reconstruction, including when interacting regions are occluded.They serve as generation cues, not only criteria for evaluating a completed scene.Source: E DISCUSSION AND LIMITATIONS · p. 22
  • Because interaction conditions are learned guidance rather than hard constraints, Tetris3D cannot guarantee physically valid configurations in every case.Depth, segmentation, and physical-relation errors can distort generation conditions and propagate through the autoregressive process; reducing these dependencies and accounting for uncertainty are future directions.Source: E DISCUSSION AND LIMITATIONS · p. 23

F PROMPT EXAMPLES

  • The physical-dependency prompt distinguishes directed support relations from undirected touch and uses an optimal topological sort to determine generation order.Directed cycles may be retained, with conflicting directed edges deleted according to confidence costs.Source: F PROMPT EXAMPLES · pp. 27–28
  • Represent semantic objects as separate meshes, reproduce observed appearance and spatial relationships, and infer occluded geometry from the image while treating unconstrained scale and hidden structure as assumptions.Estimate camera configuration and lighting from the image for visual comparison.Source: F PROMPT EXAMPLES · p. 30
  • Render the scene from the estimated viewpoint, revise geometry, placement, materials, or camera parameters to reduce visible discrepancies, and validate object separation and mesh usability.Source: F PROMPT EXAMPLES · p. 30
  • Provide an editable Blender scene, complete scene and individual-object mesh exports, generation scripts, preview renderings, an object inventory, and a provenance record.Source: F PROMPT EXAMPLES · p. 30

Related papers

Loading 2610.10539v1…