PaperlayerView exact revision on arXiv

Source-linked AI summary

TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation

Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, Hu Su

arXiv:2512.01204v4cs.CV

TL;DR

Existing scene-generation methods often produce tabletop scenes with broken geometry or physically implausible layouts, while real-world data collection is costly and difficult to scale. TabletopGen creates interactable tabletop scenes from text or a single image, and its scenes and synthetic trajectories show promise for real-world policy learning.

  • Problem

    Existing single-image reconstructions can yield fused or broken topology and physically implausible layouts, while real-world tabletop data collection is costly and difficult to scale.

    Source: §1 · pp. 2–3

  • Method

    TabletopGen is a training-free engine that extracts 3D instances from text or a single image and uses DRO and TSA to recover orientations, translations, and metric scales for tabletop layouts.

    Source: §1 · pp. 3–4, §5 · p. 15

  • Results

    TabletopGen significantly outperforms existing methods in scene quality and physical validity, and zero-shot real-to-sim-to-real transfer experiments confirm the reliability of its scenes and synthetic trajectories for real-world policy learning.

    Source: §1 · p. 4

  • Takeaways & Limitations

    TabletopGen can serve as a reliable data engine for robotic manipulation policy learning.

    Source: §1 · p. 4

    TabletopGen currently focuses on typical tabletop scenes; extending it to multi-level shelves and cabinets remains future work.

    Source: §5.1 · p. 15

Abstract

from arXiv · show

Simulation provides a low-cost, scalable pathway to large-scale robotic manipulation data collection. However, existing 3D scene generation methods can rarely be applied directly to manipulation data synthesis, as their generated scenes often lack instance-level interactivity and physical plausibility. Focusing on tabletop manipulation, we propose TabletopGen, a training-free and automated tabletop scene generation and interactive simulation engine. Starting from text or a single image, we first obtain independent 3D object models via generative instance extraction. Second, we introduce a novel pose and scale alignment approach that recovers a collision-free scene layout using a Differentiable Rotation Optimizer and a Top-View Spatial Alignment mechanism. Finally, we assemble the generated scene in a physics simulator with collision geometry, yielding a stable, interactable environment for synthesizing multimodal manipulation data. Extensive experiments and user studies demonstrate that TabletopGen achieves state-of-the-art performance in visual fidelity, layout accuracy, and physical plausibility. Furthermore, we validate the executability of the collected trajectories on a real robotic arm via zero-shot real-to-sim-to-real policy transfer, indicating that TabletopGen can serve as a reliable data engine for robotic manipulation data synthesis.

1 Introduction

TabletopGen targets the gap between generated tabletop scenes and simulation-ready environments for manipulation data synthesis. It combines instance reconstruction, pose and scale alignment, and simulation assembly, with experiments and real-robot transfer supporting its reported utility.

1 Introduction

  • Existing scene-generation methods often lack instance-level interactivity or physical plausibility, limiting their direct use for tabletop manipulation data synthesis.The cited challenges include fused or broken meshes, interpenetration, and floating objects.Source: §1 · pp. 2–3
  • From text or a single image, TabletopGen extracts independent 3D instances, aligns their rotations, translations, and scales, then assembles them with physical properties in a simulator.DRO estimates rotations, while TSA uses tabletop physical priors to infer globally consistent translations and scales.Source: §1 · pp. 3–4
  • Independent instances make scenes editable by adding, removing, replacing, or rearranging objects, while the framework synthesizes trajectories for zero-shot real-to-sim-to-real policy transfer.The reported transfer uses trajectories synthesized with domain randomization and deploys the resulting policy without real-world data fine-tuning.Source: §1 · pp. 3–4
  • TabletopGen significantly outperforms existing methods in scene quality and physical validity, and zero-shot transfer experiments support the reliability of its scenes and synthetic trajectories for real-world policy learning.The transfer experiment deploys a policy on a real robotic arm without real-world data fine-tuning.Source: §1 · pp. 3–4

2 Related Works

Related work covers digital-twin reconstruction and generative 3D scene synthesis as routes to simulation environments. TabletopGen is positioned as a tabletop-specific data engine that combines instance reconstruction with physical-prior-based alignment.

2 Related Works

  • Prior work primarily follows digital-twin reconstruction from real observations or generative 3D scene synthesis from text or a single image.Simulation data synthesis is presented as a promising response to the cost and scaling constraints of manually collected robot data.Source: §2 · p. 4, §2.2 · p. 4

2.1 Digital Twin and Real-to-Sim-to-Real

  • Digital twins support real-to-sim-to-real deployment with spatial geometry consistent with the real world, but multi-view scanning or manual intervention constrains scene diversity.That dependence makes it difficult to provide the massive scene coverage sought for embodied learning.Source: §2.1 · p. 4

2.2 Generative 3D Scene Synthesis

  • Text-driven scene methods use LLM reasoning or layout solvers, but room-scale designs and fixed asset libraries limit their fit for dense, functional tabletop arrangements.MesaTask targets tabletop tasks but relies on an asset library and places objects on a rectangular plane without modeling the table’s full 3D structure.Source: §2.2 · pp. 4–5
  • Image-driven methods face asset-library limits, broken instance meshes, and physically implausible layouts; compositional methods can accumulate depth or calibration errors and lack physical constraints.Reported layout problems include interpenetration, floating objects, and objects outside the tabletop boundary.Source: §2.2 · p. 5
  • TabletopGen differs by using DRO for visually guided instance rotations and TSA for top-view and stacking-relation priors that constrain boundaries and support relations.The paper positions the resulting globally consistent scenes for manipulation data synthesis and zero-shot real-to-sim-to-real transfer.Source: §2.2 · p. 5

3 Method

TabletopGen turns text or an image into a simulator-ready tabletop scene through canonical instance reconstruction, decoupled pose and scale alignment, and interactive simulation.

3.1 Generative Instance Extraction

  • A multimodal model and GroundedSAM-v2 identify object masks; generative mask completion supports Image-to-3D reconstruction, then rotation correction canonicalizes assets to world Z-up.The resulting assets can be directly placed in tabletop scenes.Source: §3.1 · pp. 6–7

3.2 Pose and Scale Alignment

  • DRO optimizes each instance’s rotation by matching rendered appearance, soft silhouette, and edge contours to the target instance.It combines silhouette, contour, and perceptual appearance losses; gradient descent minimizes the rotation loss.Source: Differentiable Rotation Optimizer (DRO) · p. 7
  • TSA estimates translation and scale from synthesized top-view boxes, stacking relations, and an RMA-selected anchor, constraining objects to tabletop bounds and support relations.The anchor sets global scale and horizontal translations; stacking relations determine vertical translation.Source: Top-View Spatial Alignment (TSA) · p. 8

3.3 Interactive Simulation

  • Isaac Sim applies object transforms and collision geometry with gravity and friction; task planning generates collision-free trajectories for manual or LLM-generated tasks.Adding, removing, replacing, or rearranging instances expands scenes into variants for data synthesis, which includes multi-view images, joint states, and actions.Source: §3.3 · p. 8

4 Experiments

Experiments compare TabletopGen with image- and text-driven baselines, ablate its alignment components, and test synthetic-data policy transfer and scene editing. Results report strong scene-quality and physical-validity performance, real-robot transfer, and low-cost scene expansion.

4.1 Setup

  • The evaluation uses a 78-sample test set spanning synthesized images and real photos, three table shapes, and office, dining, workbench, and crafting settings.Comparisons use identical inputs; evaluation covers perceptual similarity, semantic alignment, collisions, and GPT-4o ratings.Source: §4.2 · p. 9, Metrics · p. 9, Baselines · p. 9

4.2 Comparisons

  • TabletopGen achieves the best image-driven baseline results across evaluation metrics, with a 6.19 GPT-4o average, 43% above MIDI, and collision rates of 0.42% Col_O and 7.69% Col_S.The reported collision rates compare with baseline Col_O values around 8%–17% and Col_S values of 67%–98%.Source: Quantitative Evaluation · p. 10, Table 1 · p. 10
  • For text-to-scene generation, TabletopGen’s average GPT-4o rating was 6.27 versus 4.12 for MesaTask and 4.08 for Holodeck-table, while maintaining near-zero mesh-level collisions.The evaluation covered 30 text-scene pairs; the caption identifies text consistency, object quality, layout realism, overall quality, and collision rates as comparison dimensions.Source: Quantitative Evaluation · pp. 10–11, Table 2 · p. 10
  • In the user study, TabletopGen’s average score was 5.56 versus 3.57 for the second-best method, and it received 83.13% of Overall Preference selections.The study involved 128 participants who rated scenes on Visual Fidelity, Image Alignment, and Physical Plausibility.Source: User Study · p. 11, Table 3 · p. 11
  • Qualitative comparisons show realistic, coherent, collision-free scenes across synthesized and real inputs, and more complete text-generated scenes than MesaTask, including consistent tables and richer instances.MesaTask uses retrieved assets on a fixed plane, while TabletopGen generates the full scene.Source: Qualitative Evaluation · pp. 11–12, Figure 3 · p. 12, Figure 4 · p. 13

4.3 Ablations

  • Removing either DRO or TSA degrades visual and perceptual scores and increases collisions; removing both compounds rotation and placement errors, producing severe occlusions and collisions.The full model closely matches the input and presents a collision-free layout.Source: §4.3 · pp. 12–13, Figure 5 · p. 13, Table 4 · p. 14

4.4 Data Engine Demonstrations

  • Policies fine-tuned on 2,000 synthetic trajectories per task achieved high success rates in simulation and on a real AgileX PiPER arm without real-world fine-tuning.Three tasks were tested for 50 trials each in simulation and the original real-world scene; Task 3 showed a larger sim-to-real gap.Source: Zero-shot Real-to-Sim-to-Real Policy… · pp. 13–14, Figure 6 · p. 14
  • Independent 3D instances allow Add, Remove, Replace, and Rearrange edits without rerunning the full pipeline, producing variants at low additional cost for broader task and interaction-data coverage.The edits preserve overall scene semantics.Source: In-scene Editing for Scene Expansion · pp. 14–15, Figure 7 · p. 15

5 Conclusion

TabletopGen is presented as a training-free, automatic engine for producing interactive, physically plausible tabletop scenes from text or a single image. The paper reports strong evaluation results and zero-shot policy transfer, while noting that its current scope is typical tabletop scenes.

5 Conclusion

  • TabletopGen converts text or a single image into high-fidelity, instance-level, physically plausible tabletop scenes for manipulation data synthesis.Its alignment pipeline uses DRO for instance orientations and TSA for globally consistent translations and metric scales.Source: §5 · p. 15
  • Quantitative evaluations and a large-scale user study report state-of-the-art visual quality, layout accuracy, and physical validity, alongside zero-shot deployment of policies fine-tuned only on synthetic trajectories.The conclusion also presents in-scene editing and text-driven expansion as routes to deriving diverse simulation environments at low cost.Source: §5 · p. 15

5.1 Limitations and Future Work.

  • The current system focuses on typical tabletop scenes; extending it to multi-level shelves and cabinets is future work.The authors also plan to synthesize large-scale, diverse simulated interaction data for policy training.Source: §5.1 · p. 15

Supplementary Material for TabletopGen

The supplementary material adds qualitative results, video descriptions, and experimental configurations. Its scene gallery spans varied tabletop types and shows environments equipped for interactive simulation and data synthesis.

Supplementary Material for TabletopGen

  • The supplementary material provides further qualitative results, video demonstration descriptions, and comprehensive experimental configurations.It includes stage-wise pipeline outputs, extended comparisons, test-set construction, a camera-sweep protocol, user-study details, and prompt templates.Source: Supplementary Material for TabletopGen · p. 20
  • The gallery showcases varied tabletops, including office desks, coffee tables, dining tables, and workbenches, populated with domain-specific objects..Source: Figure 8 · p. 20
  • Every gallery scene has collision geometries and physical properties for immediate use in interactive physics simulation and large-scale data synthesis..Source: Figure 8 · p. 20

6 Additional Qualitative Results

TabletopGen’s additional results show diverse tabletop scenes, traceable intermediate generation stages, and strong visual and physical-reliability comparisons. Its pipeline also supports configurable multimodal data synthesis in simulation.

6.1 Scene Gallery

  • The scene gallery demonstrates generation across varied tabletop types and complex instance layouts.The authors present it as evidence of the framework’s diversity, scalability, and robust generation capabilities.Source: §6.1 · p. 21

6.2 Stage-wise Intermediate Outputs

  • For text input, an LLM expands a short prompt into a detailed scene description before image synthesis; instance masks are then refined into clean, high-resolution images.The reference image conditions later stages, while generative completion addresses holes and blurry mask boundaries.Source: Reference Image Synthesis · p. 21, Figure 9 · p. 21, Instance Completion · p. 22
  • Image-to-3D reconstruction is followed by coordinate canonicalization that aligns each object’s local vertical axis with world up.This prepares the independently reconstructed objects for placement.Source: Canonical 3D Model Generation · p. 22, Figure 11 · p. 23
  • DRO estimates each instance’s rotation by minimizing a tri-modal loss against reference appearance, silhouette, and edge information.For the toy car example, the final rendered projections closely match the reference images.Source: Differentiable Rotation Optimizer · p. 22, Figure 12 · p. 24
  • TSA estimates instance translation and scale from a synthesized top view, detected bounding boxes, commonsense sizes, and stacking relations.The table center defines the origin; bounding-box centers determine xy placement, while stacking order informs z placement.Source: Top-View Spatial Alignment · p. 23, Figure 13 · p. 25
  • After assembly with collision geometries and physical properties, the simulator supports trajectory-driven synthesis of multimodal manipulation data.The basic configuration records synchronized side, front, and wrist RGB views alongside robot joint states; other modalities can be configured.Source: Data Synthesis in Interactive Simulation · p. 24, Figure 14 · p. 26

6.3 Comparison with Recent Methods

  • TabletopGen achieves the best visual scores, near-zero collision rates, 100.00% SS, and 92.13% DSS among the compared methods.SceneGen and FlashSculptor often produce unsupported or out-of-bounds objects and severe interpenetration.Source: Table 5 · p. 25, §6.3 · p. 25

7 Video Demonstrations

The demonstrations show object-level interaction and robot manipulation in generated scenes, culminating in a pick-and-place trajectory replayed successfully on a real arm without fine-tuning.

7.1 Interactive Scene Showcase

  • The interactive showcase demonstrates moving individual objects in scenes, alongside geometry and full-texture views across multiple generated tabletops.The authors describe these demonstrations as evidence of physical viability in physics-enabled simulation.Source: §7.1 · p. 27

7.2 Robotic Manipulation Demonstrations

  • A Franka Emika Panda arm completes kitchen and coffee-table pick-and-place tasks without spurious penetrations or unstable contacts.The demonstrations include moving food items and swapping a cup and an apple.Source: §7.2 · p. 27

7.3 Zero-Shot Real-to-Sim-to-Real Trajectory Transfer

  • A simulated pick-and-place trajectory planned from a real tabletop photo is replayed successfully on a physical AgileX PiPER arm without fine-tuning.The demonstration is presented as validation of the spatial and physical executability of trajectories synthesized in TabletopGen scenes.Source: §7.3 · p. 27, Figure 16 · p. 28
  • The real-world camera viewpoint was not strictly aligned with the simulation camera in the transfer demonstration.The stated objective was to validate the spatial and physical viability of the synthesized trajectories.Source: §7.3 · p. 28

8 Details of Experiments

The experiments evaluate scene generation across diverse tabletop scenes and camera views, with visual comparisons, a human study, and prompts documented. The appendix also discusses how best-view scoring can favor visually plausible views of physically invalid layouts.

8.1 Test Set Construction

  • The 78-scene test set includes 74 synthesized images and 4 real-world photographs, spanning four tabletop-use categories and rectangular, round or oval, and irregular tables.The set contains 66 rectangular, 8 round or oval, and 4 triangular or irregular tables.Source: §8.1 · p. 28

8.2 Visual & Perceptual Quality Details

  • Visual metrics use the best-matching view from a 160-view camera sweep, but this selection can mask physically implausible layouts.The discussion notes that a favorable view can hide floating or out-of-boundary objects in MIDI scenes.Source: Camera Sweep Protocol · p. 29, Discussion on Fairness · p. 29
  • TabletopGen outperforms MIDI on all three visual-perceptual metrics under mean-view evaluation, with larger relative gains than under best-view evaluation.The mean-view protocol averages scores over all 160 rendered views rather than selecting only the closest-matching view.Source: Table 6 · p. 30, Discussion on Fairness · p. 30

8.3 User Study Details

  • The user study gathered anonymous ratings from 128 unpaid volunteers, retaining 126 valid questionnaires after discarding submissions under 200 seconds.The mean completion time was 716 seconds.Source: §8.3 · pp. 30–31
  • Participants rated Visual Fidelity, Image Alignment, and Physical Plausibility on a 1–7 scale, then chose an overall favorite among four randomized method results.Each participant was assigned eight scenes, with results shown in randomized order to mitigate position bias.Source: §8.3 · p. 30

8.4 Prompts

  • The documented prompt set covers text expansion, generative completion, object size and canonical-axis analysis, coordinate canonicalization, top-view synthesis, and stacking-order analysis.The prompt templates are shown in Figs. 21–26.Source: Figure 21 · p. 34, Figure 22 · p. 34, Figure 23 · p. 35

Related papers

Loading 2512.01204v4…