PaperlayerView exact revision on arXiv

Source-linked AI summary

TableVerse: A Large-scale Tabletop Dataset with Real-world Grounded Layouts for Generalizable Manipulation

Boyuan Wang, Yue Zhang, Xutao Xue, Xueyu Song, Yu Sun

arXiv:2607.21017v1cs.RO

TL;DR

Existing automated methods struggle to produce physically consistent tabletop scenes that capture dense, real-world clutter. TableVerse deterministically reconstructs such environments from internet imagery and generates manipulation trajectories, yielding the 100,000-environment TableVerse-100K dataset with strong evaluation results.

  • Problem

    Existing automated generation methods produce simplistic, sparse layouts with geometric collisions and cannot faithfully reconstruct dense clutter and complex topologies from real-world data.

  • Method

    TableVerse converts unstructured internet media into simulation environments through deterministic perception, collision rectification, physics stabilization, and task-conditioned trajectory generation.

  • Results

    TableVerse outperformed all baselines across evaluated dimensions, achieving a 0.0% Scene Collision Rate and leading layout, visual, and geometric quality scores.

  • Takeaways & Limitations

    TableVerse-100K provides 100,000 diverse, physically consistent tabletop environments paired with expert trajectories as a scalable foundation for robotic manipulation research.

  • Takeaways & Limitations

    Low-resolution objects inside containers can be reconstructed incorrectly, while generating 3D models for every scene object is time-consuming.

Abstract

from arXiv · show

The development of generalizable robotic manipulation policies is inherently bounded by the availability of large-scale, high-fidelity scene data. While recent automated synthesis methods attempt to bridge this gap via text-to-layout hallucination or simplified procedural generation, they frequently suffer from physical implausibility and fail to capture the complex, dense clutter of actual human environments. In this paper, we introduce TableVerse, a fully automated Real2Sim pipeline that shifts the paradigm from imaginative layout generation to deterministic reconstruction from unstructured, in-the-wild image data. Our framework seamlessly processes unscripted internet media into high-fidelity, simulation-ready tabletop environments with accurate metric scales, authentic topologies, and verified mechanical stability. Furthermore, an automated task-conditioned trajectory generation framework is integrated to synthesize high-quality, collision-free pick-and-place demonstrations. Leveraging this complete pipeline, we construct the TableVerse-100K Dataset, a large-scale corpus comprising 100,000 unique, physically consistent environments paired with interactive manipulation trajectories. By capturing diverse asset compositions, realistic spatial distributions, and high-quality demonstrations, TableVerse-100K establishes a highly scalable and high-fidelity data foundation, providing significant value to facilitate future research in generalizable robotic manipulation tasks.

1 Introduction

TableVerse addresses the lack of large-scale, physics-ready simulation data by reconstructing interactive tabletop environments from unstructured internet media. Its closed-loop pipeline produces TableVerse-100K, pairing 100,000 physically consistent environments with continuous expert manipulation trajectories.

  • Motivation: Existing automated generation methods produce simplistic, sparse layouts with severe geometric collisions, making them unsuitable for stable physics simulation.These methods fail when confronted with unstructured, in-the-wild real-world data.
  • TableVerse Pipeline: TableVerse converts unstructured internet media into interactive simulation environments through an automated perception-to-simulation workflow.The workflow extracts tabletop assets from single-view observations, restores metric scales, disentangles intersecting meshes, and stabilizes scenes with MuJoCo physics.
  • TableVerse-100K Dataset: 100,000 unique, physically consistent tabletop environments are paired with continuous expert manipulation trajectories in the TableVerse-100K Dataset.The dataset captures dense clutter and heterogeneous physics from real human environments for scaling downstream policy learning.
  • Contributions: The paper contributes the TableVerse Pipeline and LCCR Optimization framework for observation-driven Real2Sim reconstruction and layout-consistent geometric rectification.LCCR disentangles intersecting meshes through hierarchical scene graphs and is tightly coupled with physics-based restoration.

2 Related Work

Prior tabletop-generation methods struggle with geometric fidelity, metric grounding, and physical feasibility, while existing tabletop benchmarks and demonstration pipelines remain limited or dependent on simplified assumptions. Recent work improves grasp synthesis by mapping raw point clouds to continuous 6-DoF grasp spaces, but the supplied passages frame these advances alongside persistent scene-generation and dataset limitations.

  • Automated 3D tabletop generation: Single-view reconstruction methods struggle with dense occlusion, metric constraints, or mesh interpenetration in clustered tabletop scenes.MIDI degrades under dense occlusion, SAM3D lacks metric constraints, and SceneMaker induces severe mesh interpenetrations in dense clusters.
  • Automated 3D tabletop generation: Language-conditioned scene synthesis uses text queries, symbolic graphs, or generated 2D priors but suffers from severe scale errors without continuous geometric grounding.The cited paradigms include LLM-based text queries, symbolic graphs, and generated 2D priors.
  • Tabletop datasets: Specialized tabletop benchmarks remain limited despite abundant large-scale indoor room-level datasets.The passage contrasts abundant room-level datasets with the limited availability of benchmarks focused exclusively on tabletop environments.
  • Tabletop datasets: TO-Scene composes simulation scenes with CAD models through simplistic top-down click-and-drop placement, neglecting hierarchical nesting and vertical stacking.The passage explicitly identifies both omitted spatial structures.
  • Demonstration generation: Modern dense grasping architectures map raw point clouds directly to continuous 6-DoF grasp spaces, replacing earlier reliance on templates or heuristic descriptors.Earlier analytical or data-driven methods are described as depending heavily on pre-modeled templates or heuristic descriptors.

3 Method

TableVerse presents a deterministic monocular real-to-sim pipeline that reconstructs high-fidelity, simulation-ready tabletop environments and continuous expert trajectories from unstructured internet media. Its method combines hybrid object extraction, physics-guided asset assembly, metric-scale pose alignment, layout-consistent collision rectification, and task-conditioned trajectory generation.

  • Method overview: TableVerse replaces probabilistic spatial hallucination with a deterministic perception-to-physics workflow for reconstructing simulation-ready tabletop scenes and expert trajectories.The workflow is designed to preserve real-world metric scales, object topologies, and contact mechanics.
  • Instance extraction: Open-vocabulary detection classifies regular and composite objects, while SAM2 produces precise instance segmentation masks from representative bounding boxes and instance counts.Seed-1.8 performs direct open-vocabulary detection, avoiding error accumulation from cascading an MLLM with GroundingSAM-v2.
  • Composite asset generation: SAM3D reconstructs container and nested-content meshes separately, then MuJoCo free-fall simulation assembles composite objects into physically valid configurations.This physics-guided assembly addresses severe occlusions in in-the-wild images and enriches asset diversity and interaction complexity.
  • Metric scale and pose alignment: Depth Anything 3 extracts scene point clouds, while table or floor masks estimate gravity and a global alignment transformation canonicalizes scene orientation for metric-scale pose recovery.The procedure is applied before segment decomposition to recover accurate metric scales and spatial intervals.
  • Layout-consistent collision rectification: LCCR resolves mesh interpenetrations through contact grouping, radial horizontal rectification, and vertical disentanglement while preserving macroscopic layout and azimuthal semantics.The module addresses destabilizing repulsive torques caused by naïve physics initialization in dense clutter and translates smaller-footprint intersecting objects upward when needed.
  • Trajectory generation: An automated task-conditioned framework converts high-level manipulation instructions into continuous executable trajectories through closed-loop coupling of MLLM scenario assessment and trajectory synthesis.Procedurally generated task instructions are routed directly into a specialized manipulation trajectory-generation process.

4 Experimental Results

On 100 unscripted tabletop samples, TableVerse achieves physically valid, high-fidelity scene reconstruction and outperforms representative single-view baselines. Ablations show that LCCR removes collisions, while simulation resolves artifacts into stable layouts.

  • Quantitative and Qualitative Analysis: TableVerse achieves a 0.0% Scene Collision Rate and the top average cross-method GPT-Score ranking of 1.38.The evaluation covers 100 unscripted tabletop samples spanning dense layouts, stacking, nested assets, low resolution, and cluttered backgrounds.
  • Quantitative and Qualitative Analysis: SceneMaker suffers under dense clutter and complex backgrounds, while another baseline reaches only 4.83 GQ on fine-grained geometric structures.The reported deficiencies reflect structural and physical vulnerabilities on unscripted internet data.
  • Quantitative and Qualitative Analysis: TableVerse leads in Layout Fidelity at 7.14 LF, Visual Quality at 7.08 VQ, and Geometry Quality at 7.03 GQ.These scores are reported against real-world references and accompany elimination of physical anomalies.
  • LCCR Ablation: Direct alignment produces a 79.0% collision rate, making raw outputs unsimulable because of perception noise.This configuration aligns 3D meshes to feed-forward depth point clouds without collision handling.
  • LCCR Ablation: LCCR reduces the collision rate to 0.0% through geometric mesh disentanglement, validating its layout-preserving optimization.The method uses horizontal radial expansion and vertical footprint-based sorting; geometric rectification can still leave micro-gaps or floating assets.
  • LCCR Ablation: The complete LCCR-plus-simulation pipeline settles rectified objects into a stable, physics-ready resting state.Forward simulation addresses artificial micro-gaps and floating assets left by pure rigid translations.

5 Limitation

The method remains limited by SAM3D and its image inputs: low-resolution objects inside containers may be reconstructed incorrectly, while modeling every scene object is time-consuming.

  • Input and reconstruction limitations: Low-resolution container objects occupying few pixels can cause SAM3D to generate entirely different objects.The failure arises when small objects provide insufficient visual detail in the input data.
  • Input and reconstruction limitations: Generating 3D models for all objects in an entire scene using SAM3D is time-consuming.This computational burden limits the efficiency of processing complete scenes.

6 Conclusion · A Open-Vocabulary Object Detection and Prompt Details

The paper concludes with a fully automated Real2Sim pipeline that generates simulation-ready desktop scene assets from a single in-the-wild image, including tangible container contents through LCCR+Mujoco. Its open-vocabulary perception front-end uses structured prompts and hybrid post-processing to organize detections and container composites for downstream mask generation.

  • 6 Conclusion: The pipeline generates simulation-ready desktop scene assets from a single in-the-wild image without human intervention.It is described as a fully automated Real2Sim scene-synthesis pipeline.
  • 6 Conclusion: LCCR+Mujoco enables generated scene assets, including objects inside containers, to be directly loaded and used in simulation.The combined asset-synthesis method addresses objects being visible but not tangible.
  • A Open-Vocabulary Object Detection and Prompt Details: Seed-1.8 serves as the open-vocabulary detector for localizing arbitrary, non-predefined tabletop objects from single-view images.A structured text prompt guides scene reasoning and background-clutter filtering, including human hands and body parts.
  • A Open-Vocabulary Object Detection and Prompt Details: The hybrid detection prompt separates outputs into regular detections and composite objects labeled container_fill.The prompt is presented as an exact compiled template for deterministic open-vocabulary detection.
  • A Open-Vocabulary Object Detection and Prompt Details: Human elements are excluded, and when a full person is visible, only objects in front of the person are retained.Objects behind the person are dropped when their bounding boxes have more than 50% area overlap with the person’s body.
  • A Open-Vocabulary Object Detection and Prompt Details: Container composites include only discrete, solid, independently generatable objects, while empty or unsuitable containers remain regular detections.Liquids, powders, sauces, foam, reflections, glare, shadows, and generic materials are excluded from container contents.
  • A Open-Vocabulary Object Detection and Prompt Details: For repeated contents, the prompt requires one representative-instance bounding box and uses count to record the total number.Bounding-box coordinates are normalized in [0, 1000], and each foreground object must appear exactly once.
  • A Open-Vocabulary Object Detection and Prompt Details: Post-processing sends regular detections to SAM2, while container coordinates anchor composites and their content boxes seed downstream masks.Content bounding boxes are dynamically expanded into pixel-level seeds.

B Detailed Dataset Curation and MLLM Annotation Pipeline

The curation pipeline uses Gemini 2.5 Pro to analyze two-view tabletop scenes, combining usability, labeling, physical-property, quality, relabeling, and task-generation outputs. It enforces structured criteria for scene validity, object annotation, and physically feasible pick-and-place tasks.

  • Input and annotation: Gemini 2.5 Pro analyzes front tilted and top-down views of the same tabletop scene, using object IDs, bounding boxes, and user-provided label mappings.The two viewpoints jointly support scene judgment and object-level annotation.
  • Pipeline outputs: The system produces seven outputs: usability gate, severe label-error check, label confidence, physical properties, quality score, object relabeling, and pick-and-place task generation.Parts A–E guide curation, while relabeling and task generation are required regardless of usability.
  • Curation gates: Usability is false for hard failures such as extreme regional partitioning or non-tabletop objects, while severe label errors require clearly incorrect labels and corrected alternatives.Synonyms, hypernyms, stylistic or material inaccuracies, low-poly geometry, and generic labels are not severe errors.
  • Object attributes: Each object receives confidence in [0, 1] and predicted mass_kg, hinge status, and an allowed hinge_category, with conservative property estimates below 0.7 confidence.Hinge categories are restricted to specified enums or null.
  • Quality assessment: The quality score is 1–10 and weights object count and category diversity at 45%, geometric plausibility at 15%, graspability at 15%, and layout clarity at 25%.Scenes with clutter density n ≥5 are rewarded, while distorted geometry, poor graspability, and severe layout violations are penalized.
  • Task generation: Generated tasks use the fixed pick-and-place template and relation enum, require distinct valid object IDs, an initially unsatisfied relation, and physical feasibility.The picked object must be graspable and the target placement must fit; containment relations are restricted by the specified constraints.

C Implementation and Evaluation Details · C.1 Baseline Implementation Details

The evaluation standardizes visual inputs across baselines by supplying high-fidelity instance masks, then addresses SceneMaker’s sparse point clouds under challenging real-world conditions through instance-level up-sampling. These implementation choices target fair comparison, object omission, and regression instability.

  • C.1 Baseline Implementation Details: All evaluated baselines receive a standardized visual input front-end for rigorous and fair comparison.The design uses a common input protocol across baselines.
  • C.1 Baseline Implementation Details: MIDI, SAM3D, and SceneMaker are uniformly supplied with high-fidelity instance masks from the perception pipeline.These baselines lack native open-vocabulary object discovery and autonomous mask extraction.
  • C.1 Baseline Implementation Details: The isolate-and-compare design uses shared masks to evaluate downstream baseline differences under a controlled visual-input setting.The supplied passage states that this design is intended to ensure a rigorous and fair comparison.
  • C.1 Baseline Implementation Details: SceneMaker’s pose and scale regression networks condition on a fixed 1024-dimensional point cloud extracted for each instance.This fixed-dimensional representation is an inherent condition of SceneMaker’s implementation.
  • C.1 Baseline Implementation Details: Low resolution and dense object clustering make the in-the-wild benchmark especially difficult for heavily occluded or small objects.Such objects can produce minuscule mask pixel footprints and deficient, sparse point counts.
  • C.1 Baseline Implementation Details: An instance-level point cloud up-sampling step mitigates data sparsity, prevents object omission, and safeguards regression stability.Sparse point counts from small or heavily occluded objects can trigger catastrophic failures in 6-DoF pose estimation.

C.2 GPT-Score Evaluation Prompt Source

The GPT-Score evaluation uses an MLLM prompt that compares a reference tabletop image with standardized renderings from four reconstruction methods. It evaluates layout, visual quality, geometry, and overall simulation readiness using unique rankings and structured JSON output.

  • Evaluation Metrics: The prompt scores Layout Fidelity, Visual Quality, and Geometry Quality on a 1–10 scale.Layout Fidelity covers object relationships, scale, pose, and position; Visual Quality covers rendering and texture naturalness; Geometry Quality covers category, proportions, and structure.
  • Input Layout: The MLLM receives a concatenated 2×5 grid containing the reference image, front-oblique renderings, and pure front-view renderings for MIDI, SAM3D, SceneMaker, and TableVerse.The first row contains the reference followed by front-oblique views; the second row contains a blank slot followed by front views in the same method order.
  • Evaluation Scope: Evaluation is restricted to tabletop objects, excluding the background and tabletop surface, with reconstructed front views using the default upward tabletop-normal direction.The reference may use non-standard viewpoints, but reconstructed front views must follow the specified vertical direction.
  • Overall Ranking: Methods are assigned unique integer ranks from 1 to 4 based on the three quality metrics and overall simulation readiness.Rank 1 denotes the best method and Rank 4 the worst.
  • Output Format: The evaluator must return only a valid JSON object, with single-line reason strings and no additional prose or markdown.The prompt also specifies an exact JSON schema defined in the source code.

D Algorithmic and Mathematical Details of Trajectory Generation … D.3 Coordinate Alignment and Hybrid Obstacle Arbitration

The trajectory-generation pipeline preprocesses point clouds, relaxes grasp constraints hierarchically, constructs relation-specific placement domains, and aligns obstacles to the robot frame with mesh-based collision handling and fallbacks.

  • D.1 Point Cloud Pre-processing and Multi-Stage Grasp Relaxation: The object point cloud is centered at its spatial centroid before GraspGen predicts 6D grasp candidates.The translated cloud is ingested by GraspGen for candidate generation.
  • D.1 Point Cloud Pre-processing and Multi-Stage Grasp Relaxation: Grasp selection uses three stages: strict top-down filtering, hemispheric downward relaxation, and confidence-based fallback when geometry constraints eliminate all candidates.The final stage bypasses directional filtering and selects the highest-confidence grasp.
  • D.2 Mathematical Construction of Relation-Specific Placement Domains: The placement domain ΩR(Oref) is analytically derived from the reference asset’s bounding-box half-extents and centroid according to the spatial relation R.Candidates are sampled within these relation-specific domains and rejected until AABB non-penetration criteria are satisfied.
  • D.2 Mathematical Construction of Relation-Specific Placement Domains: For top or in relations, Ωtop restricts placements to a horizontal plane over the reference asset’s upper face.The domain bounds x and y by [cx − bx, cx + bx] and [cy − by, cy + by], with z = cz + bz.
  • D.2 Mathematical Construction of Relation-Specific Placement Domains: For directional adjacency, Ωright is a projected half-space offset along a principal axis, with y > cy + by + δmargin and z = cz − bz.δmargin is a baseline physical parsing buffer intended to prevent initial contact.
  • D.3 Coordinate Alignment and Hybrid Obstacle Arbitration: Before cuRobo trajectory generation, non-target scene items are mapped into the robot-base frame using the known transform Tworld→robot ∈ SE(3).Each asset pose in the world frame is converted into planning coordinates.
  • D.3 Coordinate Alignment and Hybrid Obstacle Arbitration: The planner preferentially represents environmental boundaries as high-fidelity non-convex triangular meshes to match real-world geometries in narrow spaces.This hierarchy targets difficult regions such as hollow-container interiors and clustered shelves.
  • D.3 Coordinate Alignment and Hybrid Obstacle Arbitration: If mesh generation encounters non-manifold vertices or topological degeneracy, the planning scene dynamically activates a fallback collision representation.The passage specifies the fallback arbitration mechanism but does not provide its complete representation in the supplied text.
Loading 2607.21017v1…