PaperlayerView exact revision on arXiv

Source-linked AI summary

MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning

Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu, Feng Zheng, Lizhuang Ma, Jiangmiao Pang

arXiv:2509.22281v1cs.CVcs.RO

TL;DR

The paper addresses the gap between high-level manipulation instructions and realistic, task-conforming tabletop scenes. It introduces MesaTask-10K and an LLM framework built around a Spatial Reasoning Chain and DPO, and reports superior performance over baselines. The approach is scoped to six indoor table types and depends on the available 3D object database.

  • Problem

    Task-oriented tabletop scene generation is difficult because datasets are scarce and high-level task instructions must be mapped to structured scene layouts.

  • Method

    MesaTask combines the MesaTask-10K dataset, a Spatial Reasoning Chain for object and relation reasoning, LLM-based layout generation, asset retrieval, and DPO training.

  • Results

    MesaTask outperforms baseline methods across FID, VLM-based metrics, and user studies while generating physically plausible, realistic, task-conforming scenes.

  • Takeaways & Limitations

    The paper establishes task-oriented tabletop scene generation as a benchmark supported by a large manually crafted dataset and a framework for structured 3D spatial reasoning.

  • Takeaways & Limitations

    MesaTask covers only six indoor table types and limits object diversity to the available 3D object database.

Abstract

from arXiv · show

The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce MesaTask-10K, a large-scale dataset comprising approximately 10,700 synthetic tabletop scenes with manually crafted layouts that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a Spatial Reasoning Chain that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present MesaTask, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts. Project page is at https://mesatask.github.io/

1. Introduction

MesaTask addresses the gap between high-level manipulation instructions and plausible tabletop scenes by introducing a dataset and an LLM framework with structured spatial reasoning. Its experiments report stronger task conformity, physical plausibility, and realism than baseline methods.

  • 1. Introduction: MesaTask-10K and the MesaTask framework target task-oriented tabletop scene generation, a problem constrained by limited datasets and the gap between task descriptions and scene layouts.The dataset uses manually crafted layouts, while the framework generates scenes from high-level instructions.
  • 1. Introduction: MesaTask-10K contains approximately 10,700 synthetic scenes spanning six indoor table categories, built from over 12,000 3D assets with manually refined layouts.The layouts are refined to address inaccuracies and complex inter-object relations.
  • 1. Introduction: The Spatial Reasoning Chain decomposes generation into object inference, inter-object relation reasoning, and scene graph construction before producing a concrete 3D layout.This structured chain is designed to bridge natural-language tasks and spatially structured scenes.
  • 1. Introduction: MesaTask combines an LLM-based generation framework with DPO training to reduce collisions, incorrect relations, and omissions of task-relevant objects.DPO is used to improve conformity with task descriptions and physical plausibility.
  • 1. Introduction: Experiments report that MesaTask outperforms baselines in FID, VLM-based metrics, and user studies while generating realistic layouts and rich relations such as stacking and containment.The reported evaluation covers task-scene alignment and physical viability among other dimensions.

2. Related Work

Prior work reconstructs or generates tabletop scenes from images, language, or abstract geometric relations, but existing approaches do not fully resolve complex 3D relations or under-specified task instructions.

  • 2. Related Work: Prior tabletop datasets include 2D scene images and 3D rearrangement data guided by language or abstract geometric relations.These resources differ in whether they provide 2D appearance, 3D structure, or language-guided rearrangement.
  • 2. Related Work: Single-image reconstruction methods map image features into 3D space, while other systems generate scenes with diverse objects but struggle with complex inter-object relationships.The related methods use encoder-decoder reconstruction or 3D object generation pipelines.
  • 2. Related Work: LLM-based methods use in-context learning or multi-stage prompting for open-vocabulary and dataset-free 3D scene generation.These approaches demonstrate the use of language models for direct layout generation and scene synthesis.

3. MesaTask-10K Dataset

MesaTask-10K is constructed by converting language-generated tabletop descriptions and synthesized images into refined, physically plausible 3D scenes. It provides broad object coverage and diverse layouts for task-oriented scene generation.

  • 3. MesaTask-10K Dataset: MesaTask-10K is built from diverse scene images generated from object lists and spatial relations for six indoor table types.The categories include office tables, dining tables, kitchen counters, coffee tables, bathroom vanities, and dressing tables.
  • 3. MesaTask-10K Dataset: The dataset uses an asset library of over 12,000 rigid and interactive objects with semantic information for retrieval and coarse 3D scene construction.Object detection, semantic labeling, depth estimation, and 3D bounding boxes support the initial layouts.
  • 3. MesaTask-10K Dataset: The construction pipeline converts language-generated descriptions into reference images, coarse layouts, and human-refined scenes validated through physical simulation.Human annotation corrects scale, positions, redundant instances, collisions, and floating objects.
  • 3. MesaTask-10K Dataset: MesaTask-10K contains approximately 10,700 scenes, more than 200 object classes, and roughly 15 objects per scene on average.The dataset statistics characterize both scene diversity and object-count distribution.
  • 3. MesaTask-10K Dataset: The dataset supports the framework’s reasoning data and final scene generation by pairing scene descriptions, spatial relations, and 3D layouts.The framework overview connects dataset-derived scene graphs with reasoning and DPO data construction.

4. Method

MesaTask maps task instructions to 3D tabletop layouts through a spatial reasoning chain and asset retrieval, then uses SFT and DPO to improve layout quality. The method explicitly models objects, relations, and scene graphs while training against collision, relation, and object-omission errors.

  • 4. Method: Given a task instruction, MesaTask extracts the environment, decomposed goals, and task-relevant objects before generating a 3D layout and retrieving corresponding assets.The complete scene may include recommended objects and additional objects for realistic layouts.
  • 4. Method: The Spatial Reasoning Chain performs object-list completion, interrelationship inference, and scene graph construction to bridge task language and 3D layouts.The resulting graph is used to guide concrete layout generation.
  • 4. Method: The method enriches scene-graph nodes with coarse positions and orientations, using eight orientation categories and a 3 × 3 table-position grid.These discretized spatial cues provide additional guidance for layout generation.
  • 4. Method: MesaTask combines an LLM layout generator with post-processing for 3D asset retrieval and trains the model using spatial reasoning data and supervised fine-tuning.The framework is designed to generate structured 3D layouts from high-level task instructions.
  • 4. Method: DPO contrasts high-quality layouts with negatives containing collisions, damaged relations, or removed critical objects to reduce SFT-era layout failures.The preference objective favors normal layouts and targets the three stated corruption types.

5. Experiment

MesaTask is evaluated against language-model and modular scene-generation baselines using quantitative, qualitative, ablation, and generalization studies. It achieves stronger task-scene alignment, realism, layout plausibility, and inter-object reasoning, while DPO and spatial reasoning improve generation quality and performance extends to unseen tabletop categories.

  • Experiment setup: The evaluation generates 500 tabletop scenes per method and combines FID, GPT-based metrics, qualitative comparisons, and a 127-participant user study.The user study further confirms the favorable human preference for MesaTask’s generated scenes.
  • Comparison to baselines: MesaTask achieves the best generation performance across all evaluation metrics, outperforming the compared baselines.Table 1 reports the overall quantitative comparison, while FID and GPT-based metrics assess realism and task-related qualities.
  • Comparison to baselines: MesaTask produces more realistic, diverse, and task-aligned layouts than three representative baselines, including meaningful stacking, containment, and object relocation.Baselines more often generate simplistic or symmetrical layouts, omit key objects, or mishandle complex spatial commands.
  • Ablation study: DPO improves over SFT-only training by reducing collisions, preserving task-relevant objects, and strengthening alignment with task instructions.The ablation attributes more coherent layouts, correct interrelations, visual plausibility, and functional fidelity to supplementary DPO training.
  • Generalization capability: MesaTask generalizes to four unseen tabletop categories with performance comparable to the six seen categories, although FID cannot be computed for these new scenes.On cashier counters, it describes and places cash registers correctly despite their absence from MesaTask-10K.

6. Conclusion

The paper concludes that task-oriented tabletop scene generation addresses the gap between high-level instructions and concrete scene layouts through MesaTask-10K and a spatial reasoning-based LLM framework. Its implementation combines asset retrieval, spatial reasoning, scene-graph construction, layout generation, and human refinement to support task-conforming 3D scenes.

  • 6. Conclusion: MesaTask-10K contains roughly 10,700 tabletop scenes across six indoor table categories, supporting the proposed task-oriented generation problem.The conclusion positions the dataset as support for a task with a substantial disparity between task instructions and scene layouts.
  • 6. Conclusion: MesaTask sequentially reasons about scene composition, spatial interrelationships, scene graphs, and final 3D layouts before retrieving assets to form tabletop scenes.The framework operationalizes the proposed spatial reasoning chain for task-conditioned generation.
  • 6. Conclusion: The paper reports superior performance over existing baselines in following task instructions and modeling complex inter-object relations.The claimed advantages concern task conformity and relational modeling rather than a single evaluation metric.
  • 6. Conclusion: Human annotators refine coarse layouts to correct issues such as occlusion, inaccurate depth, retrieval errors, unreasonable object sizes, and floating objects.Scenes are rendered from four directions and revised until they meet acceptance criteria.

A.5. Dataset statistics

The 3D asset database contains over 200 common tabletop object categories with numerous high-fidelity 3D models.

  • The 3D asset database covers over 200 common tabletop object categories and includes numerous high-fidelity 3D models.

A.6. Tabletop scene generation benchmark

Table 3 benchmarks ATISS, DiffuScene, and PhyScene on five MesaTask-10K tabletop environments using FID, KID, and CKL. Results reveal a trade-off between visual quality and semantic alignment, while MesaTask uses a task-instruction setting that is not directly comparable.

  • Benchmark setup: Table 3 evaluates ATISS, DiffuScene, and PhyScene across five tabletop environments using FID, KID, and CKL, with lower metric values indicating better performance.Bathroom vanity is excluded because its sink area poses a problem these methods cannot handle.
  • Method-specific patterns: ATISS remains consistent across scenes despite slightly lagging in FID and KID, while PhyScene is competitive on complex Kitchen Counter and Dressing Table layouts.The passage attributes PhyScene’s performance in interaction-heavy settings to its physical constraints.
  • Overall findings: DiffuScene produces visually coherent layouts, whereas ATISS and PhyScene better align with real-world object-category distributions.CKL exposes semantic-distribution differences that traditional image-level metrics may miss.
  • Comparison scope: MesaTask generates scenes from task instructions, unlike the compared methods, which generate scenes from simple scene descriptions, precluding direct comparisons.MesaTask also supports open-vocabulary tabletop scene generation through a fine-tuned language model.
  • MesaTask inference: Asset retrieval combines textual similarity and size similarity into a weighted score, prioritizing textual relevance with α=0.9 and β=0.1.The highest-scoring library asset is placed according to the target position and size.

B.2. Reasoning data construction

Reasoning data construction extracts interpretable spatial relations from tabletop scenes and enriches them with multimodal scene descriptions and task information. These rules form scene graphs used to train the Spatial Reasoning Chain.

  • Task annotation: GPT-4o supplements rule-based graphs with rendered-scene descriptions, then generates complete object lists, inter-object relations, and corresponding task instructions.The multimodal process helps compensate for spatial relations missing from the initial scene graph.
  • Scene graph extraction: Scene graphs encode object relations using geometric rules based on relative positions, distances, orientations, overlap, and table-scale thresholds.Horizontal relations use centroid differences, vertical relations require sufficient horizontal overlap, and containment requires high overlap ratios.
  • Relation types: The extraction rules represent directional, vertical, containment, orientation, grid-location, and equal-spacing relations between objects.Face to uses eight directional bins, Is at uses a 3×3 table grid, and equal spacing detects linear arrangements of at least three objects.

B.3. DPO data construction

DPO data construction pairs high-quality layouts with corrupted alternatives that model collision, semantic-misalignment, and missing-object failures. These preference pairs train alignment toward task-compliant scene layouts.

  • Preference-pair design: Each DPO sample pairs a prompt with a preferred completion and a dispreferred completion, allowing relative layout-quality judgments to provide alignment signals.Positive layouts come from MesaTask-10K and include reasoning traces, structured geometry, and symbolic scene graphs.
  • Negative-sample construction: Negative layouts simulate three failure modes: geometric perturbation, scene-graph corruption, and removal of task-relevant objects.These correspond respectively to physical collisions, semantic misalignment, and geometrically valid but functionally incomplete scenes.
  • Training implementation: Two independent rejected completions are sampled per prompt using different corruption methods or random seeds to diversify supervision.The chosen completion retains the original reasoning trace followed by the correct structured layout.
  • Baseline construction: Holodeck-Table generates layouts by using a language model for object and relation specification, followed by asset retrieval and optimization-based placement.The desktop adaptation removes room-scale modules and modifies optimization for tabletop constraints.

C.2. Details of metrics

Evaluation combines output validity, rendered-image realism, multimodal semantic assessment, and human ratings. The metrics cover both perceptual similarity and task-relevant spatial quality.

  • Validity: Success rate measures the proportion of LLM responses that follow the expected format and contain at least one valid object.It is computed as successful responses divided by total test cases.
  • Image realism: FID compares deep-feature distributions from rendered generated and ground-truth scenes as a proxy for perceptual realism and coherence.Images are rendered from a front-facing view, resized to 299 × 299, and represented using 2048-dimensional Inception-V3 features.
  • Multimodal evaluation: GPT-based scoring rates task consistency, object-size reasonableness, placement plausibility, layout realism, and object visibility on a 1–10 scale.The model evaluates both front-view and perspective-view images together with the task description.
  • Human evaluation: The user study rates realism, task alignment, and spatial coherence on seven-point scales, averaging the three dimensions into a scene score.127 participants evaluated randomly ordered, method-anonymous images sampled through the study interface.

D. More result

Additional evaluations show that MesaTask handles tasks across four difficulty levels and produces physically plausible scenes after physics-based post-processing. Qualitative comparisons further illustrate differences in object selection and arrangement across methods.

  • Collision-free scenes are produced after physics-based post-processing, satisfying the paper’s physical-plausibility criterion.
  • Scenes across all four task difficulty levels achieve high scores in the multi-dimensional assessment.The evaluation uses 500 tasks per level spanning six common indoor tabletop types.
  • FID varies across task levels, while the training set is dominated by Level 4 tasks.The reported training distribution is 83.8% Level 4, 11.4% Level 3, 3.8% Level 2, and 1% Level 1.
  • Qualitative comparisons under identical instructions show differences among MesaTask, GPT-4o, I-Design-table, and Holodeck-table in object selection and arrangement.Additional MesaTask examples cover cleaning, organizing, and preparing tabletop scenes.

E. Limitation and future work

MesaTask is currently limited to six common indoor table types and to objects available in its 3D database. Future work proposes integrating bounding-box-conditioned 3D object generation to broaden object variety and realism.

  • MesaTask covers six common indoor table types rather than the full range of everyday tables.Examples outside the current scope include cashier counters and conference tables.
  • 3D object retrieval limits object diversity to assets available in the 3D object database.
  • Future work will integrate bounding-box-conditioned 3D object generation into the tabletop scene generation pipeline.The stated goal is to create more varied objects and more realistic tabletop scenes.
Loading 2509.22281v1…