PaperlayerView exact revision on arXiv

Source-linked AI summary

Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation

Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, Manolis Savva

arXiv:2306.11290v3cs.CV

TL;DR

Synthetic scene datasets must balance scale with realism for ObjectGoal navigation transfer to real-world environments. This paper introduces HSSD-200 and systematically compares scene scale and realism, finding that smaller, higher-quality scenes generalize better than larger, procedurally generated datasets.

  • Problem

    Limited evidence has compared synthetic scene scale and realism for training ObjectGoal agents to generalize to real-world scans.

  • Method

    The paper constructs HSSD-200, a human-authored dataset of 211 realistic houses and 18,656 real-world object models, then compares it with prior synthetic datasets.

  • Results

    Smaller, higher-quality synthetic scenes produce better ObjectNav generalization than larger or lower-quality datasets, with HSSD-122 exceeding ProcTHOR-122 on HM3DSem success, 19.15 vs 9.33.

  • Takeaways & Limitations

    For ObjectNav transfer to realistic scenes, realism and correspondence to real-world environments can matter more than synthetic scene scale.

  • Takeaways & Limitations

    The investigation is limited to monolithic pixels-to-actions agents trained end-to-end with reinforcement learning, excluding modular and imitation-learning approaches.

Abstract

from arXiv · show

We contribute the Habitat Synthetic Scene Dataset, a dataset of 211 high-quality 3D scenes, and use it to test navigation agent generalization to realistic 3D environments. Our dataset represents real interiors and contains a diverse set of 18,656 models of real-world objects. We investigate the impact of synthetic 3D scene dataset scale and realism on the task of training embodied agents to find and navigate to objects (ObjectGoal navigation). By comparing to synthetic 3D scene datasets from prior work, we find that scale helps in generalization, but the benefits quickly saturate, making visual fidelity and correlation to real-world scenes more important. Our experiments show that agents trained on our smaller-scale dataset can match or outperform agents trained on much larger datasets. Surprisingly, we observe that agents trained on just 122 scenes from our dataset outperform agents trained on 10,000 scenes from the ProcTHOR-10K dataset in terms of zero-shot generalization in real-world scanned environments.

1. Introduction

Simulation enables systematic and scalable embodied-AI training, but existing 3D scene datasets involve unresolved tradeoffs between scale, realism, and real-world generalization. HSSD-200 addresses this gap with human-authored, compositional scenes designed to more closely mirror real environments.

  • Motivation: Simulation supports safe, systematic, and scalable training and evaluation of embodied AI agents before real-world deployment.These agents perform navigation, language following, object finding and rearrangement, and other embodied sensing, planning, and acting tasks.
  • Motivation: Existing reconstruction and synthetic datasets have complementary limitations, including noisy or incomplete geometry, artifact-specialized agents, difficult scaling, and limited scene manipulation.Reconstructions capture real-world diversity but can contain holes and missing thin or shiny geometry, while their monolithic representations hinder object additions, removals, and state changes.
  • Contribution: Synthetic scenes provide compositionality that enables object removal and addition, operations that are difficult in reconstructions because they require segmentation and hole infilling.This flexibility is important for embodied tasks requiring environment interaction.
  • Research Gap: Prior work lacked a systematic analysis of how synthetic-dataset scale and realism affect embodied-agent training and real-world generalization.The paper identifies dataset scale and realism—including visual fidelity and correlation to real-world statistics—as central tradeoffs.
  • Contribution: HSSD-200 is a human-authored 3D scene dataset intended to mirror real scenes more closely than prior datasets.It recreates real houses using 18,656 unique, high-quality 3D models of real objects and is planned for free release under a permissive academic research license.

2. Related work

Prior ObjectNav research spans scanned and synthetic 3D environments, with performance shaped by training-data scale and environmental complexity. However, limited synthetic datasets have constrained systematic study of the tradeoff between scene scale and quality.

  • ObjectGoal navigation: ObjectNav methods comprise end-to-end reinforcement learning, imitation learning, and modular learning approaches.Reinforcement-learning methods map visual observations to action probabilities and learn from rewards based on progress toward the goal object.
  • 3D scene datasets: ObjectNav research uses both scanned real-world environments and synthetic datasets, with performance depending on training-data scale and environmental complexity.Studies have progressed from small single-room environments toward larger multi-room environments.
  • 3D scene datasets: Procedurally generated scenes have improved ObjectNav performance and transfer to real-world scans.Deitke et al. demonstrated both improvements using procedurally generated scenes.
  • 3D scene datasets: Systematic analysis of the scale–quality tradeoff for synthetic ObjectNav scenes remains limited because existing datasets are scarce, small in scope, or incomplete.Examples include single-room or limited-scene datasets and 3D-FRONT, which lacks populated kitchens and bathrooms and has object interpenetration issues.

3. HSSD-200: Habitat Synthetic Scenes Dataset

HSSD-200 is a high-quality synthetic 3D interior dataset with 211 houses, 18,656 objects, and 466 semantic categories. Its scenes largely recreate real homes, while professional assets provide realistic, consistently scaled objects and a processing pipeline for efficient simulation.

  • Dataset contribution: HSSD-200 contains 211 houses with 18,656 objects spanning 466 semantic categories, primarily recreating real homes designed in Floorplanner.Individual objects were created by professional 3D artists and generally match specific real-world brands or products.
  • Dataset construction: The dataset preparation pipeline extracts, decomposes, aligns, categorizes, deduplicates, and compresses scene assets into reusable architectural and object components.Object decomposition identified 1,791 models containing multiple semantically distinct, detachable objects.
  • Dataset construction: Unlike heuristic size estimates, HSSD-200 objects are modeled with real-world dimensions and consistently scaled.
  • Simulation efficiency: Asset compression reduced on-disk size by 12.4x while producing a comparable reduction in GPU memory consumption.The pipeline used mesh simplification, texture downsampling, and Basis supercompression for textures.
  • Scene realism: HSSD’s per-scene navigable-area distribution more closely matches real-world scans than ProcTHOR’s distribution.The comparison includes HM3DSem, Gibson, and MP3D scans, with MP3D spanning a broader range because it includes larger public and commercial spaces.

4. Dataset analysis

HSSD is a relatively small synthetic dataset that offers broad, complex scenes with over 18K unique objects and stronger alignment with real-world environments than prior synthetic datasets. Its visual and semantic realism is reflected in closer image statistics and object co-occurrence patterns to real scans than ProcTHOR.

  • Visual realism: HSSD is closer to HM3D, Gibson, and MP3D real images than ProcTHOR according to FID and KID comparisons.Images were rendered from ProcTHOR and HSSD using Habitat and compared with real image datasets.
  • Scale and complexity: HSSD has over 18K unique objects and 252.2 m2 average navigable area per scene, exceeding prior synthetic datasets in scene complexity and scale.Its object categories and instances per scene are also closer to HM3DSem, Gibson, and MP3D real-world scenes.
  • Semantic realism: HSSD’s object co-occurrences correlate more strongly with HM3DSem than ProcTHOR, with ρHSSD-HM3D = 0.219 versus ρTHOR-HM3D = 0.083.The comparison uses pairwise co-occurrence statistics over 28 common object categories, where higher correlation is better.
  • Semantic realism: For MP3D, HSSD also achieves higher co-occurrence correlation than ProcTHOR: ρHSSD-MP3D = 0.103 versus ρTHOR-MP3D = 0.017.These statistics quantify how closely object placement relationships match real-world scan datasets.

5. Experimental setup

The experiments study how dataset scale and realism affect transfer in ObjectNav, using a six-category task and a recurrent agent that combines visual, goal, action, GPS, and compass inputs. Success requires issuing STOP within 0.1 m of a goal viewpoint.

  • Task: The transfer experiments evaluate ObjectNav agents across six goal categories: bed, chair, sofa, tv, plant, and toilet.Episodes are generated across available train and validation scenes for each dataset.
  • Agent architecture: A frozen CLIP ResNet-50 encoder produces 2048-dimensional visual features, which are combined with learned goal and action embeddings and sensor outputs in a two-layer LSTM.The LSTM predicts the next action from the concatenated inputs.
  • Task: Success requires predicting STOP within 0.1 m of a viewpoint around the target object.The LoCoBot agent has a 0.18 m base radius, 0.88 m height, RGB, compass, and GPS sensors.
  • Agent architecture: The agent receives 224 × 224 RGB frames, the goal category, previous action, GPS, and compass observations at each step.The action space includes STOP, MOVE_FORWARD, TURN_LEFT, TURN_RIGHT, LOOK_UP, and LOOK_DOWN, with 0.25 m forward steps and 30° turns.

6. Results

Agents trained on HSSD generalize better than scale-matched ProcTHOR agents to HM3DSem and MP3D, while increasing ProcTHOR scale to 10K scenes provides no significant generalization improvement. Fine-tuning reduces differences between agents initialized from different datasets, but the study is limited to monolithic pixels-to-actions agents trained with reinforcement learning.

  • Zero-shot generalization: Zero-shot evaluation compares agents trained on iTHOR, ProcTHOR-10K, HSSD, HM3DSem, and MP3D across every dataset’s validation set, with performance averaged over three training runs.Agents generally perform best on the dataset used for training, while shared object assets support transfer between iTHOR and ProcTHOR.
  • Fine-tuned agents: Fine-tuning on each target dataset reduces performance differences between agents initialized from different training datasets, bringing them to more similar target-dataset levels.Fine-tuning uses the target dataset’s training split before evaluation on its validation split.
  • Disentangling scene dataset scale and realism: HSSD-60 and HSSD-122 agents outperform scale-matched ProcTHOR-60 and ProcTHOR-122 agents on HM3DSem and MP3D in both success rate and SPL.The difference is particularly pronounced in the 122-scene setting; ProcTHOR-10K does not significantly improve zero-shot generalization and generalizes less well than the HSSD variants.
  • Limitations: The study examines only monolithic pixels-to-actions agents trained end-to-end with reinforcement learning, excluding modular and imitation-learning approaches.The authors identify broader comparisons across these agent families as a needed extension.

7. Conclusion · A. Dataset details · A.1. HSSD dataset construction details

HSSD examines how synthetic-scene scale and realism affect ObjectGoal navigation generalization, finding that fewer, higher-quality scenes generalize better. Its construction extracts, cleans, annotates, decomposes, and filters assets from 211 human-authored scenes.

  • 7. Conclusion: Fewer, higher-quality synthetic 3D scenes produce better ObjectGoal navigation generalization than larger, lower-quality datasets.The conclusion frames HSSD as a systematic comparison of scale and realism across synthetic scene datasets.
  • A. Dataset details: The dataset represents real interiors with diverse real-world object models and supports systematic evaluation of navigation-agent generalization to realistic 3D scenes.This conclusion follows from the dataset’s human-authored scenes and the reported generalization analysis.
  • A.1. HSSD dataset construction details: HSSD begins with 211 Floorplanner scenes and extracts over 18K unique furniture and object assets while preserving each scene’s architectural layout.The assets and architecture are subsequently compressed as described in the main paper.
  • A. Dataset details: The extracted assets form a cleaned, semantically annotated database with WordNet, typical room, and attachment-side labels.Categories initially supplied by Floorplanner are refined and corrected through a user interface.
  • A.1. HSSD dataset construction details: Assets containing multiple objects are segmented automatically with connected-component analysis and then manually painted and labeled by users.Clean geometry enables clean segmentations through this combined automatic and manual process.
  • A.1. HSSD dataset construction details: Scene cleaning removes floater objects, interior doors for ObjectNav, and animate objects, while retaining exterior doors to prevent agents from leaving scenes.These filters are applied at the scene level for ObjectNav experiments.

A.2. HSSD dataset statistics … A.5. Why not use 3D-FRONT?

HSSD combines diverse object categories, realistic physical sizes, and varied geometry and appearance to support realistic embodied-AI environments. The appendix also describes Habitat-compatible asset conversion and explains why 3D-FRONT is less suitable because its scenes are sparsely populated and omit important object types and placements.

  • A.2. HSSD dataset statistics: HSSD objects have realistic physical sizes, with narrow distributions for some categories and broad distributions for pictures, shelves, and beds.Sizes are measured by bounding-box diagonal length in meters across 28 shared object categories.
  • A.3. HSSD qualitative visualizations: HSSD contains diverse object geometry, appearance, and physical sizes, producing more realistic environments and supporting embodied-agent perception generalization.Example object instances and first-person and top-down views illustrate this diversity.
  • A.4. AI2-THOR datasets in Habitat: ProcTHOR and related AI2-THOR assets were ported to glTF and metadata formats for efficient use in Habitat experiments.The conversion records semantic labels, positions, and orientations, while structural assets and procedural architectures are exported for Habitat.
  • A.4. AI2-THOR datasets in Habitat: The HSSD construction pipeline decomposes 211 scenes into more than 18K object models and architectural geometry, then annotates and aligns the extracted objects.Multi-object assets undergo segmentation, manual grouping, decomposition, deduplication, and orientation alignment.
  • A.2. HSSD dataset statistics: HSSD provides broad category coverage, including many unique instances in diverse real-world categories such as wall art and rugs.The category taxonomy extends WordNet with additional common object categories, and the word cloud summarizes unique instance counts by category.
  • A.2. HSSD dataset statistics: HSSD uses an augmented WordNetCO taxonomy that adds common-object synsets and organizes categories such as lamps into hierarchical labels.The lamp hierarchy includes introduced synsets, instance counts, and representative examples.
  • A.5. Why not use 3D-FRONT?: 3D-FRONT scenes are sparsely populated because their original assets are replaced with 3D-FUTURE models under limited release rights.This replacement restricts object presence to some room categories and leaves kitchens, bathrooms, and closets empty.
  • A.5. Why not use 3D-FRONT?: 3D-FRONT also has few small-object placements on larger furniture and no wall decorations, limiting scene object diversity.The passage distinguishes these omissions from undecomposed multi-object combinations and identifies them as another consequence of the replaced assets.

B. Episode generation details

Episodes are generated by sampling and filtering viewpoints near indoor goal objects, while restricting starts and goals to indoor regions. Scene-size differences produce many more easy, short-distance episodes in ProcTHOR than in HSSD and HM3D, whose distributions are more similar.

  • Viewpoint sampling: Viewpoints are sampled around each indoor goal object and rejected when unnavigable, too far from the object, or outside the house.Valid viewpoints are snapped to nearby navigable positions before further filtering.
  • Indoor episode constraints: Episodes require both the goal object and starting position to be indoors, with exterior doors closed and ambiguous indoor–outdoor scenes excluded.Training episodes are generated for 122 of the 125 training scenes under these restrictions.
  • Episode distance distributions: ProcTHOR generates substantially more easy, low-geodesic-distance episodes than HSSD and HM3D, while HSSD and HM3D have similar train and validation distributions.ProcTHOR’s distribution shows exponential decay as distance increases, whereas HSSD and HM3D reflect larger, more similarly sized scenes.

C. Analysis & experiment details

The experiments disentangle scene-dataset scale from realism using scale-matched ProcTHOR-122 and reduced 60-scene variants, while hierarchical clustering derives object co-occurrence structure from dissimilarity matrices.

  • Hierarchical clustering algorithm details: Hierarchical clustering converts the object co-occurrence matrix into dissimilarities, computes pairwise distances, and uses SciPy’s farthest-point linkage implementation.The resulting linkage matrix supports clustering based on object co-occurrence structure.
  • ProcTHOR-122: ProcTHOR-122 matches HSSD’s training scale using 122 ProcTHOR-10K scenes selected to approximate HSSD’s navigable-area distribution.The matched distributions are shown in Figure 19.
  • HSSD-60 and ProcTHOR-60: HSSD-60 and ProcTHOR-60 are created by randomly sampling 60 scenes from the corresponding 122-scene datasets to measure scale effects.These variants isolate the impact of reducing scene-dataset scale while preserving the original dataset identities.

D. Training plots and finetuning results

Training curves show validation performance saturating by approximately 200M experience steps. In finetuning, HSSD- and ProcTHOR-pretrained agents converge comparably, with HSSD retaining a small combined success and SPL lead.

  • Training plots: The reported zero-shot results use each training run’s checkpoint with the highest validation-set SPL.Validation performance is measured on the same dataset’s validation set during training.
  • Training plots: All agents reach validation-set convergence by approximately 200M steps of experience.The plots cover agents trained on iTHOR, ProcTHOR, HSSD, HM3DSem, and MP3D, with results from three independent training runs.
  • Finetuning results: HSSD-pretrained agents perform better throughout zero-shot evaluation than ProcTHOR- and iTHOR-pretrained agents on HM3DSem and MP3D.The comparison tracks zero-shot success and SPL across training steps using three independent runs.
  • Finetuning results: After finetuning on HM3DSem, HSSD-pretrained agents retain a small lead over ProcTHOR-pretrained agents in combined success and efficiency (SPL), despite comparable convergence.Table 6 summarizes performance on HM3DSem validation scenes after finetuning.

E. Agent failure case analysis

Failure analysis of 100 unsuccessful HM3DSem validation episodes shows that exploration inefficiency is the dominant failure mode, often involving repetitive looping. Incorrect goal predictions and inter-floor navigation are additional prominent causes.

  • Failure case analysis: Exploration failure is the largest category at 33%, with agents failing to reach goal objects because they inadequately explore the scene.Excessive looping, including repeatedly revisiting the same region, is a common cause.
  • Failure case analysis: Incorrect prediction accounts for 19% of failures, while inter-floor navigation accounts for 14%.Incorrect prediction involves stopping at a non-goal object, whereas inter-floor failures occur when the goal is absent from the agent’s starting floor.
Loading 2306.11290v3…