PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI

Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, Xihui Liu, Cewu Lu, Dahua Lin, Jiangmiao Pang

arXiv:2312.16170v1cs.CVcs.AIcs.RO

TL;DR

Embodied agents need holistic 3D understanding from first-person observations, but prior research and datasets focus mainly on global-view inputs or lack comprehensive ego-centric annotations. EmbodiedScan introduces a large multimodal dataset and benchmark with Embodied Perceptron, whose experiments validate fundamental and language-grounded 3D perception across varying inputs and in the wild.

  • Problem

    Prior studies mainly examine global-view reconstructed scenes, while existing ego-centric RGB-D datasets are too small or lack comprehensive annotations for holistic 3D perception.

  • Method

    EmbodiedScan repurposes indoor RGB-D datasets into a multimodal ego-centric benchmark, and Embodied Perceptron uses shared encoders and multimodal fusion to process arbitrary numbers of views and text.

  • Results

    Experiments validate Embodied Perceptron across fundamental 3D perception and language-grounded benchmarks, including generalization in the wild.

  • Takeaways & Limitations

    EmbodiedScan provides a data and baseline foundation for holistic, language-grounded 3D perception from ego-centric views while exposing challenges in this setup.

Abstract

from arXiv · show

In the realm of computer vision and robotics, embodied agents are expected to explore their environment and carry out human instructions. This necessitates the ability to fully understand 3D scenes given their first-person observations and contextualize them into language for interaction. However, traditional research focuses more on scene-level input and output setups from a global view. To address the gap, we introduce EmbodiedScan, a multi-modal, ego-centric 3D perception dataset and benchmark for holistic 3D scene understanding. It encompasses over 5k scans encapsulating 1M ego-centric RGB-D views, 1M language prompts, 160k 3D-oriented boxes spanning over 760 categories, some of which partially align with LVIS, and dense semantic occupancy with 80 common categories. Building upon this database, we introduce a baseline framework named Embodied Perceptron. It is capable of processing an arbitrary number of multi-modal inputs and demonstrates remarkable 3D perception capabilities, both within the two series of benchmarks we set up, i.e., fundamental 3D perception tasks and language-grounded tasks, and in the wild. Codes, datasets, and benchmarks will be available at https://github.com/OpenRobotLab/EmbodiedScan.

1. Introduction

Embodied agents need holistic 3D scene understanding from ego-centric observations, but prior work largely studies global-view inputs and lacks sufficiently large, comprehensive ego-centric datasets. EmbodiedScan addresses this gap with a multimodal dataset, benchmark, and baseline framework for language-grounded 3D perception.

  • Embodied agents require scene-level understanding covering object semantics, scene geometry, and language grounding from ego-centric observations.
  • Prior studies mainly use globally reconstructed point clouds or meshes, while existing ego-centric RGB-D datasets are too small or incompletely annotated.
  • More than 5k scans, nearly 1M ego-centric RGB-D images, 160k oriented 3D boxes across 760+ categories, 80-category occupancy labels, and 1M language descriptions compose EmbodiedScan.
  • Embodied Perceptron accepts RGB-D sequences and text, fuses multimodal features, and decodes occupancy, 3D boxes, and language-grounded representations.
  • Two benchmark series evaluate fundamental 3D perception and language-grounded scene understanding, with experiments validating the baseline and its in-the-wild generalization.

2. Related Work

Existing indoor 3D datasets and benchmarks provide useful foundations but remain limited in scene and object diversity, annotation richness, realism, or ego-centric coverage. EmbodiedScan targets these gaps with larger multimodal data and broader continuous and language-grounded tasks.

  • Indoor scene datasets generally offer less scene and object diversity than autonomous-driving datasets, whereas EmbodiedScan provides richer multimodal annotations.
  • EmbodiedScan offers more than 3× images and categories than the compared indoor datasets, while extending evaluation to continuous perception and visual grounding.
  • Synthetic or imperfect-mesh embodied datasets can transfer poorly to real-world scenarios, whereas EmbodiedScan uses real-scanned RGB-D images.
  • Prior indoor work is limited in object orientations, semantic categories, and input formats, motivating holistic ego-centric perception of semantics, geometry, and poses.
  • EmbodiedScan and Embodied Perceptron provide a data and baseline foundation for language-grounded 3D scene understanding alongside detection and occupancy prediction.

3. Dataset

EmbodiedScan integrates compatible indoor RGB-D datasets into a unified ego-centric multi-view resource and adds oriented boxes, semantic occupancy, and language annotations. Its processing supports varied view sampling, open-vocabulary labels, and large-scale spatial prompts.

  • Data Collection & Processing: The dataset integrates high-quality RGB-D captures with camera poses from ScanNet, 3RScan, and Matterport3D, while reserving ARKitScenes for future inclusion.
  • Data Collection & Processing: Data formats are unified into a general multi-view representation while preserving sequential continuity for ScanNet and 3RScan during inference.
  • Annotation: Three annotation types—3D bounding boxes, semantic occupancy, and language descriptions—enrich object, geometric, and relational scene understanding.
  • Annotation: SAM-assisted annotation supports oriented 3D boxes and addresses missing orientation and small-object labels in existing annotations.
  • Statistics: Open-vocabulary categories are clustered with Sentence-BERT, matched to WordNet, and manually revised or merged for a consolidated vocabulary.
  • Statistics: 801711 training and 168322 validation language prompts cover spatial relations including proximity, support, allocentric, and between, nearly 10× the original SR3D datasets.

4. Embodied Perceptron

Embodied Perceptron is a unified multimodal encoder-decoder framework that accepts variable numbers of RGB-D views and text. It fuses image, point-cloud, and language features into sparse and dense representations for detection and occupancy prediction, with multimodal features supporting grounding.

  • Multi-Modal 3D Encoder: The framework uses separate ResNet/FPN, Minkowski ResNet34, and BERT encoders for images, point clouds, and text before task-specific fusion and decoding.
  • Multi-Modal 3D Encoder: It accepts any number of RGB-D views by transforming depth-derived point clouds into a global coordinate system and querying corresponding 2D features.
  • Benchmarks: The benchmark suite evaluates continuous multi-view detection, occupancy prediction, and monocular detection on EmbodiedScan.
  • Sparse & Dense Decoder: Dense fusion constructs feature volumes for occupancy, while isomorphic sparse fusion supplies multilevel features for 3D object prediction.
  • Sparse & Dense Decoder: The sparse decoder predicts oriented boxes using classification, regression, centerness, rotation representations, and corner-based Chamfer-distance objectives.
  • Sparse & Dense Decoder: The dense decoder aggregates multilevel features with a 3D FPN to produce multiscale occupancy predictions supervised across resolutions.

5. Benchmark

EmbodiedScan benchmarks continuous, multi-view, and language-grounded ego-centric 3D perception across detection, occupancy prediction, and visual grounding. Results show modality, representation, data realism, and annotation complexity materially affect performance.

  • Benchmark design: The benchmark covers scene-based continuous perception, view-based ego-centric tasks, and prompt-based 3D visual grounding.Continuous benchmarks use sequential views, while visual grounding uses constructed language prompts.
  • Metrics: 3D detection and visual grounding use IoU-based AP at 0.25 and 0.5, while occupancy prediction uses mIoU.Average recall is also reported for reference.
  • Fundamental 3D perception benchmarks: RGB-D inputs outperform single modalities for continuous detection, while depth-only detection remains close to RGB-D, indicating depth’s strong role in 3D perception.For semantic occupancy, the RGB-D advantage is more pronounced because fine-grained semantics expose depth-only weaknesses.
  • Language-grounded benchmark: The Embodied Perceptron baseline outperforms reimplemented ScanRefer, BUTD-DETR, and L3Det on multi-view 3D visual grounding.Performance remains lower than previous works because the benchmark changes input formats and annotations and includes more categories and small objects.
  • Analysis: Oriented-box prediction substantially lowers detection performance, showing that object orientation estimation is harder than conventional axis-aligned detection.Replacing reconstructed point clouds with aggregated multi-view depth has minor AP25 effects but heavily impacts AP50.
  • Analysis: Training with EmbodiedScan improves AP by 2.74% on ScanNet and 5.93% on the overall validation split.Head-category gains are 4.01% AP and 7.55% AP, respectively.

6. Conclusion

The paper presents EmbodiedScan as a multi-modal suite for language-grounded holistic 3D scene understanding from ego-centric views. It combines rich annotations, a unified baseline, and benchmarks spanning basic and language-grounded perception while exposing new challenges.

  • Conclusion: EmbodiedScan combines diverse sensor data with oriented 3D boxes, semantic occupancy, and language descriptions for ego-centric 3D perception.The suite targets holistic, language-grounded scene understanding.
  • Conclusion: The baseline accepts any number of views through a unified multimodal encoder and task-specific decoders.It supports both basic and language-grounded 3D perception benchmarks.
  • Conclusion: Experiments highlight the suite’s value while revealing new challenges in ego-centric, language-grounded 3D perception.The paper frames the dataset and annotations as opportunities for embodied 3D perception and related fields.

A. Implementation Details

The supplementary implementation details specify input settings, modality encoders, fusion and decoding components, training procedures, and adaptations for baseline models and source datasets.

  • Input and feature processing: Training and inference use task-specific numbers of input images, including 20/50 for multi-view detection and grounding, 10 for continuous detection, and 10/20 for occupancy.The settings reflect differing memory requirements across tasks.
  • Input and feature processing: Depth maps are converted to point clouds, sampled to at most 100k points, voxelized, and processed with sparse convolutional networks.The voxel size is 0.01 meters for 3D detection.
  • Input and feature processing: Embodied Perceptron uses shared ResNet50, MinkResNet34, and RoBERTa-Base encoders for images, depth-derived point clouds, and text.Sparse fusion produces multi-level channels, while dense fusion uses FPN-enhanced 2D features to derive a 3D feature volume.
  • Decoders and objectives: The dense decoder compresses 3D features and predicts multi-scale semantic occupancy through three shared 3D convolutional heads.The decoder produces outputs at resolutions from 40 × 40 × 16 to 10 × 10 × 8.
  • Decoders and objectives: The visual grounding decoder repeatedly predicts 3D box locations while refining position encodings and fusing sparse 3D features with text features.Its contrastive losses pull object queries toward positive text features and target text features toward corresponding visual features.
  • Baseline adaptations: Baseline implementations reuse task-matched input settings and backbones, while adapting models for oriented 3D box prediction and dataset-specific experiments.The supplementary section describes adaptations for ImVoxelNet, VoteNet, OccNet, SurroundOcc, ScanRefer, and BUTD-DETR.

B.3. Statistics

The statistics analysis characterizes EmbodiedScan’s long-tailed instance distribution, spatial-language prompt patterns, annotation differences, and dataset-comparison clarifications.

  • Instance and prompt statistics: EmbodiedScan has a long-tailed instance distribution and more categories and instances than previous datasets.The complete distribution is shown in Figure 5.
  • Instance and prompt statistics: Horizontal Proximity and Allocentric prompts dominate, while Vertical Proximity and Support prompts form smaller portions.The analysis relates this pattern to objects being primarily distributed on the XY plane.
  • Spatial relationships: Tables are frequently associated with support objects such as books and lamps and nearby objects such as windows, doors, couches, cabinets, curtains, bins, and chairs.These associations are reported as consistent with common indoor object configurations.
  • Spatial relationships: Books most often appear in Support prompts involving tables, stands, desks, cabinets, boxes, and dressers.The passage identifies support relations as the majority for this smaller-object category.
  • Spatial relationships: Mirrors frequently appear above sinks, stools, cabinets, and sockets, while pictures often appear above beds, couches, tables, desks, toilets, and cabinets.These recurring pairs are presented as fixed spatial relationships in the generated prompts.
  • Annotation and comparison: The annotations highlight changed box orientations and newly annotated small objects, while expanded categories make language prompts more ambiguous.Figure 6 compares previous and current annotations, and the text notes efforts to maintain unique references.
  • Annotation and comparison: The test-set discussion reports that only scan and image statistics are included in the main paper, with more complete annotations considered for future benchmarks.Release plans depend on communication and collaboration with the official source-dataset hosts.

C. Supplementary Results

The supplementary results expand the reported benchmark evidence, ablations, and qualitative visualizations to further examine the dataset and baseline in real-world settings.

  • Supplementary evaluation: The supplement provides additional benchmark details, ablation studies, and qualitative prediction visualizations on the dataset and in the real world.These materials extend the main paper’s overall results and key ablations.

C.1. Detailed Benchmark Results

Category-level results show that some common objects remain challenging, while improved orientation decoding mainly benefits objects whose length and width differ substantially.

  • Pictures and bottles remain challenging despite having many annotations.
  • Orientation-aware decoding mainly improves categories with substantial length–width differences, including pictures, doors, windows, shelves, and towels.
  • The category-level analysis highlights orientation estimation as important for 3D detection in this setting.

C.2. Supplementary Ablation Studies

The supplementary studies examine view-count scaling, training-data scaling, decoder and loss designs, and category-level detection results. Performance improves with more scans, while IoU-based loss is identified as more faithful to the final metric for general 9-DoF boxes.

  • Number of Views: The baseline supports any number of views during training and inference, with view-count effects evaluated using continuous 3D detection.
  • The supplementary material reports continuous, multi-view, and monocular 3D detection results alongside dense-fusion, sparse-decoder, and training-data ablations.
  • IoU-based loss is more faithful to the final metric and merits further study for general 9-DoF boxes.
  • Performance with Different Training Data: 1.5k–3k–5k scans produce performance improvements that appear approximately linear, especially for heavily annotated head categories.

C.3. Qualitative Results

Qualitative results cover continuous detection, occupancy prediction, monocular detection, and visual grounding. Exploration can preserve earlier predictions while correcting some errors, and the model performs decently in unselected real-world test cases despite sensor and environment differences.

  • Qualitative visualizations cover continuous 3D detection, occupancy prediction, monocular 3D detection, and multi-view 3D visual grounding.
  • Continuous perception preserves most previous predictions and corrects some of them as exploration proceeds.
  • The trained model shows decent performance without cherry-picking in different environments and with a different RGB-D sensor.
  • The project demo video provides additional visualizations of the dataset, annotation tool, and perception results.
Loading 2312.16170v1…