PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models

Huy Ha, Shuran Song

arXiv:2207.11514v2cs.CVcs.RO

TL;DR

Robots need to recognize long-tail categories in open-world 3D scenes, while limited 3D data and fine-tuning can undermine the breadth or robustness of learned models. SemAbs projects 2D VLM relevancy maps into 3D for a semantic-abstracted module whose output represents occupancy for the text label. SemAbs significantly outperforms the SemAware baseline across the evaluated open-world settings and transfers from simulation to real-world scans.

  • Problem

    Robots need to recognize long-tail categories in open-world 3D scenes, while limited 3D data and fine-tuning can undermine the breadth or robustness of learned models.

    Source: §1 · p. 1

  • Method

    SemAbs projects 2D VLM relevancy maps into 3D for a semantic-abstracted module whose output represents occupancy for the text label.

    Source: §3.1 · p. 3

  • Results

    SemAbs significantly outperforms the SemAware baseline across the evaluated open-world settings and transfers from simulation to real-world scans.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    The authors conclude that SemAbs generalizes to novel vocabulary, visual properties, semantic classes, and domains, supporting its use toward open-world formulations of robotic tasks such as object retrieval.

    Source: §5 · p. 8

    The semantic-agnostic geometry assumption does not hold for completing detailed object geometry, which is often semantic-dependent.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in a semantic-agnostic manner. We demonstrate the usefulness of SemAbs on two open-world 3D scene understanding tasks: 1) completing partially observed objects and 2) localizing hidden objects from language descriptions. Experiments show that SemAbs can generalize to novel vocabulary, materials/lighting, classes, and domains (i.e., real-world scans) from training on limited 3D synthetic data. Code and data is available at https://semantic-abstraction.cs.columbia.edu/

1 Introduction

Robots need to reason about long-tail objects in open-world 3D environments, but available 3D datasets are limited in scale and diversity, while fine-tuning 2D VLMs can weaken their zero-shot robustness. SemAbs uses VLM relevancy maps as semantic-agnostic inputs to 3D reasoning, targeting open-world generalization from limited synthetic training data.

1 Introduction

  • Available 3D datasets [1] [3] [4] [5] [6] have less diversity and scale than internet-scale 2D data [7] [8] [9], limiting preparation for open-world 3D tasks.These tasks require generalization to novel classes, vocabulary, visual properties, and domains.Source: §1 · p. 1
  • Fine-tuning 2D VLMs can reduce their open-set robustness because small, less diverse datasets encourage task specialization.The paper contrasts these datasets with internet-scale image-caption data.Source: §1 · p. 1
  • SemAbs learns spatial and geometric reasoning in a semantic-agnostic manner, hypothesizing that these skills can generalize from limited synthetic 3D data.The proposed framework aims to retain the 2D VLM’s zero-shot robustness while adding 3D capabilities.Source: §1 · p. 2
  • SemAbs combines a relevancy-map abstraction with a semantic-aware wrapper and a semantic-abstracted 3D module that predicts occupancy for partially observed or hidden objects.The wrapper derives a map from an RGB-D image and category label; the 3D module uses it for object completion or language-conditioned localization.Source: §1 · p. 2
  • The framework targets two open-world tasks: completing partially observed objects and localizing hidden objects from language descriptions.Its 3D occupancy output represents either object geometry or possible locations of a hidden object.Source: §1 · p. 2

2 Related Works

Prior work brings pretrained 2D VLMs’ open-set robustness to downstream vision but does not directly provide the pixel-level information needed for spatial reasoning, and zero-shot transfer techniques remain confined to 2D. Existing 3D scene-understanding work is typically limited by the visual diversity and categories in its training data.

2 Related Works

  • Pretrained 2D VLMs offer zero-shot robustness, but their image-level representations do not directly support pixel-level spatial reasoning.Dense features [19] [20] [21] and relevancy maps [18] [22] address this mismatch and can interface VLMs with open-world 3D tasks.Source: §2 · p. 2
  • Fine-tuning VLMs on small task-specific datasets can weaken zero-shot robustness [7] [10] [12] [13] [19], motivating transfer methods that retain pretrained models.Such methods use prompting [29] or attention extraction [15] [19] [20] [21], but existing zero-shot transfer techniques have been confined to 2D vision.Source: §2 · p. 2
  • Prior 3D detection [31] [32], semantic scene completion [33] [34] [35], and language-informed localization [6] [36] typically address limited visual diversity and object categories.Existing 3D datasets [1] [3] [4] [5] [6] have less scale, diversity, and coverage than internet-scale image-text pairs [7] [8] [9].Source: §2 · p. 2
  • The paper’s key insight is to separate visual-semantic reasoning from 3D reasoning through an appropriate abstraction.The section’s Figure 2 describes open-world generalization requirements as cumulative tiers, each with a distinct property outlined in yellow.Source: §2 · p. 2, Figure 2 · p. 3

3 Method: Semantic Abstraction

SemAbs separates broad semantic recognition from 3D spatial reasoning: it uses VLM relevancy maps as semantic-agnostic inputs to 3D modules. The method applies this abstraction to object completion and hidden-object localization, while its multiscale extractor detects small objects efficiently.

3 Method: Semantic Abstraction

  • Semantic Abstraction is motivated by the greater diversity of semantic concepts than spatial and geometric concepts, suggesting limited 3D data may cover spatial reasoning.VOOL learns embeddings for spatial relations and uses target and reference occupancy features to produce localization occupancy.Source: §3 · p. 3, §3.4 · p. 5

3.1 Abstraction via Relevancy

  • SemAbs converts an RGB-D image and class label into a relevancy map, projects it into 3D, and completes the resulting point cloud into occupancy.Only the 3D encoder and decoder are trained; the 3D module receives relevancy abstractions rather than the text label, allowing outputs for labels recognized by the VLM.Source: §3.1 · p. 3

3.2 A Multi-Scale Relevancy Extractor

  • At 0.4 seconds per text label on average and over ×60 speedup, the multiscale extractor detects tiny long-tail objects often missed by augmentations alone.It averages relevancy maps across augmentations and scales; augmentations help reduce irrelevant regions but can still miss small objects.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3 Application to Open Vocabulary Semantic Scene Completion (OVSSC)

  • For OVSSC, SemAbs processes each open-vocabulary object label and completes the 3D geometry of its partially observed object.The task allows object labels to vary across scenes and between training and testing; occupancy probabilities are compared across classes, with low-confidence query points labeled empty.Source: §3.3 · pp. 4–5

3.4 Application to Visually Obscured Object Localization (VOOL)

  • VOOL predicts possible occupancies for a hidden object from an RGB-D image and a description specifying its target, visible reference object, and spatial relation.Its occupancy may comprise disjoint regions, representing uncertainty that bounding boxes do not encode.Source: §3.4 · p. 5

4 Experiments

Experiments test SemAbs across novel rooms, visual conditions, synonyms, classes, and domains, comparing it with baselines on open-world 3D tasks and NYUv2. SemAbs outperforms the SemAware baseline across open-world evaluations and transfers to real-world scans, while the paper also identifies limitations in its assumptions and VLM inputs.

4 Experiments

  • The benchmark tests novel rooms, visual conditions, synonyms, classes, and domains; baselines span learned semantic reasoning, VLM-delegated reasoning, and naive relevancy extraction.Performance is measured with voxel IoU at 32×32×32; “novel” means unseen during 3D training.Source: §4 · pp. 5–6

4.1 Open-world Evaluation Results

  • SemAbs significantly outperforms SemAware across the open-world evaluations; CLIPSpatial trails it by 8.0–9.0 IoU in every generalization scenario.The reported comparisons cover open-vocabulary scene completion and visually obscured object localization.Source: Table 1 · p. 7, Table 2 · p. 7, §4.1 · p. 7
  • Despite simulation training, SemAbs completes and localizes objects in real-world Matterport scans, and VOOL localizes a keychain with +0.13 IoU when OVSSC misses it.The examples span novel material descriptions, colors, conjunctions, and semantic classes.Source: Figure 5 · p. 6, §4.1 · p. 7

4.2 Zero-shot Evaluation Results

  • On NYUv2, SemAbs outputs all 894 object categories, including 108 absent from training, and its predictions can map back to the collapsed 11 classes.The evaluation also includes a SemAbs model trained from scratch on NYUv2.Source: §4.2 · pp. 7–8
  • SemAbs robustly detects small and long-tail objects, including puppy toys, iPods, and hand sculptures in the NYUv2 qualitative results.The paper notes that many natural-image objects occur too infrequently to build usable recognition models.Source: Figure 6 · p. 7, §4.2 · p. 8
  • Wall and ceiling IoU were only 33.9 and 22.6, respectively, which the authors associate with caption-data bias for classes seldom described in internet captions.The authors also state that fine-tuning 2D VLMs on small datasets hurts their robustness.Source: §4.2 · p. 8

4.3 Limitations and assumptions

  • The semantic-agnostic geometry assumption does not cover detailed object completion; the paper also flags simple localization syntax and relevancy-map confusion between related concepts.The authors identify richer natural-language spatial descriptions as an extension and note that other VLMs may address relevancy-map limitations.Source: §4.3 · p. 8

5 Conclusion

SemAbs adds 3D spatial capabilities to 2D VLMs for open-world scene understanding, with learned reasoning that generalizes across vocabulary, visual properties, classes, and domains.

5 Conclusion

  • SemAbs equips 2D VLMs with 3D spatial capabilities for open-world 3D scene understanding.The authors identify scene completion and 3D object localization as fundamental primitives for robotic tasks.Source: §5 · p. 8
  • Its learned spatial and geometric reasoning generalizes to novel vocabulary, visual properties, semantic classes, and domains.The conclusion describes inputs as RGB and semantic labels abstracted into relevancy maps.Source: §5 · p. 8
  • The authors believe scene completion and 3D object localization can help bridge close-world robotics tasks, such as object retrieval, to open-world settings.Source: §5 · p. 8

6 Appendix

The appendix details SemAbs’s relevancy-map design, network and data-generation choices, extraction speed, and additional qualitative and spatial-relation results, alongside limitations of relevancy-based approaches.

6.1 Relevancy as VLM’s confidence

  • Raw relevancy magnitudes retain class- and view-dependent strength and potential VLM uncertainty information that thresholding or clipping loses.The authors hypothesize that activation magnitudes provide information about VLM confidence; their experiments found raw maps worked better than thresholding or clipping.Source: §6.1 · p. 12, §6.5 · p. 14

6.2 Network

  • The network uses a 3D U-Net encoder, a two-layer MLP decoder, max-reduction scattering, and a bounded voxel grid.Input and output point clouds receive random translation, rotation, and scale transformations, with points outside the grid filtered.Source: §6.2 · p. 12

6.3 Relevancy Extractor details

  • 39.5±0.1 seconds for 100 labels versus 2420.8±14.1 seconds for direct Chefer et al. sliding-window extraction.The SemAbs configuration uses multi-scale relevancy, five random RGB augmentations, horizontal flipping, and crop sizes from h to h/4.Source: §6.3 · p. 12

6.4 Data Generation and Training Details

  • Training data comes from custom AI2-THOR scenes; VOOL uses six prepositions, with viewer-centric relations based on pairwise displacement and distance relative to object dimensions.“On top of” and “inside” use receptacle information, and ambiguous descriptions label all consistent points positive.Source: §6.4 · pp. 12–13
  • Novel-class VOOL evaluation holds out six classes, while synonym evaluation substitutes semantically similar words in model text inputs.The held-out classes include mug, pot, safe, wine bottle, teddy bear, and basketball; synonym examples include pillow/cushion and fridge/refrigerator.Source: §6.4 · pp. 13–14

6.5 Things which did not work

  • Thresholding relevancy maps is problematic because activation strengths vary, maps may highlight only object parts, and raw values may encode uncertainty.The authors also report that relevancy maps from CLIP models other than B/32 did not look promising with their extraction approach.Source: §6.5 · p. 14

6.6 More results

  • Qualitative OVSSC comparisons show baseline errors on small and novel objects; VOOL examples show failures on unknown classes, small references, and spatial relations.The confusion matrix’s largest non-diagonal pairs include door/door knob, floor/carpet, photo board/poster, and photo frame/poster.Source: §6.6 · pp. 14–15, Figure 10 · p. 15

6.7 VOOL Performance Breakdown by Spatial Relation

  • SemAbs consistently outperforms all other approaches across VOOL spatial relations.Table 4 reports visually obscured object localization broken down by spatial relation.Source: §6.7 · p. 15, Table 4 · p. 16
Loading 2207.11514v2…