PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Concept Learning and Reasoning from Multi-View Images

Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, Chuang Gan

arXiv:2303.11327v1cs.CVcs.AIcs.LGcs.RO

TL;DR

Single-view images provide limited coverage for questions about occluded objects and whole-scene 3D relations. The paper introduces 3DMV-VQA and 3D-CLR, which builds compact 3D representations from multi-view images, grounds concepts, and reasons over them. 3D-CLR outperforms baseline models, but small-object grounding and instance separation remain challenges.

  • Problem

    Single-view images cover only a limited region of a scene, making occluded-object counting and 3D relations such as closeness difficult to assess.

    Source: §1 · p. 2

  • Method

    3D-CLR combines a compact neural-field scene representation, pretrained 2D vision-language features for 3D concept grounding, and neural reasoning operators.

    Source: §1 · p. 2

  • Results

    3D-CLR outperforms all baseline models on the proposed 3DMV-VQA benchmark.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The paper presents 3DMV-VQA and 3D-CLR as a first step toward 3D reasoning from actively collected multi-view images.

    Source: §1 · p. 2

    Small-object grounding and separating close object instances remain challenging, and relation reasoning still needs improvement.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Humans are able to accurately reason in 3D by gathering multi-view observations of the surrounding world. Inspired by this insight, we introduce a new large-scale benchmark for 3D multi-view visual question answering (3DMV-VQA). This dataset is collected by an embodied agent actively moving and capturing RGB images in an environment using the Habitat simulator. In total, it consists of approximately 5k scenes, 600k images, paired with 50k questions. We evaluate various state-of-the-art models for visual reasoning on our benchmark and find that they all perform poorly. We suggest that a principled approach for 3D reasoning from multi-view images should be to infer a compact 3D representation of the world from the multi-view images, which is further grounded on open-vocabulary semantic concepts, and then to execute reasoning on these 3D representations. As the first step towards this approach, we propose a novel 3D concept learning and reasoning (3D-CLR) framework that seamlessly combines these components via neural fields, 2D pre-trained vision-language models, and neural reasoning operators. Experimental results suggest that our framework outperforms baseline models by a large margin, but the challenge remains largely unsolved. We further perform an in-depth analysis of the challenges and highlight potential future directions.

1. Introduction

The paper introduces 3D reasoning from actively gathered multi-view images and the 3DMV-VQA benchmark, then proposes 3D-CLR to represent scenes, ground concepts, and reason over them. Its experiments show 3D-CLR outperforming baseline models, while identifying remaining challenges.

1. Introduction

  • Single-view images cover only part of a scene, making occluded-object counting and 3D relations such as closeness difficult to judge.The paper frames compact 3D representations, semantic grounding, and step-by-step relation reasoning as challenges for multi-view reasoning.Source: §1 · p. 2
  • 3DMV-VQA evaluates concept, counting, relation, and comparison questions using multi-view observations gathered by an embodied agent exploring scenes.The benchmark contains approximately 5k scenes and 50k question-answering pairs, with scene-graph annotations for model diagnosis.Source: §1 · p. 2
  • 3D-CLR combines a compact neural-field representation, features from a pretrained 2D vision-language model, and neural reasoning operators.The operators take 3D object representations as input and output predictions.Source: §1 · p. 2
  • On 3DMV-VQA, 3D-CLR outperforms all baseline models by a lot, but failure cases and model diagnosis reveal remaining challenges in grounding small objects and separating close object instances.Source: §1 · p. 2
  • The authors analyze task challenges and identify potential future directions for 3D reasoning from multi-view images.They frame the work as a first step toward addressing the task’s challenges.Source: §1 · pp. 2–3

2. Related Work

Prior visual-reasoning benchmarks mainly use single-view 2D images, while prior 3D question-answering work uses point clouds. This paper instead studies reasoning from actively gathered multi-view images and uses 2D vision-language models for open-vocabulary concept grounding in 3D.

Visual Reasoning

  • Earlier visual-reasoning benchmarks mainly use single-view 2D images, while this work introduces reasoning from multi-view images.Prior neural methods model reasoning implicitly; neural-symbolic methods use operators over pre-segmented masks and are limited to predefined concepts on simple scenes.Source: §2 · p. 3

3D Reasoning

  • Prior 3D visual question-answering methods use point clouds, whereas this work builds a compact 3D representation by integrating actively gathered multi-view observations.The paper contrasts this with humans exploring an environment rather than receiving an entire 3D representation at once.Source: §2 · p. 3

Embodied Reasoning

  • For open-vocabulary 3D grounding, the paper leverages 2D vision-language models, addressing the difficulty of obtaining real-world 3D assets paired with language.Embodied question-answering work explores environments but focuses on navigation outcomes or histories from 2D images rather than holistic 3D understanding.Source: §2 · p. 3

3. Dataset Generation

The authors construct 3DMV-VQA from annotated HM3D indoor scenes, capturing multi-view images through simulated embodied exploration. They generate templated questions across scene scales and question types, with controls for question and answer distributions.

3.1. Multi-View Images

  • The HM3D source dataset includes 5k 3D scenes and approximately 600k rendered images; the 3DMV-VQA benchmark contains 4,100 scenes: 2,000 single-room, 2,000 multi-room, and 100 whole-house.A Habitat robot agent explores each benchmark scene, taking 12 pictures at each sampled navigable point while rotating to observe the 360° view.Source: §3.1 · pp. 3–4

3.2. Questions and Answers

  • Questions are machine-generated from templates using semantic annotations, bounding boxes, scene graphs, and 11 defined relationships.The four question types are concept, counting, relation, and comparison.Source: §3.2 · p. 4
  • The benchmark uses 41 manually defined templates and controls template, answer, and concept counts to balance question generation.Rejection sampling discards questions when their answers or concepts would make a template’s distribution unbalanced.Source: §3.2 · p. 4

4. Method

3D-CLR uses a voxel-based neural field to build compact scene representations from images, grounds open-vocabulary concepts by aligning 2D vision-language features with 3D features, and executes questions through neural reasoning operators.

4. Method

  • The framework combines 3D scene learning, 2D feature generation, concept grounding, and step-by-step reasoning to produce an answer.Its pipeline includes FILTER, GET INSTANCE, and COUNT RELATION operators.Source: §4 · pp. 4–5

4.1. Learning 3D Compact Scene Representations

  • DVGO stores scene density and color in voxel grids, renders views by sampling along rays, and learns the representation by minimizing L2 loss against observed images.The extracted density grid provides the compact 3D scene representation.Source: §4.1 · p. 5

4.2. 3D Semantic Concept Grounding

  • The method adds feature voxels and uses CLIP-LSeg pixel features with an alignment loss to transfer 2D semantic features into the 3D representation.Each pixel corresponds to multiple 3D points along its ray, which supports learning 3D features from multi-view images.Source: §4.2 · pp. 5–6
  • Dot-product attention between 3D point features and CLIP concept embeddings provides zero-shot, view-independent concept grounding in the 3D representation.The feature grid stores 512-dimensional features in each cell.Source: §4.2 · p. 6

4.3. Neural Reasoning Operators

  • A semantic parser maps questions to operators that execute reasoning on the grounded 3D representation.FILTER selects cells of a semantic concept, DBSCAN groups points into instances, and learned relation networks predict object relations.Source: §4.3 · p. 6

5. Experiments

On 3DMV-VQA, 3D-CLR outperforms the baselines, but results and model diagnosis expose persistent difficulties with grounding and 3D reasoning. The experiments evaluate accuracy across question types and analyze both baseline behavior and component-level failure cases.

5.1. Experimental Setup

  • The benchmark reports visual question-answering accuracy for four question types, using a 7:1:2 train/validation/test split.Baselines range from vanilla and attention-based neural networks to fine-tuned vision-language and graph-based methods.Source: §5.1 · p. 6, §5.2 · p. 6, §5.3 · p. 7

5.3. Experimental Results

  • Temporal attention in MAC(V) improves over average pooling, while ALPRO remains near the random baseline on counting and 3D-based baselines outperform 2D counterparts.The authors also report that NS-VQA underperforms 3D-CLR on counting and relation questions, which they associate with the need for holistic 3D scene understanding.Source: §5.3 · p. 7
  • 3D-CLR outperforms other baselines by a large margin, yet grounds only approximately 66% of semantic concepts.The authors describe 3DMV-VQA as challenging despite this comparative result.Source: §5.3 · p. 7

5.4. Discussions

  • Even with ground-truth semantic labels, DBSCAN can merge touching objects such as chairs, leaving close-instance counting unreliable; small objects also remain difficult to ground.The discussion identifies unsupervised instance segmentation of very close objects as a future direction.Source: §5.4 · p. 8
  • Relation reasoning remains weak even with ground-truth semantic and instance segmentations, with many failures involving “inside” and objects in cabinets.The authors suggest joint depth and segmentation predictions as a potential solution.Source: §5.4 · p. 8

6. Conclusion

The conclusion presents 3DMV-VQA as a benchmark for 3D reasoning from actively gathered multi-view images and introduces 3D-CLR, combining a neural field, a 2D vision-language model, and reasoning operators. It also reports an analysis of the benchmark’s challenges and potential future directions.

6. Conclusion

  • The paper introduces 3D reasoning from multi-view images as a task and collects 3DMV-VQA through embodied robots actively exploring indoor environments.The benchmark is presented as large-scale.Source: §6 · p. 8
  • 3D-CLR combines a neural field, a 2D vision-language model, and reasoning operators for the task.The conclusion states that the model’s effectiveness is illustrated.Source: §6 · pp. 8–9
  • The paper analyzes challenges in the dataset and points to potential future directions.The authors hope 3DMV-VQA will help push the frontiers of 3D reasoning.Source: §6 · p. 9

A.1. Dataset Statistics

The dataset statistics show that relation questions form the largest share, while question lengths span a wide range; the supplement also includes more dataset examples.

A.1. Dataset Statistics

  • Relation questions make up the largest portion of the dataset, reflecting its focus on 3D reasoning and spatial relations.Figure 5 presents the dataset statistics.Source: §A.1 · p. 13, Figure 5 · p. 13
  • The questions cover a wide range of word lengths.This distribution is shown in Figure 5(b).Source: §A.1 · p. 13

A.2. More Dataset Examples

  • Figure 6 provides more examples from the 3DMV-VQA dataset.Source: §A.2 · p. 13

B. Implementation details

The supplement details voxel-grid reasoning operators and baseline implementations, then examines transfer to Replica and unseen concepts, with counting remaining challenging.

B.1. Reasoning Operators

  • The reasoning operators manipulate voxel grids: FILTER selects by concept, GET INSTANCE uses DBSCAN to cluster instances, and RELATION scores instance tuples.Room instances are obtained by extracting walls and segmenting the scene; relation scores above 0.5 can be treated as true.Source: §B.1 · pp. 13, 15

B.2. Baselines

  • The listed baselines include image-based CNN-LSTM/MAC, 3D-Feature + LSTM, ALPRO, LGCN, and NS-VQA implementations.Their described inputs include image features, 3D features, or per-pixel semantic labels, depending on the baseline.Source: §B.2 · pp. 15–16

C.1.1 Generalization to Replica

  • Replica examples show zero-shot grounding of concepts such as “projector screen” and recognition of relations such as “between,” but counting remains poor.Counting fails when objects touch and are not spatially separated, preventing DBSCAN from distinguishing them.Source: §C.1.1 · pp. 16–17
  • On Replica, 3D-CLR maintains its HM3D performance across question types, with counting even better on simpler scenes.The authors attribute the counting difference probably to fewer tiny-object details in Replica, which makes instance segmentation easier.Source: §C.1.1 · p. 16, Table 2 · p. 17

C.1.2 Generalization to Unseen Concepts

  • For unseen concepts, 3D-CLR and NS-VQA retain good conceptual-question results, while MAC and MAC(V) perform much worse than on HM3D.The authors suggest the modular design and CLIP-LSeg support 3D-CLR's zero-shot generalization.Source: Table 3 · p. 17, §C.1.2 · p. 17

C.2. More Qualitative Examples on 3DMV-VQA

  • On 3DMV-VQA, qualitative examples show grounding of unseen concepts such as “Christmas tree” and successful counting when instances are well separated.Source: §C.2 · p. 17

A.1. Dataset Statistics

The supplement describes the dataset's question distribution and examples, then documents its voxel-grid reasoning operators and baseline implementations.

A.1. Dataset Statistics

  • Relation questions form the largest portion of the dataset, and its questions span a wide range of word lengths.Source: §A.1 · p. 20

A.2. More Dataset Examples

  • Figure 2 provides additional examples from the 3DMV-VQA dataset.Source: §A.2 · p. 20

B.1. Reasoning Operators

  • The reasoning operators use voxel grids to filter concepts, obtain object and room instances, and evaluate existence, counts, and relations.GET INSTANCE uses DBSCAN, while room instances are segmented using wall instances.Source: §B.1 · pp. 20, 22

B.2. Baselines

  • The baseline implementations include CNN-LSTM/MAC, 3D-Feature + LSTM, ALPRO, LGCN, and NS-VQA.They use image, 3D, or per-pixel semantic features, depending on the method.Source: §B.2 · pp. 22–23

C.1.1 Generalization to Replica

On Replica, 3D-CLR maintains performance across question types and shows zero-shot concept grounding, while qualitative examples still identify counting as difficult.

C.1.1 Generalization to Replica

  • Qualitative Replica examples show zero-shot grounding of unseen concepts such as “projector screen” and recognition of relations such as “between.”Source: §C.1.1 · p. 23
  • 3D-CLR maintains its HM3D performance on Replica across question types, with counting even better on Replica's simpler scenes.The authors suggest the simpler scenes contain fewer tiny-object details, making instance segmentation easier.Source: §C.1.1 · p. 23, Table 1 · p. 24

C.1.2 Generalization to Unseen Concepts

On unseen concepts, 3D-CLR and NS-VQA retain good performance on conceptual questions, while MAC and MAC(V) perform very poorly. Qualitative examples also show 3D-CLR handling an unseen Christmas tree and counting well-separated instances.

C.1.2 Generalization to Unseen Concepts

  • 3D-CLR and NS-VQA retain good results on conceptual questions with unseen concepts, while MAC and MAC(V) perform very poorly.The table reports question-answering accuracy by question type when generalizing to unseen categories.Source: §C.1.2 · p. 24
  • The authors suggest 3D-CLR’s modular design and incorporation of CLIP-LSeg equip it with zero-shot generalization ability.Source: §C.1.2 · p. 24

C.2. More Qualitative Examples on 3DMV-VQA

  • Qualitative examples show 3D-CLR generalizing to an unseen Christmas tree and counting well when instances are well apart.Source: §C.2 · p. 24
Loading 2303.11327v1…