PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation

Xu Yan, Zhihao Yuan, Yuhao Du, Yinghong Liao, Yao Guo, Zhen Li, Shuguang Cui

arXiv:2112.11691v3cs.CV

TL;DR

Current 3D understanding work remains deficient due to the lack of data. CLEVR3D uses 3D scene graphs to generate attribute- and relationship-based questions, and modifies graph structure to create simulated common-sense-independent scenes and questions. TransVQA3D achieves superior VQA-3D results against pure language models and current state-of-the-art, and CSI-VQA-3D further improves real-world VQA performance.

  • Problem

    Current 3D understanding work remains deficient due to the lack of data.

    Source: §I · p. 1

  • Method

    CLEVR3D uses 3D scene graphs to generate attribute- and relationship-based questions, and modifies graph structure to create simulated common-sense-independent scenes and questions.

    Source: §I · p. 1, A. Data Generation · p. 3

  • Results

    TransVQA3D achieves superior VQA-3D results against pure language models and current state-of-the-art, and CSI-VQA-3D further improves real-world VQA performance.

    Source: §I · p. 2

  • Takeaways & Limitations

    CLEVR3D improves scene graph analysis across its reported metrics, especially for relationship triplets.

    Source: C. Scene Graph Analysis · p. 9

Abstract

from arXiv · show

Visual Question Answering on 3D Point Cloud (VQA-3D) is an emerging yet challenging field that aims at answering various types of textual questions given an entire point cloud scene. To tackle this problem, we propose the CLEVR3D, a large-scale VQA-3D dataset consisting of 171K questions from 8,771 3D scenes. Specifically, we develop a question engine leveraging 3D scene graph structures to generate diverse reasoning questions, covering the questions of objects' attributes (i.e., size, color, and material) and their spatial relationships. Through such a manner, we initially generated 44K questions from 1,333 real-world scenes. Moreover, a more challenging setup is proposed to remove the confounding bias and adjust the context from a common-sense layout. Such a setup requires the network to achieve comprehensive visual understanding when the 3D scene is different from the general co-occurrence context (e.g., chairs always exist with tables). To this end, we further introduce the compositional scene manipulation strategy and generate 127K questions from 7,438 augmented 3D scenes, which can improve VQA-3D models for real-world comprehension. Built upon the proposed dataset, we baseline several VQA-3D models, where experimental results verify that the CLEVR3D can significantly boost other 3D scene understanding tasks. Our code and dataset will be made publicly available at https://github.com/yanx27/CLEVR3D.

I. INTRODUCTION

The paper introduces CLEVR3D for varied question answering over 3D scenes, addressing limited data and the challenge of reasoning beyond common-sense object co-occurrences. It also proposes TransVQA3D and reports strong VQA-3D results, with CSI-VQA-3D improving real-world VQA performance.

I. INTRODUCTION

  • CLEVR3D generates questions from semantic scene graphs to cover object attributes and relationships, including existence, counting, comparison, object, and attribute queries.The real-scene portion contains over 44K questions covering 13 question types.Source: §I · p. 1
  • Compositional scene manipulation modifies scene-graph structure to create simulated scenes and questions that test reasoning beyond common-sense co-occurrence patterns.The authors describe this setup as addressing spurious associations, such as expecting tables to occur with sofas.Source: §I · pp. 1–2
  • The authors position CLEVR3D against ScanQA by emphasizing scene-graph generation, broader attributes and question types, greater scale, and simulated common-sense-independent scenes.The comparison states that CLEVR3D has 4× the questions from 11× the scenes, and ScanQA focuses only on real-world scenes.Source: §I · pp. 1–2
  • TransVQA3D achieves superior VQA-3D results against pure language models and current state-of-the-art, while CSI-VQA-3D further improves real-world VQA performance.The framework jointly learns VQA-3D and scene graph generation, using explicit relationship supervision to model scene context.Source: §I · p. 2
  • CLEVR3D provides 171K questions, including over 44K on real-world scenes and 127K from 7,438 simulated scenes generated to support common-sense-independent VQA-3D.Source: §I · pp. 1–2

II. RELATED WORK

Related work spans image-based VQA and scene graph analysis, while prior 3D scene graph methods largely focus on object classes and relationships. The paper also situates CLEVR3D among image-VQA datasets and structured scene representations.

A. Visual Question Answering on Images

  • CLEVR is presented as a diagnostic image-VQA dataset using synthetic images to construct spatial and comparative relationships between chromatic shapes.CLEVR3D draws inspiration from CLEVR but uses 3D scene graphs to generate more diverse questions.Source: A. Visual Question Answering on Images · p. 2

B. Scene Graph Analysis

  • Scene graphs represent objects as nodes and their relationships as edges, providing an explicit structured representation of object relations.The passage describes 2D scene graph generation pipelines and notes early 3D scene graph analysis based on 3RScan.Source: B. Scene Graph Analysis · p. 2
  • Prior 3D scene graph methods address object classes and relationships but largely ignore object existence and fine-grained objects.The cited 3D scene graph work includes benchmarks and models for predicting object classes and inter-object relationships.Source: B. Scene Graph Analysis · pp. 2–3

C. 3D Vision and Language

Prior 3D language tasks include grounding and dense captioning, while ScanQA derives questions from ScanRefer descriptions and may lack sufficient attribute variety. CLEVR3D uses real and simulated scene-graph data for VQA-3D.

C. 3D Vision and Language

  • Earlier 3D vision-language work includes object grounding and dense captioning, whereas ScanQA transforms ScanRefer descriptions into VQA-3D question-answer pairs.The authors state that this approach cannot generate sufficient data with variant object attributes, while CLEVR3D is generated from 3D scene graphs.Source: C. 3D Vision and Language · p. 3

III. CLEVR3D

  • CLEVR3D has REAL and SIM subsets for real-scene VQA-3D and common-sense-independent VQA-3D, with generation based on selected scene graphs, question templates, and bias reduction.The dataset section introduces question-engine generation and comparisons with CLEVR and ScanQA.Source: §III · p. 3

A. Data Generation

CLEVR3D questions are generated from annotated 3D scene graphs, with compositional scene manipulation creating simulated scenes for common-sense-independent VQA-3D. The dataset contains 171,174 questions across 8,771 scenes, split between real and simulated subsets.

A. Data Generation

  • The question engine uses 3DSSG objects, attributes, and relations to generate diverse questions from 90 question families, each averaging four parameters.The parameters represent color, material, shape, relationship, and object; ill-posed or degenerate questions are rejected, and answers appearing fewer than 100 times are eliminated.Source: A. Data Generation · pp. 3–4

B. Data Statistics

  • CLEVR3D-REAL is generated from 1,333 annotated scenes, while the dataset statistics describe question types and distributions of relationships, colors, materials, sizes, shapes, and objects.The question-type categories are defined by the outermost function in each question program.Source: A. Data Generation · p. 3, B. Data Statistics · pp. 4–5, Figure 4 · p. 5

A. Data Generation

  • Compositional scene manipulation addresses common-sense co-occurrence bias by replacing scene-graph objects from an object pool to form CLEVR3D-SIM scenes.Objects retain normalized orientation and are selected so their bounding-box volume ratio to the original is 1–1.5; ten simulated scenes are generated per real scene, then unreasonable occlusion or overlap is manually discarded.Source: A. Data Generation · p. 4

B. Data Statistics

  • 171,174 questions from 8,771 scenes form CLEVR3D, divided into CLEVR3D-REAL and CLEVR3D-SIM train and test sets.CLEVR3D-REAL has 38,806 training and 5,765 test questions; CLEVR3D-SIM has 109,351 training and 17,252 test questions.Source: B. Data Statistics · p. 4
  • CLEVR3D covers a wider range of attribute, object-class, and interrelationship categories than CLEVR.Table I reports category counts for attributes, objects, and relationships.Source: B. Data Statistics · p. 5, Table I · p. 5
  • Table II presents a statistical comparison of CLEVR3D with ScanQA.The supplied passage identifies the comparison but does not provide its values.Source: B. Data Statistics · p. 5, Table II · p. 5

IV. METHOD

TransVQA3D combines object embedding, a cross-modal Transformer, and scene graph aware attention (SGAA). Its inputs are a point-cloud scene and a natural-language question, with object and positional features embedded for processing.

IV. METHOD

  • TransVQA3D has three modules: object embedding, a cross-modal Transformer, and scene graph aware attention (SGAA).The architecture first fuses language and object features, then initializes a scene graph and uses additional edges for SGAA.Source: §IV · p. 5, Figure 6 · p. 6

A. Input Embedding

  • The model input is a 3D point-cloud scene with N points and a natural-language question containing l words.Source: A. Input Embedding · p. 5
  • Object embeddings are formed from instance point-cloud features and positional encodings.The scene's class-agnostic point-to-instance indicator groups points into object instances, whose features are extracted with a shared-weight feature extractor and symmetric pooling.Source: A. Input Embedding · p. 6
  • Object positional encoding uses each object's center, size, and heading orientation, enabling view-dependent relationship reasoning for objects with orientation annotations.Source: A. Input Embedding · p. 6

B. Cross-modal Transformer

A cross-modal Transformer fuses object and language embeddings before SGAA processes their enhanced features.

B. Cross-modal Transformer

  • The model concatenates object and language embeddings into a sequence of m + l tokens and applies three four-head self-attention layers to enhance both feature sets.Source: B. Cross-modal Transformer · p. 6

C. Scene Graph Aware Attention

SGAA jointly supports answer prediction and scene-graph construction by updating graph nodes and edges and attending across language and graph features.

C. Scene Graph Aware Attention

  • SGAA aims to obtain the answer while creating a scene graph whose nodes represent instances and whose edges represent their structural relationships.Source: C. Scene Graph Aware Attention · p. 6
  • Scene graph initialization uses enhanced object features as nodes and creates one-to-one edges containing both endpoint features, object centers, and relative offsets.Source: C. Scene Graph Aware Attention · p. 6
  • Twinning attention updates nodes from bidirectional edge relationships and updates edges from newly generated node features.Node updates aggregate edge features across both the row and column associated with each node.Source: C. Scene Graph Aware Attention · p. 7
  • Cross-modal attention weights and aggregates node features for each language feature.The language and node features are concatenated to produce normalized weights, which are used for weighted aggregation and pooling.Source: C. Scene Graph Aware Attention · p. 7

D. Prediction and Objectives

TransVQA3D predicts answers and separately classifies scene-graph nodes and edges, training the three outputs with a combined loss.

D. Prediction and Objectives

  • The model predicts candidate answers and classifies object classes at nodes and object-to-object relationships at edges.Answer prediction and node classification use cross-entropy, while the multi-label edge branch uses binary cross-entropy; the final loss linearly combines all three.Source: D. Prediction and Objectives · p. 7

V. EXPERIMENTS

The experiments benchmark VQA-3D on CLEVR3D and compare language, image-based, and 3D methods. TransVQA3D performs best in nearly all real-scene question categories, and simulated-data pretraining improves real-world VQA-3D.

V. EXPERIMENTS

  • The experiments benchmark VQA-3D and analyze scene-graph understanding and network architecture.Accuracy is reported across question types on CLEVR3D-REAL and CLEVR3D-SIM.Source: §V · p. 7, B. VQA-3D · p. 7, Table IV · p. 8

A. Implementation Details

  • Implementation uses BERT language features, PointNet++ object features, and three four-head cross-modal Transformer layers.Objects contain 4,000 randomly sampled points; training runs for 20 epochs with Adam and batch size 32.Source: A. Implementation Details · p. 7

B. VQA-3D

  • CLEVR3D-REAL training and test sets contain 49,650 and 10,455 questions from 996 and 133 scenes, respectively.All methods are trained on the training set and evaluated on the test set.Source: B. VQA-3D · p. 7
  • TransVQA3D achieves the best result in almost all CLEVR3D-REAL question categories, and CLEVR3D-SIM pretraining effectively boosts real-world VQA-3D.Human responses still outperform the model, especially on count and query-object questions; CLEVR3D-SIM results are higher than CLEVR3D-REAL because SIM has much more data.Source: Table III · p. 8, B. VQA-3D · pp. 8–9
  • The baselines span language-only, BEV-image, and 3D approaches, including L+LSTM, BEV+MCAN, ReferIt3D, and ScanQA.BEV images use a bird’s-eye view to represent the scene in one shot, omitting ceilings and walls to reduce occlusion.Source: Figure 7 · p. 7, B. VQA-3D · p. 8

C. Scene Graph Analysis

TransVQA3D is evaluated on 3DSSG scene-graph analysis, where it improves scene-graph metrics, with especially strong gains on relationship triplets.

C. Scene Graph Analysis

  • TransVQA3D improves all reported scene-graph analysis metrics on 3DSSG, especially relationship-triplet recall.Evaluation covers 160 object classes and 26 predicate classes, including node classification, predicate classification, and relationship triplets.Source: C. Scene Graph Analysis · p. 9

D. Ablation Study

The ablations examine joint VQA-3D and scene-graph learning, SGAA components, and cross-modal Transformer design. The conclusion presents CLEVR3D and its simulated variant as resources for 3D visual question answering and scene understanding.

D. Ablation Study

  • Adding object classification raises VQA-3D Overall Accuracy from 39.7 to 42.7, while edge prediction with SGAA adds 2% absolute accuracy.Scene-graph-only training degrades node and edge prediction by about 5%; removing twinning attention lowers both VQA-3D and scene-graph performance.Source: D. Ablation Study · p. 10, Table VI · p. 10
  • Removing positional encoding lowers VQA-3D Overall Accuracy by 3% and ClassAvg accuracy by 6%, while changing CMT layer count has little effect.Cross-modal attention affects VQA-3D by about 2% while scene-graph results remain stable.Source: D. Ablation Study · p. 10

VI. CONCLUSION

  • CLEVR3D uses scene-graph questions about object attributes and relationships, and compositional scene manipulation produces the simulated common-sense-independent CLEVR3D-SIM dataset.The conclusion states that TransVQA3D performs strongly on the dataset and that CLEVR3D can significantly help 3D scene understanding.Source: §VI · p. 10

Related papers

Loading 2112.11691v3…