Source-linked AI summary
Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, Dhruv Batra
TL;DR
Embodied agents need benchmarks and methods suited to active perception in realistic environments rather than static internet-vision settings. The paper extends EmbodiedQA to Matterport3D, evaluates RGB and point-cloud navigation policies, and finds strong naive baselines, useful Inflection Weighting, and richer obstacle-avoidance signals from point clouds.
Problem
Internet-vision datasets provide limited evidence for training embodied agents that actively perceive, act, and navigate novel environments.
Method
The paper extends EmbodiedQA to photorealistic Matterport3D environments and develops end-to-end-trainable navigation models using point clouds, RGB images, or both.
Results
Forward-only and random are strong navigators in the default EmbodiedQA evaluation setting, while Inflection Weighting improves behavior cloning for recurrent navigation models.
Takeaways & Limitations
Point clouds provide a richer signal than RGB images for learning obstacle avoidance, supporting continued study of 3D perception in embodied navigation.
Takeaways & Limitations
The question set excludes preposition questions because they are relatively few and have strongly biased answer distributions.
Abstract
from arXiv · showhide
To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D). We thoroughly study navigation policies that utilize 3D point clouds, RGB images, or their combination. Our analysis of these models reveals several key findings. We find that two seemingly naive navigation baselines, forward-only and random, are strong navigators and challenging to outperform, due to the specific choice of the evaluation setting presented by [1]. We find a novel loss-weighting scheme we call Inflection Weighting to be important when training recurrent models for navigation with behavior cloning and are able to out perform the baselines with this technique. We find that point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating the use (and continued study) of 3D deep learning models for embodied navigation.
1. Introduction
The paper adapts EmbodiedQA to photorealistic Matterport3D environments and develops end-to-end navigation models using point clouds, RGB images, or both. Its analysis identifies strong naive baselines, Inflection Weighting as useful for behavior cloning, and point clouds as richer for obstacle avoidance.
- Motivation: Internet-vision training on static, human-annotated web datasets is poorly suited to embodied agents that actively perceive and change their observations through actions.Embodied agents require visual perception, language understanding, and navigation in potentially novel environments.
- Task and contributions: The task extends EmbodiedQA to photorealistic Matterport3D environments, requiring agents to navigate from egocentric vision without an environment map and answer questions about novel surroundings.Agents must locate the queried entity and produce the correct answer after navigating from a random starting location.
- Task and contributions: The MP3D-EQA dataset contains 1136 questions and answers grounded in 83 environments, with increased visual variation that makes navigation more challenging.Questions are generated from Matterport3D annotations, with object colors collected from Amazon Mechanical Turk.
- Analysis: The study evaluates 16 navigation models, 3 visual question answering models, and 2 perception models while varying perception, memory, and goal specification.The evaluation compares multiple architectures, language variations, and perception variations.
- Findings: Forward-only and random navigation are strong baselines in the default EmbodiedQA evaluation setting and are challenging to outperform.The authors caution that models can perform surprisingly well without learning meaningful behavior.
- Findings: Point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating continued study of 3D perception for embodied navigation.The work presents end-to-end-trainable 3D perception for goal-driven navigation in photorealistic environments.
- Findings: Inflection Weighting balances cross-entropy contributions between action transitions and continuations, improving behavior cloning with shortest-path experts.The scheme emphasizes time steps where the ground-truth action differs from the previous action.
2. Related Work
Prior embodied-perception work largely relies on synthetic environments or 2D RGB perception. This paper situates EmbodiedQA in real scanned indoor environments and empirically evaluates point-cloud representations for navigation.
- Embodied agents and environments: Embodied perception lacks a standardized, large-scale, diverse real-world benchmark comparable to COCO for embodied vision.Existing efforts include synthetic environments and scans of real indoor houses with varied input modalities.
- Embodied agents and environments: Prior EmbodiedQA work predicts actions from egocentric RGB frames, while this paper asks whether 3D navigation must be restricted to a single RGB camera.The authors extend the task to real environments sourced from Matterport3D.
- 3D representations: The paper empirically shows that point-cloud representations are more effective than RGB images for navigation in this task.This comparison motivates examining 3D representations directly for embodied navigation.
- 3D representations: Volumetric 3D representations are limited by data sparsity and the computational cost of 3D convolutions, motivating architectures that operate directly on point clouds.The related work positions point clouds as a distinct input for 3D deep learning.
3. Questions in Environments
The paper instantiates EmbodiedQA in Matterport3D homes and constructs MP3D-EQA questions from annotated objects, rooms, relationships, and crowdsourced colors. The dataset tests navigation and perception in new homes under constrained single-floor episodes.
- Environments: Matterport3D provides 90 home environments reconstructed from panoramic RGB-D images, with aligned point clouds, meshes, semantic labels, and dense object and room annotations.The dataset covers 40 object categories, approximately 50,000 instances, and more than 2050 rooms.
- Environments: The simulator supports occupancy checking, mesh-based RGB rendering, and shortest-path calculation, while point-cloud rendering is constructed from environment panoramas.The authors reimplement shortest paths for higher accuracy and speed.
- Questions: MP3D-EQA generates templated location, color, and color-room questions from Matterport3D annotations.The original preposition question type is excluded because it is relatively rare and has strong answer biases.
- Questions: Object colors are crowdsourced because Matterport3D lacks human color annotations; workers select each object’s dominant color from a 24-color palette.The palette extends Kenneth Kelly’s 22 colors with off-white and slate-grey.
- Questions: The dataset filters low-entropy answer distributions and removes ambiguous or structural entities so questions require navigation and perception to answer accurately.Examples include excluding nearly deterministic room answers and ambiguous “misc” rooms.
- Dataset statistics: Approximately 1100 questions are generated across 83 home environments, with color-room questions making up the majority.The dataset contains approximately 13 question-answer pairs per environment.
- Dataset statistics: Testing uses entirely new homes, potentially introducing unseen objects, while agent starts and targets are restricted to the same floor and episodes span a single floor.This split tests semantic and perceptual generalization under a single-floor setting.
4. Perception for Embodied Agents
The paper develops perception and navigation components for EmbodiedQA using point clouds, RGB images, or both. Point-cloud representations are built with PointNet++ and pretrained on semantic, color, and structural reconstruction tasks before being used by navigation and question-answering models.
- Embodied Agent Models: EmbodiedQA agents predict actions from past trajectory information, the current state, and the question, then answer questions after navigating to relevant entities.The models consider decisions about action selection, question representation, trajectory encoding, and observation representation.
- Point-Cloud Perception: Point clouds represent unordered 3D points with associated colors, and PointNet++ maps them to fixed-dimensional observation features.PointNet++ alternates spatial clustering and feature summarization, using a three-level architecture with 1045, 256, and 64 points.
- Point-Cloud Perception: The point-cloud encoder is pretrained through semantic segmentation, color autoencoding, and structure autoencoding to capture objects, colors, spatial information, free space, and depth.Approximately 100,000 Matterport3D frames are used; the decoder networks are discarded afterward and the encoder remains a fixed feature extractor.
- Question Answering Models: Question-answering models include question-only baselines, question-guided attention over recent frames, and spatial attention over the last frame using RGB inputs.Questions are encoded separately for navigation and question answering with two-layer LSTMs having 128-dimensional hidden states.
- Navigation Models: Navigation models vary perception among None, point cloud, RGB, and point-cloud-plus-RGB inputs, and vary language conditioning between no question and the question.The action space is forward, turn-left, turn-right, and stop; reactive models use current observation and previous action, while memory models incorporate a GRU.
- Training Navigation Policies: Behavior cloning trains navigation policies to imitate shortest-path trajectories, but validation accuracy can be misleading when models repeat previous actions and miss action transitions.Inflection Weighting emphasizes time steps where the ground-truth action differs from the previous action, and the paper reports significant gains for recurrent models.
5. Experiments and Analysis
The experiments compare navigation models across memory, perception, goal specification, and training-loss choices using geodesic distance, collision rate, IoUT, and QA accuracy. Memory and inflection weighting improve navigation, while point clouds and RGB offer complementary strengths for obstacle avoidance and target viewing.
- Evaluation protocol: Navigation is evaluated on novel environments at 10, 30, or 50 primitive actions from the target, corresponding to 0.35, 1.89, or 3.54 meters.Agents are initially randomly spawned and then walked along the shortest path until the specified distance is reached.
- Memory and baselines: Models with memory significantly outperform memory-less counterparts, while random and forward-only remain surprisingly strong baselines.The evaluation procedure often hands agents control while facing the correct direction, favoring forward motion.
- Inflection weighting: Inflection-weighted cross-entropy significantly outperforms unweighted cross-entropy and baselines, especially for memory models on dT and IoUT.Inflection weighting balances loss contributions between action transitions and repeated actions during behavior cloning.
- Perception: Vision improves IoUT and QA and usually lowers collision rates, but generally worsens distance-to-target navigation metrics.For reactive models, fewer collisions remove the forward-only baseline’s functional-stop behavior; the effect is less clear for memory models.
- Goal specification: Adding the question does not substantially improve memory navigation models trained with behavior cloning.The authors suspect behavior cloning limits exploration and error correction, reducing the models’ ability to use question information.
- Perception: Point clouds provide a richer obstacle-avoidance signal, RGB provides richer semantic information, and combining them improves both aspects.The study evaluates collision rate for obstacle avoidance and IoUT plus QA for semantic and view-related performance.
6. Conclusion
The paper extends EmbodiedQA to photorealistic Matterport3D environments and studies navigation using different inputs and architectures. It identifies strong simple baselines, Inflection Weighting, and point clouds as important findings for embodied navigation.
- The work extends EmbodiedQA to photorealistic Matterport3D environments and introduces the MP3D-EQA v1 dataset.
- The study evaluates two navigation baselines, two architectures, and eight input variations, while developing end-to-end navigation from 3D point clouds.
- Inflection Weighting improves the effectiveness of behavior cloning, while random and forward-only remain strong under the evaluation setting.
- The work aims to bridge internet vision-style problems and embodied perception through analysis of navigation factors in realistic environments.
7. Color Label Collection Interface
Figure 8 presents the interface used to collect dominant-color annotations from Amazon Mechanical Turk workers.
- Figure 8 shows the annotation interface.
- The interface was used to collect dominant-color annotations.
- Annotations were collected from workers on Amazon Mechanical Turk.
8. Point Cloud Rendering
Point-cloud rendering first selects points inside the agent's camera view and then removes occluded or free-space points using mesh intersections. Rasterization and sparse preprocessing reduce the cost of these visibility checks.
- The renderer constructs a global point cloud from Matterport3D panoramas before selecting points visible from the agent's current view.
- Visibility begins by testing whether each point lies inside the camera view frustum using intrinsic and extrinsic camera matrices.
- Mesh intersections remove points that are occluded or lie in free space, using an error tolerance of ϵ = 0.25 cm.
- Rasterizing the mesh and using a depth buffer approximates ray-tracing visibility checks to improve rendering speed.
- Because global Matterport3D point clouds contain hundreds of millions of points, rendering first checks a significantly sparser cloud before rechecking dense regions.
9. Perception Models
The perception models use point-cloud and RGB encoders with task-specific decoder heads and pretraining objectives. Their design includes multi-scale point-cloud processing, feature propagation, RGB decoding, and jointly weighted losses.
- Point-cloud models: The point-cloud encoder is specified with PointNet++ notation and uses set abstraction modules to transform hierarchical point-cloud representations.
- Point-cloud models: Multi-scale grouping computes features at different radii and concatenates the resulting descriptors, analogous to multi-scale convolution.
- Point-cloud models: Feature propagation finds three nearest neighbors, applies inverse-distance weighted interpolation, and optionally concatenates skip features before an MLP.
- Point-cloud models: The model includes semantic segmentation, color prediction, and structure prediction heads for point-cloud pretraining.
- RGB models: The RGB encoder uses the initial convolution and four ResNet50 residual blocks, with semantic, depth, and color decoders built from 1x1 convolutions and bilinear interpolation.
- Pretraining: Joint pretraining does not sacrifice depth or autoencoding performance, according to the comparison in Table 2.
10. Question answering model training
Question-answering models use varied training configurations, with checkpoint selection based on held-out validation performance.
- LSTM question-only models are trained with a batch size of 40.
- Attention+* models are trained with a batch size of 20.
- Spatial+RGB+Q models are trained with a batch size of 32.
- All models select their best checkpoint using performance on a held-out validation set.
11. Navigation model training
Navigation models are trained and evaluated with consistent checkpoint procedures, including separate handling of inflection-weighted and vanilla cross-entropy losses.
- Models trained with and without inflection-weighted loss use the same training and checkpoint-selection procedure.
- Inflection-weighted cross-entropy is a significantly better predictor of end-to-end performance than vanilla cross-entropy.
- Checkpoint selection applies nonmaximum suppression to teacher-forcing validation loss, evaluates the top five checkpoints end-to-end, and chooses the highest QA + d∆at T−50.
12. Results
The results compare navigation models, baselines, and perception modalities across evaluation settings. They emphasize strong simple baselines, behavioral differences from questions, and the value of point-cloud perception for obstacle avoidance.
- The reported navigation results include both inflection-weighted and non-inflection-weighted training conditions with 90% bootstrap confidence intervals.
- Question usage significantly changes the behavior of vision-based navigators, while its average performance impact is also significant.
- Reactive navigators form a strong correlation cluster, whereas memory models are less correlated.
- The evaluation includes models with RGB, point-cloud, and combined RGB-plus-point-cloud perception.The tables identify RGB models as using ResNet50 and point-cloud models as using PointNet++.