Source-linked AI summary
Embodied Question Answering
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra
TL;DR
EmbodiedQA addresses question answering when an agent must first explore a partially observed 3D environment rather than receive a fixed visual view. The paper develops an end-to-end hierarchical agent, environments, and evaluation protocols, using imitation learning followed by reinforcement-learning fine-tuning. The experiments report that the ACT navigator performs best on navigation distance, while its RL-finetuned version achieves the highest answering accuracy despite often overshooting the target.
Problem
EmbodiedQA asks how an agent can answer questions about an environment when it must actively navigate from a random location using only egocentric vision.
Method
The paper develops an end-to-end hierarchical agent with vision, language, navigation, and answering modules, initializing with imitation learning and fine-tuning with reinforcement learning.
Results
ACT+Q achieves the smallest final distance to the target, while ACT+Q-RL achieves the highest answering accuracy and often explores beyond the target.
Takeaways & Limitations
EmbodiedQA provides a benchmark and agent framework for studying language-grounded active perception, goal-driven navigation, and visual question answering together.
Abstract
from arXiv · showhide
We present a new AI task -- Embodied Question Answering (EmbodiedQA) -- where an agent is spawned at a random location in a 3D environment and asked a question ("What color is the car?"). In order to answer, the agent must first intelligently navigate to explore the environment, gather information through first-person (egocentric) vision, and then answer the question ("orange"). This challenging task requires a range of AI skills -- active perception, language understanding, goal-driven navigation, commonsense reasoning, and grounding of language into actions. In this work, we develop the environments, end-to-end-trained reinforcement learning agents, and evaluation protocols for EmbodiedQA.
1. Introduction
EmbodiedQA introduces agents that must navigate rich 3D environments from egocentric vision to gather information before answering questions. The task jointly exercises language understanding, visual reasoning, active perception, navigation, grounding, and commonsense reasoning.
- EmbodiedQA task: The task jointly requires language understanding, visual reasoning, and navigation rather than passive visual question answering.Success depends on interpreting the question, identifying relevant entities, and moving through the environment to obtain useful observations.
- EmbodiedQA task: EmbodiedQA spawns an agent randomly in a 3D environment and requires it to navigate, gather visual information, and answer a question.The agent receives only first-person visual input while pursuing the information needed for question answering.
- Core challenges: Because the answer may not be visible from the starting location, the agent must actively control which pixels it perceives through movement.This active-perception requirement links visual input, physical constraints, world understanding, and question-specific actions.
- Core challenges: Without a floor plan or map, the agent must navigate from egocentric views and use common sense about locations such as garages and cars.The task also grounds language into action sequences, such as navigating toward a house exterior when seeking a car.
- Approach: The proposed system trains vision, language, navigation, and answering end-to-end from raw pixels and words, without preprocessing or handcoded environmental knowledge.The introduction also describes a planner-controller navigator whose controller executes primitive actions for a variable number of steps.
2. Related Work
EmbodiedQA combines vision, language, and action in a partially observable setting where agents actively control their visual trajectory. Related work provides separate slices of these capabilities, whereas EmbodiedQA targets goal-driven question answering in richer multi-room environments.
- Positioning: EmbodiedQA occupies a novel configuration: single-shot question answering over videos captured by goal-driven active agents.The related-work framing organizes prior research by vision, language, and action, then positions EmbodiedQA across all three axes.
- Vision and language: Unlike VQA, EmbodiedQA agents control their trajectories instead of answering from fixed visual views.This flexibility lets agents seek informative viewpoints, although most random paths are unhelpful and the task is harder than VQA.
- Vision and action: Visual-navigation research maps visual observations to actions, while EmbodiedQA adds question-driven information gathering and answering.Classical navigation separates mapping and planning; newer deep-RL systems can instead map egocentric observations directly to actions.
- Language and action: Situated-language systems vary from declarative programs to free-form instructions, but EmbodiedQA requires extracting spatial information from first-person visual sensing.The agent lacks a floor plan, object labels, attributes, and other global structured representations.
3. EQA Dataset: Questions In Environments
EQA v1 constructs grounded questions over realistic indoor environments, using functional programs and filtering to reduce ambiguity and exploitable answer bias. The resulting benchmark spans question types requiring navigation, perception, spatial reasoning, memory, and language grounding.
- EQA uses House3D scenes built from realistic SUNCG room and furniture layouts, with publicly released environments, dataset, and code.
- Question Types: Questions are programmatically generated from templates over queryable rooms and objects, with answers ranging from names and colors to yes/no values and counts.
- Questions as Functional Programs: Functional programs retain unique objects to generate unambiguous questions, while inapplicable or ambiguous queries can be excluded.
- Question Types: The nine question types test object detection, scene recognition, counting, spatial reasoning, color recognition, and logical operators, often requiring multiple capabilities.
- The benchmark is framed as a curriculum of capabilities rather than a static dataset, and shortest paths to queried objects provide expert demonstrations for imitation learning.
- Question-Answer Generation: Answer distributions are filtered by normalized entropy and minimum environment frequency to discourage memorization and unreliable estimates.
- EQA v1 Statistics: EQA v1 contains over 5000 questions across over 750 environments, 45 unique objects, and 7 unique room types, with no environment overlap across splits.
4. A Hierarchical Model for EmbodiedQA
The proposed EmbodiedQA agent integrates vision, language, navigation, and answering from egocentric RGB input and questions without global environment or task representations. Its ACT navigator separates high-level action selection from variable-duration execution, followed by staged imitation learning and policy-gradient fine-tuning.
- The agent contains vision, language, navigation, and answering modules trained from raw sensory input without global maps, object or room representations, or functional programs.
- Vision: The vision encoder processes egocentric RGB images and is pretrained to reconstruct RGB values, semantic classes, and depth through multiple prediction heads.
- Language: Separate LSTM question encoders support navigation and answering because each module may need different parts of the question.
- Navigation: ACT divides navigation between an LSTM planner that selects primitive actions and a controller that repeats them for a variable number of steps.
- Navigation: This planner-controller division separates navigational intent from primitive action sequences and gives the planner variable time steps between decisions.
- Question Answering: The answering module attends to the last five observed frames, combines visual attention with question encoding, and predicts over 172 possible answers.
- Training: Training first uses imitation or supervised learning for navigation and answering, then jointly fine-tunes the full architecture with policy gradients.
5. Experiments and Results
Experiments evaluate EmbodiedQA agents on navigation and question answering, separating intermediate navigation from downstream answer accuracy. ACT-based navigators perform best overall, while reinforcement learning improves answering but can cause exploratory overshooting.
- Evaluation: The evaluation reports question-answering mean rank and navigation metrics including terminal distance, distance change, minimum distance, room entry, and stopping rate.Distances are measured along shortest paths to target objects, while mean rank orders the ground-truth answer among 172 possible answers.
- Navigation results: All baseline navigators have negative d∆, ending farther from the target than their starting positions.This result supports the difficulty of EmbodiedQA navigation.
- Navigation results: Memory helps: LSTM and LSTM-Q baselines end closer to targets than their corresponding reactive models, achieving smaller dT.The comparison holds because all models start equally far from the target.
- Navigation and answering results: ACT+Q achieves the smallest terminal distance dT, while ACT+Q-RL achieves the highest answering accuracy.The two ACT variants therefore lead on different evaluation dimensions.
- Navigation and answering results: ACT+Q-RL reaches targets most closely during trajectories and enters target rooms most often, but its lower stopping rate causes frequent overshooting.Because the answering module attends to frames along the trajectory, overshooting does not hurt question-answering accuracy; a small per-action penalty could correct it.
- Navigation and answering results: Several methods outperform ShortestPath+VQA in answering accuracy because shortest paths may not provide the best viewpoint for visual question answering.The authors suggest ray tracing as a possible way to frame the target object better at termination.
6. Conclusion
The paper introduces EmbodiedQA as a task requiring agents to navigate through 3D environments before answering questions from egocentric visual observations. It combines a hierarchical neural agent, imitation-learning initialization, reinforcement-learning fine-tuning, evaluation protocols, and human demonstrations.
- Task: EmbodiedQA spawns an agent at a random 3D location, asks a question, and requires navigation, first-person information gathering, and answer prediction.The paper evaluates the task in the House3D virtual environment.
- Method: The neural architecture decomposes navigation into a planner selecting actions or directions and a controller executing primitive actions for variable durations.This hierarchy returns control to the planner after controller execution.
- Training and evaluation: The agent is initialized with imitation learning and fine-tuned with reinforcement learning for question answering.The paper also develops evaluation protocols and collects remote human demonstrations through Amazon Mechanical Turk.
Appendix Overview
The supplementary document details question generation, visual perception, answering, and human-path-conditioned question answering, and provides example agent behavior.
- Supplementary contents: Section A explains the question-answer generation engine, including functional programs and safeguards against ambiguity, bias, and redundancy.These checks support benchmark construction.
- Supplementary contents: Section B describes CNN vision models, their architectures and training details, and quantitative and qualitative results.The models serve as the visual module for EmbodiedQA.
- Supplementary contents: Section C describes the agent’s answering module.
- Supplementary contents: Section D reports machine question-answering performance conditioned on human navigation paths collected through Amazon Mechanical Turk.
- Supplementary contents: A linked video shows example navigation and answer predictions by the agent.
A. Question-Answer Generation Engine
The question-generation engine represents EQA questions as executable functional programs over room and object annotations. It applies entity selection, uniqueness and blacklist filtering, template generation, spatial relations, distance comparisons, and dataset-balance checks.
- Functional programs: Each question is generated by composing elementary operations that execute on environment annotations and yield an answer.The operations include select, unique, query, relate, and distance.
- Functional programs: The engine filters entities for uniqueness, removes template-specific blacklisted objects, and generates question strings from the remaining entities.Uniqueness retains entities occurring exactly once, while blacklists exclude objects considered unsuitable for particular templates.
- Elementary operations: The engine supports spatial relations and distance comparisons by returning related object pairs or closer/farther object triplets.Distance questions compare two objects relative to an anchor object.
- Question categories: Question types impose different ambiguity constraints: object uniqueness for location and color, room and object uniqueness for relational questions, and room uniqueness without object uniqueness for existence and counting.The corresponding functional forms add or omit unique(object) according to the question type.
- Checks and balances: Dataset checks remove low-entropy or infrequent questions, cap counting answers above 5, and enforce a 2-metre distance difference for comparison questions.These safeguards target answer-distribution imbalance, human burden, and ambiguity.
- Checks and balances: The generator collapses visually similar or hierarchically related object labels and excludes obscure room names and unsuitable objects through manually defined filters.Examples include merging teapot and coffee_kettle into kettle and excluding tiny or vague objects.
B. CNN Training Details
The visual system uses a multi-task CNN to encode egocentric RGB images while jointly reconstructing RGB, predicting semantic classes, and estimating depth. It is trained end-to-end with a weighted combination of task losses and evaluated against independently trained task-specific models.
- Architecture: The encoder transforms House3D egocentric RGB images into fixed-size representations for three decoding heads: RGB reconstruction, semantic segmentation, and depth prediction.Semantic class and depth targets are supplied by the renderer.
- Architecture: Each decoder upsamples the shared encoder output to the original image resolution, with output channels determined by the corresponding prediction task.The semantic, depth, and autoencoder branches use 191, 1, and 3 output channels, respectively.
- Training: The hybrid network combines cross-entropy segmentation loss with Smooth-ℓ1 depth and reconstruction losses in a single weighted objective.The total loss is seg_loss + 10^depth_loss + 10^reconstruction_loss.
- Training: Training uses Adam with learning rate 10^-3 for 5 epochs on 100k renderer RGB images, using batches of 20.
- Evaluation: Validation compares the jointly trained hybrid CNN with separately trained single-task networks across segmentation, depth, and autoencoder metrics.Segmentation uses pixel accuracy, mean pixel accuracy, and mean IOU; depth and reconstruction use Smooth-ℓ1.
- Evaluation: Qualitative validation results show RGB reconstructions alongside ground-truth and predicted depth and segmentation maps.
C. Question Answering Module
The question answering module combines the question representation with attended visual evidence from the agent’s recent navigation. It predicts an answer distribution by selecting the most relevant frames among the last five observations.
- Encoding: The module encodes the question with an LSTM and each of the last five navigation frames with a CNN.
- Attention: Dot-product attention selects relevant frames before their image features are combined with the question encoding.
- Prediction: The combined representation produces a softmax distribution over possible answers conditioned on the agent’s navigation.
D. Human Navigation ` Machine QA
The study compares question answering from human and shortest-path navigation endings and illustrates the visual and dataset structures used in evaluation. Human trajectories yield a lower mean rank because people stop earlier, leaving more cluttered views than shortest paths.
- Human Navigation vs. Machine QA: 3.51 mean rank of the ground-truth answer is obtained from the last five frames of human navigation, compared with 3.26 from shortest-path navigation.
- Human Navigation vs. Machine QA: The authors attribute the human-navigation difference primarily to training the QA system on shortest paths but testing it on human trajectories.
- Human Navigation vs. Machine QA: Shortest paths typically end with the target object filling most of the view, whereas humans stop once the answer is discernible and produce more cluttered observations.
- Question Answering: The QA module attends over the last five navigation frames and combines image features with question encoding to predict answers.
- Environment and Questions: House3D visualizations include queryable objects and rooms, with object instances varying substantially in shape, size, and color.
- Environment and Questions: Location, preposition, and color template questions are represented by answer distributions across environments, with entropy-and-count filtering pruning some questions.The corresponding figures show the distributions for room location, neighboring objects, and object color questions.