Embodied Question Answering

1. Introduction

• EmbodiedQA requires an agent to navigate a house or building from a random starting point, using a single RGB camera and atomic movement and turning actions to gather information for answering a question.The task is illustrated with questions such as “What color is the car?”Source: §1 · p. 1, Figure 1 · p. 1
• Success requires active perception, commonsense navigation from egocentric views without a map, grounding questions into action sequences, and learning long action sequences from sparse answer rewards.The agent may need to move to reveal relevant pixels, infer likely locations such as a garage, and coordinate exploration, detection, counting, and stopping.Source: §1 · pp. 1–2
• Training uses instrumented environments to support imitation learning and reward shaping, while test-time agents rely on egocentric RGB alone and are trained end-to-end without maps or heuristic planning.Training access includes agent location, depth, semantic annotations, and obstacle-avoiding shortest paths; test-time agents have no structured environment representation or hand-coded task knowledge.Source: §1 · p. 2
• The proposed Adaptive Computation Time navigator separates action selection by a planner from primitive-action execution by a controller, which runs for a variable number of steps before returning control.The agent stops navigating and answers when it decides it has seen the information needed.Source: §1 · p. 2
• The authors evaluate in House3D and test generalization to unseen environments, using imitation initialization and reinforcement-learning fine-tuning to improve question-answering accuracy.House3D scenes support generalization tests across floor plans, objects, and room configurations.Source: §1 · p. 2
• The EQA dataset grounds visual questions in House3D, uses functional programs to generate and execute questions with controlled types and answers, and includes AMT-collected expert navigation demonstrations as a benchmark.Question types include location, preposition, and color, enabling fine-grained performance breakdowns and reducing opportunities to exploit dataset bias.Source: §1 · p. 2

VQA: Vision + Language

• EmbodiedQA agents answer questions from video they gather by controlling their trajectories, unlike VQA agents limited to fixed visual views [13].Active perception makes the task harder because many random paths are uninformative, but lets agents seek views that increase answer confidence.Source: §2 · p. 3

Visual Navigation: Vision + Action

• Unlike visual navigators trained per reward or conditioned on goal states, EmbodiedQA specifies goals in compositional language.The authors use pixel-to-action navigators, but language goals make a separate policy for every question infeasible.Source: §2 · p. 3

Situated Language Learning: Language + Action

• Unlike situated language-learning work using declarative programs, templated commands, or natural-language instructions [26] [27], EmbodiedQA requires first-person sensing in partially observable environments.Agents lack floor plans, object labels, and attributes, and must extract this information from visual observations.Source: §2 · p. 3

Embodiment: Vision + Language + Action

• Compared with embodied language-learning agents in simple rooms or mazes [17] [21], EmbodiedQA uses multi-room homes with about 54 unique objects per home and asks higher-level questions.The cited prior agents learned low-level commands such as “go to X” or “pick up X”; EmbodiedQA homes have about 8 rooms.Source: §2 · p. 3

Interactive Environments

• House3D offers a middle ground between limited-realism game environments and realistic environments that are difficult to create in large numbers [17] [27].The section contrasts rich but few realistic environments with programmatically generated synthetic environments that typically lack realism.Source: §2 · p. 3

Hierarchical Agents

• The authors model EmbodiedQA agents hierarchically, with a high-level planner invoking lower-level controls to issue primitive actions, drawing on deep RL hierarchy and Adaptive Computation Time.Source: §2 · p. 3

3. EQA Dataset: Questions In Environments

• The authors present EQA as a curriculum of embodied capabilities and state that they will publicly release the environments, curated dataset, code, and EQA v1 generation engine.The generator is intended to be extensible with new operations, question types, and templates as models develop.Source: §3 · p. 4, §3.2 · pp. 5–6

3.1. House3D: Interactive 3D Environments

• SUNCG scenes become navigable House3D environments, and the dataset uses houses judged realistic by all three annotators, within size limits and containing four required room types.The selected houses have an internal area of 300–800 m^2 covering at least one-third of the total ground area; each must include a kitchen, living room, dining room, and bedroom.Source: §3.1 · p. 4

3.2. Question-Answer Generation

• EQA questions are programmatically generated as functional programs over scene annotations, enabling control of question-type and answer distributions to deter dataset-bias exploitation.The program operations select, make unique, and query objects or rooms; requiring unique objects helps avoid ambiguous questions when multiple instances exist.Source: §3.2 · pp. 4–5
• The generator defines nine question types, while EQA v1 focuses on location, color, color_room, and preposition questions with a single target object.The broader types test abilities including detection, scene recognition, counting, spatial reasoning, color recognition, and logic; the single target in v1 enables shortest-path expert demonstrations for imitation learning.Source: §3.2 · p. 5
• Questions are filtered to reduce answer-distribution bias and human tedium, including normalized-entropy and frequency thresholds and manual exclusions for difficult counts or unclear distances.Questions with normalized answer entropy below 0.5 or occurrence in fewer than four environments are excluded; high-count object questions and distance questions without clear differences are also omitted.Source: §3.2 · p. 5
• EQA v1 contains over 5,000 questions across over 750 environments, referring to 45 unique objects in 7 room types, with no environment overlap across splits.There are approximately 6 questions per environment on average; environments range from 1 to 22 questions.Source: §3.2 · p. 5

4. A Hierarchical Model for EmbodiedQA

• The agent processes raw pixels and words through vision, language, navigation, and answering modules, without access to a global or structured environment or task representation.Its navigation module uses Adaptive Computation Time (ACT) RNNs to separate decisions about direction from decisions about how far to move.Source: §4 · p. 6
• A CNN encodes egocentric 224×224 RGB images and is pretrained to reconstruct RGB, semantic class, and depth; separate two-layer LSTMs encode questions for navigation and answering.The separate encoders let the modules emphasize different question information, such as location for navigation and color for answering.Source: §4 · p. 6
• An ACT planner LSTM conditions on the question and selects forward, turn-left, turn-right, or stop-navigation actions at variable intervals.This separates the agent’s intended direction from the primitive actions used to carry it out.Source: §4 · pp. 6–7
• The controller uses the planner state and current frame to repeat the selected action or return control, halting after its signal or at most five controller steps.A one-hidden-layer feed-forward network implements the controller, which continues until the visual input aligns with the planner’s intent.Source: §4 · p. 7
• After navigation stops or reaches its action limit, the answering module attends over the last five frames and combines pooled visual features with the question to predict among 172 answers.Attention weights are based on image-question similarity.Source: §4 · p. 7

4.1. Imitation Learning and Reward Shaping

• Training first imitates shortest-path demonstrations, then freezes the answerer and fine-tunes navigation with REINFORCE using answer accuracy and distance-based shaping rewards.A correct answer after stopping earns 5, while forward actions receive 0.005 times the change in distance to the target; navigation training uses an increasing-distance curriculum.Source: §4.1 · p. 7

Evaluation metrics

• Answering is evaluated by mean rank among 172 possible answers, while navigation uses target-distance, target-room, and stopping metrics across 10-, 30-, and 50-action starts.Navigation distances are measured in meters along shortest paths; the measures include distance at termination, change from initial distance, and minimum episode distance.Source: §5 · pp. 7–8

Results

• ACT+Q achieves the smallest final target distance (dT), while ACT+Q-RL achieves the highest answering accuracy.Table 1 reports navigation and answering metrics on the EQA v1 test set.Source: Table 1 · p. 8, §5 · p. 9
• All baseline navigators end farther from the target than they start, while memory-augmented LSTMs finish closer than memoryless reactive models.The comparison includes LSTM versus Reactive and LSTM-Q versus Reactive-Q.Source: §5 · p. 8
• ACT+Q-RL reaches the target most closely during its trajectory and enters its room most often, but its lower stopping rate leads it to overshoot.The paper suggests a small negative reward per action could correct this behavior; overshooting does not hurt answering because the module attends to trajectory frames, consistent with.Source: §5 · p. 9
• Several methods outperform ShortestPaths+VQA in answering accuracy because the shortest route may not provide the best viewpoint for answering.The paper suggests ray tracing to frame the target appropriately at termination as a possible improvement.Source: §5 · p. 9

• EmbodiedQA requires an agent to navigate from a random starting location, gather information through egocentric vision, and answer a question.Source: §6 · p. 9
• The neural hierarchical model separates navigation into a planner that selects actions or direction and a controller that selects velocity and executes primitive actions for variable durations.Source: §6 · p. 9
• The agent is initialized through imitation learning and fine-tuned with reinforcement learning to answer questions; it is evaluated using protocols developed for EmbodiedQA in House3D.Source: §6 · p. 9
• The work collects human demonstrations through Amazon Mechanical Turk workers remotely controlling embodied agents and states that its code, data, and infrastructure will be made publicly available.Source: §6 · p. 9

Sec. A

• Sec. A details the question-answer generation engine, its question-associated functional programs, and safeguards against ambiguity, bias, and redundancy.Source: Appendix Overview · p. 10

Sec. B

• Sec. B describes the CNN vision module’s architecture, training details, and quantitative and qualitative results.Source: Appendix Overview · p. 10

Sec. C

• Sec. C describes the answering module in the EmbodiedQA agent.Source: Appendix Overview · p. 10

Sec. D

• Sec. D reports machine question answering performance conditioned on human navigation paths collected through Amazon Mechanical Turk studies.Source: Appendix Overview · p. 10

Functional Forms of Questions

• Question programs select entities, optionally retain unique instances or remove blacklisted objects, and fill templates to generate question strings.The selection operation can retrieve rooms or objects, while template generation instantiates questions from the resulting entities.Source: A. Question-Answer Generation Engine · p. 10
• Spatial-relation questions use relate to return object pairs, while distance-comparison questions use distance to return triplets ordered by closeness to an anchor.The relations include on, above, under, below, and next to.Source: A. Question-Answer Generation Engine · p. 10
• Location and color questions require a unique object, while color_room, preposition, and distance questions require both a unique room and unique objects within it.Room selection and uniqueness precede object selection and uniqueness; object selection is then scoped to those rooms.Source: A. Question-Answer Generation Engine · pp. 10–11
• Existence, logical, and count questions require a unique room but allow multiple instances of the queried object, so their forms omit object-uniqueness filtering.For example, asking whether a living room contains a television remains valid when several televisions are present.Source: A. Question-Answer Generation Engine · p. 11

Checks and balances

• Questions with normalized answer entropy below 0.5 or occurring in fewer than 4 environments are dropped; count answers above 5 are excluded, and distance comparisons require a 2-metre gap.These checks limit answer-distribution skew, unreliable low-frequency entropy estimates, tedious counts, and ambiguous closer/farther comparisons.Source: A. Question-Answer Generation Engine · p. 11
• The pipeline collapses visually similar or hierarchically related object labels and excludes obscure room names and unsuitable objects from question templates.Manual exclusions target names that are unclear, objects that are tiny or vague, and visually similar labels such as teapot and coffee_kettle.Source: A. Question-Answer Generation Engine · p. 11

Architecture

• The CNN encodes egocentric House3D RGB with four convolutional blocks and uses three upsampling decoder heads with skip connections for RGB reconstruction, 191-class segmentation, and depth.The decoder heads use 3, 191, and 1 output channels for reconstruction, segmentation, and depth, respectively.Source: B. CNN Training Details · p. 12

Training Details

• Training combines cross-entropy segmentation loss with Smooth-ℓ1 depth and reconstruction losses, weighted by 10 each, using Adam at 10^-3 for five epochs on 100k renderer images.Training uses a batch size of 20.Source: B. CNN Training Details · p. 12

Quantitative Results

• Validation compares hybrid and single-task training: segmentation uses overall pixel accuracy, mean pixel accuracy across 191 classes, and mean IoU, while depth and reconstruction use Smooth-ℓ1.The passage describes the metrics but provides no numerical results.Source: B. CNN Training Details · p. 12

Qualitative Results

• Figure 6 shows validation examples of segmentation, depth prediction, and autoencoder reconstruction.Source: B. CNN Training Details · p. 12

C. Question Answering Module

• It encodes the question with an LSTM and the last five navigation frames with CNNs, applies dot-product attention across those frames, then combines their weighted image features with the question encoding to predict a softmax answer distribution.The module predicts the agent’s beliefs over answers given its navigation.Source: C. Question Answering Module · p. 12

D. Human Navigation ` Machine QA

• The authors attribute the difference mainly to training on shortest paths but testing on human navigation, where people stop earlier and views are more cluttered.Shortest paths typically end with the object of interest filling most of the view, while humans stop once they can discern the answer, making the QA module’s task harder.Source: D. Human Navigation ` Machine QA · p. 12
• The ground-truth answer’s mean rank was 3.51 on the last five frames of human navigation, versus 3.26 on shortest paths.Figure 10 contrasts the last five frames from human and shortest-path navigations across three questions and environments.Source: D. Human Navigation ` Machine QA · p. 12, Figure 10 · p. 16