Source-linked AI summary
Multi-Target Embodied Question Answering
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, Dhruv Batra
TL;DR
EQA-v1 is limited to single-target questions, leaving questions that require visits to multiple targets and comparative reasoning outside its task scope. This paper introduces MT-EQA and a modular agent that converts questions into executable sub-programs and coordinates navigation, observation selection, and VQA; the joint model significantly outperforms baselines on navigation and question answering.
Problem
EQA-v1 contains single-target questions, while questions requiring navigation to multiple targets and comparison between them are outside the original task.
Source: §1 · p. 1Method
The paper proposes MT-EQA and a modular agent with program generation, navigation, a controller for collecting relevant observations, and VQA for answering comparative questions.
Source: §1 · p. 2Results
Takeaways & Limitations
MT-EQA serves as a benchmark for questions comparing attributes across multiple objects or rooms.
Source: §3 · p. 3MT-EQA generation is affected by noisy SUNCG annotations and inaccurate House3D occupancy maps under the axis-aligned assumption.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Embodied Question Answering (EQA) is a relatively new task where an agent is asked to answer questions about its environment from egocentric perception. EQA makes the fundamental assumption that every question, e.g., "what color is the car?", has exactly one target ("car") being inquired about. This assumption puts a direct limitation on the abilities of the agent. We present a generalization of EQA - Multi-Target EQA (MT-EQA). Specifically, we study questions that have multiple targets in them, such as "Is the dresser in the bedroom bigger than the oven in the kitchen?", where the agent has to navigate to multiple locations ("dresser in bedroom", "oven in kitchen") and perform comparative reasoning ("dresser" bigger than "oven") before it can answer a question. Such questions require the development of entirely new modules or components in the agent. To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module. The program generator converts the given question into sequential executable sub-programs; the navigator guides the agent to multiple locations pertinent to the navigation-related sub-programs; and the controller learns to select relevant observations along its path. These observations are then fed to the VQA module to predict the answer. We perform detailed analysis for each of the model components and show that our joint model can outperform previous methods and strong baselines by a significant margin.
1. Introduction
EQA-v1’s single-target questions exclude tasks requiring navigation to multiple targets and comparative reasoning; the introduction presents MT-EQA to address this gap. Its modular model integrates program generation, navigation, observation selection, and VQA, and the joint model outperforms baselines across almost every navigation and QA metric.
1. Introduction
- EQA-v1 uses single-target questions, whereas MT-EQA requires navigation to multiple targets and comparative reasoning between them.The introduction gives comparing kitchen and bedroom size as an example outside the original task’s scope.Source: §1 · pp. 1–2
- MT-EQA generalizes EQA to questions with multiple implicit targets and comparisons of attributes such as color, size, and distance.The questions compare properties of objects or rooms, and the paper frames the task as embodied visual reasoning that requires gathering information before reasoning.Source: §1 · p. 2
- The proposed agent combines a program generator, navigator, controller, and VQA module to execute question sub-programs and collect relevant observations.The controller processes first-person views, detects whether a navigation target has been reached, and extracts cues for the queried property before VQA compares attributes.Source: §1 · p. 2
- The joint model outperforms baselines by a large margin under almost every navigation and QA metric.The reported ablations also find improvements across room navigation, object navigation, and final EQA accuracy.Source: §1 · p. 2
2. Related Work
Related work places MT-EQA at the intersection of embodied perception, language-driven tasks, simulated environments, and modular models. The paper draws on these areas to situate its use of an interactive 3D environment and compositional reasoning.
2. Related Work
- Active perception and language tasks require agents to perceive surroundings, understand goals, and act, with generalization to unseen environments also identified as important.Examples include instruction-based navigation, target-driven navigation, embodied question answering, interactive question answering, and task planning.Source: §2 · p. 2
- Virtual environments offer an alternative to costly real-world interactive benchmarks, and the paper uses House3D, an interactive 3D environment based on SUNCG indoor scenes.The passage describes simulated environments as potentially overcoming challenges faced in real-world settings.Source: §2 · pp. 2–3
- Modular networks decompose questions into components and dynamically assemble computation; this paper extends the modular idea to partially observable EQA.Earlier applications cited include visual reasoning, relationship modeling, and embodied question answering.Source: §2 · p. 3
3. Multi-Target EQA Dataset
MT-EQA extends EQA-v1 from single-target property questions to questions comparing attributes across multiple targets. Its dataset uses executable programs to generate questions, filters for feasible and balanced examples, and contains 19,287 questions across 588 environments.
3.1. Multi-Target EQA Generation
- MT-EQA generalizes EQA-v1 with six question types comparing object attributes, distances, or room sizes across targets.Object comparisons can involve objects in the same room or across different rooms.Source: §3 · p. 3, §3.1 · p. 3, Table 2 · p. 4
- Each question is represented as an executable sequence of functions over room and object annotations, such as selecting and pairing objects before querying a color comparison.For example, the sequence selects a room and objects, pairs the objects, then queries their color comparison.Source: §3.1 · p. 3
- Generation filters out questions whose targets cannot be connected by a shortest path, using sampled best-view positions to determine target locations.For each target, the best-view position is selected from 100 samples using mask overlap with a centered rectangle; higher overlap indicates a more central view.Source: §3.1 · p. 3, Figure 3 · p. 3
- Entropy filtering excludes questions with normalized answer-distribution entropy below 0.93 to reduce answer memorization without environmental exploration.Questions that are always answered “Yes” are excluded, leaving the dataset both balanced and feasible.Source: §3.1 · pp. 3–4
- Question-based models make close to random predictions, consistent with the dataset’s balanced binary answers and the stated need to explore the environment.The yes/no answer distribution is 53.28%.Source: §3.1 · p. 4
- The dataset contains 19,287 questions across 588 environments, with fewer object- and room-size comparisons because many are easy to guess without exploration.It covers 61 object types and 8 room types.Source: §3.1 · p. 4, Figure 4 · p. 4
4. Model
The model combines a program generator, navigator, controller, and VQA module to answer questions involving multiple targets.
4. Model
- Four modules—the program generator, navigator, controller, and VQA module—make up the MT-EQA architecture.The controller connects the other modules during program execution.Source: §4 · p. 4, §4.3 · p. 6
4.1. Program Generator
- Template rules decompose questions into sequential executable programs, such as navigating to targets, querying attributes, and comparing them.The paper uses automatically generated templates rather than focusing on a learned program generator.Source: §4.1 · p. 5
4.2. Navigator
- Separate LSTM-based room and object navigators use egocentric views, target phrases, and previous actions to choose navigation actions.The two navigators share an architecture but are trained separately for their different targets.Source: §4.2 · p. 5
4.3. Controller
- The controller tracks observations and predicts SELECT to stop a navigator and advance the program; it extracts object attributes at selected views and room attributes from a panorama.It links generated plans, navigator observations, and VQA invocation.Source: §4.3 · p. 6
4.4. VQA Module
- Compositional VQA compares stored target features conditioned on the question’s attribute and comparison operator, and supports comparisons involving three targets.Its output is a binary yes/no prediction.Source: §4.4 · p. 6
4.5. Training
- Training first jointly uses imitation learning, then fine-tunes the navigator with reinforcement learning.Imitation learning uses shortest paths and key target positions as navigation and SELECT labels, alongside a final-step VQA objective.Source: §4.5 · p. 6
5. Experiments
The experiments evaluate question-answering and navigation, compare model variants, and use oracle settings to examine component performance.
5.1. Evaluation Setup and Metrics
- Agents start 10 actions from a randomly selected reference target; EQA accuracy is reported by question type and path-length difficulty.Navigation is evaluated per target, including object-target distance and hit accuracy, and room-target termination inside the room.Source: §5.1 · p. 7
5.2. EQA Results
- The full model includes a program generator, navigator, controller, and comparative VQA module; variants remove the controller or fine-tune navigation with RL.The reported comparisons cover navigation and EQA accuracy.Source: §5.2 · pp. 7–8
- 61.45% overall EQA accuracy after RL fine-tuning, versus 54.44% without it; hit accuracy rises from 20% to 33%.The reported distance change also shifts from −0.56 to 0.16.Source: §5.2 · p. 8
5.3. Oracle Comparisons
- With best-view images, the paper’s VQA module achieves 13.64% higher accuracy than attention-based VQA.Controller features outperform raw CNN features, while predicted SELECT features reduce overall accuracy by 13% versus oracle moments.Source: §5.3 · p. 8
6. Conclusion
The conclusion presents MT-EQA as a multi-target extension of EQA and reports a modular model evaluated on its dataset.
6. Conclusion
- MT-EQA extends single-target EQA to questions requiring comparison across multiple targets.The authors collected a dataset as a benchmark for the task.Source: §6 · p. 8
- The proposed MT-EQA model consists of a program generator, navigator, controller, and VQA module.The conclusion reports component analyses in both embodied and oracle settings.Source: §6 · p. 8
- The authors report that their model significantly outperforms baselines on question answering and navigation.They also report detailed ablations of the components.Source: §6 · p. 8