Source-linked AI summary
IQA: Visual Question Answering in Interactive Environments
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, Ali Farhadi
TL;DR
Interactive Question Answering asks agents to answer questions by navigating and interacting with dynamic visual environments, beyond passive VQA. The paper proposes HIMN, a hierarchical controller architecture with spatial-semantic memory, and evaluates it on IQUAD V1, where it outperforms a single-controller baseline. Its scope remains limited by a 2D map, inefficient exploration, and templated language.
Problem
Interactive Question Answering extends passive visual question answering by requiring navigation, environmental understanding, object interaction, and question-conditioned planning.
Method
HIMN factorizes control across a Planner and low-level controllers while using an Egocentric Spatial GRU to maintain spatial-semantic memory.
Results
HIMN outperforms a common single-controller reinforcement-learning baseline on IQUAD V1 and is evaluated in familiar and unfamiliar environments.
Takeaways & Limitations
IQUAD V1 provides over 75,000 multiple-choice questions with unique scene configurations for interactive question answering in AI2-THOR.
Takeaways & Limitations
HIMN cannot distinguish objects inside containers from objects on top of containers, explores inefficiently, and uses a simple templated language model.
Abstract
from arXiv · showhide
We introduce Interactive Question Answering (IQA), the task of answering questions that require an autonomous agent to interact with a dynamic visual environment. IQA presents the agent with a scene and a question, like: "Are there any apples in the fridge?" The agent must navigate around the scene, acquire visual understanding of scene elements, interact with objects (e.g. open refrigerators) and plan for a series of actions conditioned on the question. Popular reinforcement learning approaches with a single controller perform poorly on IQA owing to the large and diverse state space. We propose the Hierarchical Interactive Memory Network (HIMN), consisting of a factorized set of controllers, allowing the system to operate at multiple levels of temporal abstraction. To evaluate HIMN, we introduce IQUAD V1, a new dataset built upon AI2-THOR, a simulated photo-realistic environment of configurable indoor scenes with interactive objects (code and dataset available at https://github.com/danielgordon10/thor-iqa-cvpr-2018). IQUAD V1 has 75,000 questions, each paired with a unique scene configuration. Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1. For sample questions and results, please view our video: https://youtu.be/pXd3C-1jr98
1. Introduction
The paper introduces Interactive Question Answering, where agents must navigate, understand, manipulate, and plan within dynamic visual environments. It proposes HIMN and IQUAD V1 to address these challenges and evaluates the model across familiar and unfamiliar scenes.
- Interactive Question Answering requires agents to navigate, understand objects and affordances, interact with objects, and execute question-conditioned action sequences.
- HIMN factorizes question answering into hierarchical controllers operating across multiple temporal scales, including a high-level Planner and low-level task controllers.
- The Egocentric Spatial GRU provides an explicit spatial and semantic memory for tracking what the agent has seen and where it was encountered.
- Over 75,000 multiple-choice questions in IQUAD V1 are paired with unique scene configurations built on the photo-realistic AI2-THOR simulator.
- HIMN outperforms a common reinforcement-learning baseline and is evaluated in familiar and unfamiliar environments to assess generalization across scenes.
2. Related Work
Prior work includes passive VQA, hierarchical reinforcement learning, visual navigation, and language-conditioned navigation. The paper differs by jointly training hierarchical controllers for interactive question answering with semantic spatial memory in photo-realistic environments.
- Visual Question Answering: Earlier VQA architectures combined RNN question encodings and CNN image encodings, while newer modular networks represented compositional reasoning explicitly.
- Reinforcement Learning: Hierarchical reinforcement learning uses high- and low-level controllers to operate at multiple levels of temporal abstraction.
- Visual Navigation: Offline map-based navigation requires a complete environment map, limiting its use in unseen environments.
- Visual Navigation: Unlike related AI2-THOR approaches, this model jointly trains low- and high-level controllers and constructs an explicit semantic map.
- Language-Conditioned Navigation: Language-conditioned navigation models generally address navigation instructions, whereas this work handles multiple question types requiring varied navigation and interaction behaviors.
3. Learning Framework
The framework combines a simulated interactive environment, hierarchical controllers, and spatial memory to answer questions through navigation, observation, and object interaction.
- Interactive Question Answering Dataset: IQUAD V1 uses AI2-THOR’s photo-realistic indoor scenes with actionable objects and unique movable-object configurations for each question.The dataset contains over 75,000 multiple-choice questions paired with scene configurations.
- Spatial Memory: The esGRU writes only to a local memory window determined by the agent’s current location and viewpoint.Local updates speed computation and avoid corrupting memory at distant locations.
- Spatial Memory: The semantic spatial memory stores feature representations of scene locations so the agent can retain object information over long durations.This external memory is populated during interaction and can be accessed throughout an episode.
- Planner: The Planner reads the full memory for long-term decisions, whereas low-level controllers have read-write access to update it.This separates high-level planning from lower-level environmental interaction.
- Hierarchical Interactive Memory Network: HIMN factorizes control into a high-level Planner and specialized low-level controllers operating at multiple temporal scales.The Planner selects tasks, while controllers such as the Navigator execute them.
- Low-level Controllers: The Navigator uses A* search and continuously updates the global occupancy estimate from local visual observations.It produces a local 5x5 occupancy grid through the esGRU and recomputes paths as the estimate changes.
5. Experiments
HIMN is evaluated on IQUAD V1 against reinforcement-learning baselines, with analyses of ablations, invalid actions, generalization, and qualitative behavior. The experiments show benefits from hierarchical control and persistent semantic spatial memory, while identifying exploration, spatial representation, and language limitations.
- Experiments: HIMN significantly outperforms baselines across question types with both YOLO and ground-truth object detections.The comparison uses test accuracy and average episode length; A3C performs slightly worse than random chance even with ground-truth detections.
- Experiments: HIMN’s persistent semantic spatial memory supports long-duration observation retention, object-instance tracking, and question-conditioned exploration.These capabilities are especially useful for counting and spatial-relationship questions requiring thorough environmental search.
- Ablation Analysis: Ground-truth detections and depth, plus an oracle shortest-path Navigator, further improve HIMN performance.The reported ablations attribute gains to fewer detection errors and more accurate Planner-directed navigation.
- Invalid Actions: Supervised validity training makes invalid actions nearly an order of magnitude less frequent and helps the Planner learn faster.Invalid actions include navigation failures and impossible or distant object interactions; replacing the learned Navigator shows most failures are navigational.
- Generalization in Unseen Environments: HIMN loses only a few percentage points of accuracy on unseen environments despite training on relatively few rooms.Unseen tests use five never-before-seen rooms, while seen tests use 25 training rooms with new object placements and questions.
- Qualitative Results: A qualitative trajectory shows the Planner answering after the Detector adds bread to the semantic spatial map and confidence in the true answer approaches 100%.The Navigator continues its current task before returning control to the Planner, which then reads the updated map.
- Limitations: HIMN cannot distinguish objects inside containers from objects on top of containers because its semantic spatial map is two-dimensional.Proposed extensions include an explicit height parameter or multiple 2D slices forming a 3D map.
- Limitations: The method remains inefficient at exploration and uses a simple templated language model that does not extend to arbitrary questions.The authors propose traditional planning algorithms, more varied questions, and more expressive language embeddings as future directions.
6. Conclusion
The paper introduces Interactive Question Answering, HIMN, the Egocentric Spatial GRU, and IQUAD V1. HIMN learns across long trajectories, and the benchmark remains challenging for both the model and baselines.
- 6. Conclusion: The work formulates Interactive Question Answering and proposes HIMN, Egocentric Spatial GRU, and the IQUAD V1 benchmark.HIMN uses factorized controllers to learn from long trajectories, while the benchmark is built on a high-quality simulation environment.
- 6. Conclusion: IQUAD V1 continues to present challenges for HIMN and baseline models, motivating future research.