Neural Modular Control for Embodied Question Answering

1 Introduction

• EmbodiedQA requires an agent to locate a question-relevant entity from egocentric vision across potentially many rooms, with partial observability, long horizons, and sparse final-answer rewards.The agent may execute hundreds of primitive actions before receiving a reward.Source: §1 · p. 1
• The Neural Modular Controller uses a master policy to select semantic subgoals and specialized sub-policies to execute variable-length primitive actions.For example, a plan can include exiting a room, finding the relevant room and object, then answering.Source: §1 · pp. 1–2
• Semantic subgoals compress planning: the master makes orders of magnitude fewer decisions than a flat primitive-action policy, allowing the answering reward to influence high-level control more easily.The hierarchy separates semantic subgoal selection from sub-policy execution, shortening learning horizons.Source: §1 · p. 2
• Modules can be pretrained independently before combination, and semantic master-policy predictions expose the agent’s current intent more clearly than primitive actions.The introduction also reports that reinforcement learning after behavior cloning improves performance over either training regime alone.Source: §1 · p. 2
• NMC significantly outperforms prior work in both navigation and question answering on the EQA benchmark in House3D.The agent navigates closer to target objects and answers questions correctly more often.Source: §1 · p. 2

Hierarchical Reinforcement and Imitation Learning

• NMC instantiates the options-style division between a global policy proposing subgoals and local policies pursuing them, using known task structure with hierarchical behavior cloning and actor-critic learning.The authors contrast this with automatic subgoal discovery, arguing that simple heuristics can effectively break long-range planning into sequential subgoals when problem structure is known.Source: §2 · p. 2

Neural Module Networks and Policy Sketches

• Unlike neural module networks that access a fully observable image, NMC sub-policies use first-person RGB, making active reevaluation of subgoals essential.Unlike policy sketches, NMC learns to produce subgoals rather than receiving a task-specific sketch at test time.Source: §2 · pp. 2–3
• NMC sub-policies accept room and object arguments, sharing parameters across similar navigation tasks instead of learning a separate policy for each target.The paper contrasts this with policy sketches, which assume subgoal supervision at test time; NMC uses demonstrations and plans during training but must generate its own subgoals at test time.Source: §2 · p. 3

Grounded Language Learning

• The master grounds question words into semantic subgoals, while sub-policies ground those targets into primitive actions and raw pixels.Both levels are neural control policies trained end-to-end.Source: §2 · p. 3

Embodied Question-Answering Agents

• Compared with prior hierarchical EQA systems, NMC uses human-interpretable semantic subgoals and is described as navigating longer paths spanning multiple rooms.The cited prior systems include work in House3D and AI2-THOR.Source: §2 · p. 3

3.1 Hierarchical Policy

• NMC alternates between a master policy that selects subgoals and sub-policies that execute primitive actions until stopping or reaching an action limit.The sub-policy’s terminal state becomes the master policy’s state for selecting the next subgoal.Source: §3.1 · p. 3
• The hierarchy defines 64 subgoals across exit-room, find-room, find-object, and answer, with room and object arguments for the two search tasks.The 50 object targets and 12 room types are arguments; exit-room and answer take none.Source: §3.1 · p. 4
• Sharing task parameters across argument-specific sub-policies is intended to make learning more sample-efficient than training a separate policy for every task-argument combination.Source: §3.1 · p. 4

3.2 Hierarchical Behavior Cloning from Expert Trajectories

• Expert subgoal sequences and their temporal segments are extracted from shortest paths using room transitions and House3D bounding boxes.The shortest paths provide supervision on training environments; agents must navigate from egocentric vision alone at test time.Source: §3.2 · p. 5, Figure 2 · p. 5
• Hierarchical behavior cloning trains the master and sub-policies as independent supervised problems, with the master effectively assuming perfect sub-policies.The expert paths are segmented into subgoal decisions and sub-policy trajectories before cloning.Source: §3.2 · pp. 5–6

3.3 Asynchronous Advantage Actor-Critic (A3C) Training

• After behavior cloning, the policies are fine-tuned with reinforcement learning, first independently and then jointly so they can handle errors and coordinate.The master is subsequently trained using the learned sub-policies rather than expert subgoal trajectories.Source: §3.3 · p. 6
• The reward combines question-answering success with distance-based progress rewards, while sub-policies also receive a small collision penalty.Sub-policy terminal rewards indicate successful completion of their specialized goals.Source: §3.3 · p. 6
• BC+A3C significantly outperforms behavior cloning or A3C alone on the sub-policy tasks.The figure reports success rates over training iterations averaged across five runs.Source: Figure 3 · p. 7

Dataset

• The EQA v1 benchmark uses non-overlapping training, validation, and test environments to assess generalization to novel environments.Source: §4 · p. 7

Evaluating sub-policies

• The sub-policy evaluation measures success on 68 validation environments, with policies started in suitable rooms and limited to 50 steps or termination.Source: §4 · p. 7
• BC+A3C sub-policies significantly outperform policies trained with BC or A3C alone, nearly doubling BC success rates for some tasks.BC learns faster than A3C from scratch on all tasks and achieves higher success rates for Exit-room and Find-room.Source: §4 · p. 7

Evaluating master policy

• The master policy achieves approximately 48% intersection-over-union with ground-truth subgoal sequences on the validation set.The authors characterize this as a strict metric because a different subgoal sequence can still reach the target and answer correctly.Source: §4 · p. 7

Evaluating NMC

• NMC (BC+A3C) outperforms PACMAN with higher d∆, lower dT, and higher answering accuracy.Table 2 reports navigation and answering metrics on the EQA v1 test set; the comparison uses the final NMC model.Source: Table 2 · p. 8, §4 · p. 8

• NMC outperforms prior work in both navigation and question answering on the EQA v1 dataset.Source: §5 · p. 8
• A master policy proposes semantic subgoals from the question, while specialized sub-policies execute each task.Source: §5 · p. 8
• Behavior cloning and reinforcement learning together are more sample-efficient than either regime alone; cloning provides dense navigation feedback, while RL supports adapting to action consequences and recovering from errors.Source: §5 · p. 8