Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

Motivation

• Because agents may struggle in unknown environments and human assistants may be unavailable or unwilling to help frequently, the paper motivates mindful help requests.Natural language can direct agents even in unfamiliar environments, but assistance must be sought and leveraged when agents cannot manage tasks alone.Source: §1 · p. 1

HANNA problem

• HANNA places agents in map-free object-finding tasks where they interpret first-person vision and request language-and-image subtasks from region-limited assistants.ANNA’s subtasks guide agents closer to the goal but need not complete the task.Source: §1 · pp. 1–2

Model and training

• The proposed agent uses a hierarchical memory-augmented recurrent architecture to model assistance as subgoals, with imitation learning that encourages exploration and interpretable help requests.Training teaches navigation and when requesting assistance is worthwhile; testing requires those decisions without additional supervision.Source: §1 · p. 2

Results

• 47% of tasks were completed in new environments with unseen scenes and instructions, and the method outperformed competitive baselines by large margins.Training used 51 environments and fewer than 9,000 language instructions.Source: §1 · p. 2

2 Related work

• Realistic simulators support sim-to-real transfer, but simulating language interaction remains challenging; text-only interactive worlds lack the graphical component needed for visually grounded learning.Experimentation with real humans and robots is important for understanding real-world complexity, but is expensive and time-consuming.Source: §2 · p. 3
• Prior photo-realistic navigation either omits help requests or uses scripted language, whereas HANNA provides natural instructions and reverses Tellex et al.’s (2014) intervention direction.Table 1 contrasts HANNA’s adaptive language-and-vision assistance with VLN’s lack of help requests, VNLA’s templated language, and CVDN’s limited language-interaction simulation.Source: §2 · p. 3, Table 1 · p. 3
• Standard imitation-learning teachers disregard the agent’s capability and behavior, while HANNA’s objective penalizes negative-action likelihoods and combines past- and future-behavior estimates.The approach builds on an unlikelihood objective to avoid repeating mistakes and combines model-based and progress-estimating methods.Source: §2 · p. 3

Problem

• Tasks are feasible, high-level object-finding requests with a T-step budget; success means ending within ϵsuccess of any target, using only a monocular RGB camera and no location sensors.Agents may request help only when within an assistant’s zone of attention.Source: §3 · pp. 3–4

Photo-realistic Navigation Simulator

• The photo-realistic simulator spans 68 residential Matterport3D buildings and models navigation on a weighted indoor graph, with the agent’s pose determining its first-person view.Each movement updates the image to reflect the new view.Source: §3 · p. 4

Automatic Natural Navigation Assistants (ANNA)

• ANNA models assistants as collections of language-guided routes they know only partially; in route-entry zones, they select a route and departure point toward a goal and send instruction-plus-view guidance.When the agent leaves a route, the assistant also provides an image of the panoramic view at the goal nearest the departure point.Source: §3 · pp. 4–5

Constructing ANNA route system

• Under perfect instruction execution, guiding agents between any location pair requires only Θ(N log N) collected instructions rather than the O(N^2) worst-case construction.The implementation uses Room-to-room’s 21,567 crowdsourced instructions, averaging 211 routes connecting 125 locations per environment; experiments found these routes sufficient assuming perfect assistance interpretation.Source: §3 · p. 5

Agent Policies

• The agent maintains stochastic navigation and help-request policies; help can be requested only in a zone of attention, and navigation actions include movement, camera changes, and stop.The stop action can end the main task or a subtask by departing a route.Source: §4 · p. 6

Imitation Learning Objective

• HANNA frames assistance as indirect intervention that augments the environment with information, and introduces losses to enforce behaviors beyond simply mimicking reference actions.A common imitation loss is the agent-estimated negative log likelihood of the teacher’s reference action.Source: §4 · p. 6

Reference Actions

• Navigation references follow shortest paths to the nearest goal or ANNA-indicated image location, while help is requested only when lost or uncertain and not previously requested there.Uncertainty means navigation-action entropy meets or exceeds threshold γ while the most probable action differs from the navigation teacher’s suggestion.Source: §4 · p. 6

Curiosity-Encouraging Navigation Teacher

• The curiosity-encouraging loss minimizes the likelihood of non-reference actions previously taken at the current location, discouraging repeated mistakes and encouraging exploration of untried actions.The teacher records non-reference actions taken at that location while following the same language instruction.Source: §4 · p. 6

Retrospective Interpretable Help-Request Teacher

• The retrospective help-request teacher uses completed episodes to consider future actions and is optimal for teaching the agent to determine whether it is lost.At each time step, it suggests an action that leads the agent closer to the target under its current navigation policy, if such an action exists.Source: §4 · p. 7
• A reason classifier predicts which of the three help-request conditions are satisfied using multi-label binary classification, supporting more accurate and interpretable decisions.The teacher supplies a three-element binary reason vector, trained with a binary logistic loss for each condition.Source: §4 · p. 7

5 Hierarchical Recurrent Architecture

• The navigation and help-request policies use separate networks with text-encoding, inter-task, and intra-task components, self-attention, cosine-similarity attention, and ResNet-based time encoding.The help-request network is mostly similar to the navigation network, but takes the navigation action distribution as input when computing state features.Source: §5 · p. 7, Figure 2 · p. 8
• The text memory represents the current instruction, while the inter-task state tracks task execution and resets when help is requested or the agent departs a route.When the task changes, the instruction is re-encoded and the inter-task state is reset to zero.Source: §5 · p. 7
• The intra-task state represents the episode and incorporates weighted past states from nearly identical situations through a context-sensitive dissimilarity scaled by β.This formulation incorporates related past information into the current state to support optimization of the curiosity-encouraging loss.Source: §5 · p. 7
• The intra-task state is passed through a softmax layer to produce an action distribution.Source: §5 · p. 7

Dataset

• The dataset spans 289 object types and 2,332 instruction words, with goals 5–15 shortest-path locations away, evaluated in SEENENV and UNSEENALL.With average edge length 2.25 meters, reaching a goal requires about 9–32 meters of travel. Even in SEENENV, evaluation tasks and ANNA instructions were not paired with the same environments during training.Source: §6 · p. 8

Baselines and Skylines

• Navigation baselines were SHORTEST (teacher-policy skyline), RANDOMWALK, and FORWARD10; help-request heuristics were NOASK, RANDOMASK (probability 0.2), and ASKEVERY5 (after five walking steps).FORWARD10 advances for 10 time steps toward the location closest to the view center; RANDOMASK's probability matches the learned agent's average help-request ratio.Source: §6 · p. 8

Evaluation metrics

• Evaluation reports success rate (SR), navigation error, and SPL (Anderson et al., 2018a), which weights task success by travel distance.SR is the fraction of successfully solved tasks; navigation error is the average shortest-path distance from the final location to the nearest goal. In the SPL definition, Si indicates task success, Pi travel distance, and Li the shortest-path distance to the goal nearest the final location.Source: §6 · p. 8

Main results

• Help from ANNA raises success by 71.16% on TEST SEENENV and 39.35% on TEST UNSEENALL over no help, while heuristic baselines stay below 7%.With fewer than 9,000 subtask instructions, the agent succeeds in nearly half its tasks; its paths are 1.38 and 1.86 times optimal on TEST SEENENV and TEST UNSEENALL, respectively. In unseen environments, it requests help about twice as often as in seen environments, and optimal navigation during subtasks reveals substantial room to improve instruction execution.Source: §7 · pp. 8–9, Table 3 · p. 9

Does understanding language improve generalizability?

• Language assistance improves success by +15.17% on TEST UNSEENALL, compared with +3.42% on TEST SEENENV, supporting better task completion in novel environments.Source: §7 · p. 9, Table 4 · p. 9

Is learning to request help effective?

• On TEST UNSEENALL, the learned help policy beats every baseline by at least 10.40% in success while requesting help less often; ASKEVERY5 gains +26.32% over no help.The small gap between the learned policy and ASKEVERY5 on TEST UNSEENALL is expected because performance there is mostly determined by memorization and is relatively insensitive to the help-request strategy.Source: §7 · p. 9, Table 5 · p. 9

Is proposed model architecture effective?

• On TEST UNSEENALL, the hierarchical recurrent model outperforms an approximately 15M-parameter LSTM encoder-decoder by +28.2%.Source: Table 6 · p. 9, §7 · p. 10

Does the proposed imitation learning algorithm achieve its goals?

• Curiosity-encouraging training reduces repeated looping decisions and raises TEST UNSEENALL success by +4.33% over training without that objective.Source: Table 6 · p. 9, §7 · p. 10

• The authors present a photo-realistic simulator that mimics primary characteristics of real-life human assistance.Source: §8 · p. 10
• Their imitation-learning techniques teach agents to request and interpret simulated assistance, coupled with a hierarchical neural network model for representing subtasks.Source: §8 · p. 10
• Future work aims to enable more natural, linguistically realistic interaction, develop a theoretical framework for human assistance, and deploy and evaluate the methods on real-world platforms.Source: §8 · p. 10

A Proof of Lemma 1

• Θ(N log N) instructions suffice to support navigation between any two locations with O(log N) language-assisted paths.The lemma states the instruction requirement; the proof constructs at most 2 · N log2 N instructions.Source: A Proof of Lemma 1 · p. 13
• For each node, the construction collects forward and reverse instructions to ancestors at distances 2^i, for i from 0 to ⌊log2 d(v)⌋−1.The proof roots a spanning tree at vr and uses d(v), the distance from v to the root, to set the ancestor range.Source: A Proof of Lemma 1 · p. 13
• For arbitrary nodes u and v, routing through their least common ancestor w takes O(log N) language-assisted paths: first u to w, then w to v.The construction supports travel between a node and any ancestor in O(log N) paths.Source: A Proof of Lemma 1 · p. 13

B Proof of Lemma 2

• The retrospective teacher uses the agent’s future trajectory to request help only when its current navigation policy will not move it closer to the target.If the agent would make progress without help, the teacher suggests doing nothing; otherwise, it requests help when that action can improve progress.Source: B Proof of Lemma 2 · p. 13

C Features

• The agent’s visual inputs include panoramic features for all 36 current-view angles and the target panorama, with navigation actions embedded using view and orientation features.The image features come from ResNet-152 pretrained on ImageNet; stop is represented by a zero vector.Source: C Features · pp. 13–14
• Target-similarity features match each current view angle to its most similar target angle, while learned local and global time features generalize to longer episodes.The paper reports that replacing the ResNet-based decoder encodings with sinusoidal encodings in all modules significantly degraded success rate.Source: C Features · p. 14

D Model

• The model uses Transformer modules with multi-headed attention and residual, layer-normalized feed-forward layers, alongside separate navigation and help-request networks.The help-request network also receives the navigation action distribution as an extra input.Source: D Model · pp. 14–15
• The model’s similarity-attention module retrieves weighted values from past states whose keys are nearly identical to the query, supporting curiosity-encouraging training.The intra-task component uses a weighted combination of past states from nearly identical situations; the inter-task component maintains state for the current task.Source: D Model · pp. 14–15
• Training collapses with α = 1 for the NOASK, RANDOMASK, and ASKEVERY5 help-request baselines, so those experiments use α = 0.5.Table 7 summarizes the training and evaluation hyperparameters.Source: Table 7 · p. 15

E Hyperparameters

• Training the agent takes approximately 9 hours on a single Titan Xp GPU.The paper directs readers to Table 7 for the complete training and evaluation hyperparameters.Source: E Hyperparameters · p. 16

Ablation Study

• The curiosity-encouraging loss slightly reduces VAL SEENENV memorization but improves generalization to unseen environments; condition prediction boosts seen performance while slightly degrading VAL UNSEENALL.The paper attributes the memorization cost to the small model devoting parameters to curiosity behavior. Jointly training navigation and help-request policies is challenging because improving navigation leaves fewer positive examples for learning when help is needed.Source: F Analysis · p. 16
• Without help-request condition prediction, the agent encounters uncertain wrong conditions and repeats help requests at previously visited locations more often.These increases are +5.2% on VAL SEENENV and +6.4% on VAL UNSEENALL for uncertain wrong conditions, and +7.22% on VAL SEENENV and +8.59% on VAL UNSEENALL for repeat requests during the same task.Source: F Analysis · p. 16

Help-request Behavior on TEST UNSEENALL

• On TEST UNSEENALL, the final agent requests help for about 18% of time steps and has precision of 76.9% for lost and 86.6% for uncertain wrong conditions.Recall is below 50% for all conditions, and the agent predicts the already asked condition less accurately than the other two.Source: F Analysis · pp. 16–17

G Mini-batch Resetting

• Resetting all agents’ inter-task modules and local times when one agent resets helps escape loops and boosts success rates by about 1–2%, but makes behavior and success rates depend on mini-batch composition.With batches larger than one, an agent’s behavior depends on the other agents grouped with it, complicating comparison and evaluation.Source: G Mini-batch Resetting · p. 17
• Batch-size-1 evaluation removes dependence on mini-batch construction; results are slightly lower than those in § 7 and Appendix F, but the conclusions remain true, and future comparisons are recommended to use Tables 9–13.The section includes validation ablations, test-split results, assistance-type and help-request-policy comparisons, and an unseen-environment analysis.Source: G Mini-batch Resetting · p. 17, Table 9 · p. 18, Table 10 · p. 18