Source-linked AI summary
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
Khanh Nguyen, Hal Daumé
TL;DR
Mobile agents need assistance to handle navigation tasks beyond their unaided capabilities. HANNA introduces a photo-realistic simulator with requestable language-and-vision help, plus hierarchical imitation learning that avoids repeated mistakes and predicts future progress. The approach reaches 47% success in previously unseen environments and outperforms competitive baselines, though performance on one unseen split is largely driven by memorization rather than help-request strategy.
Problem
HANNA addresses how mobile agents can find objects in unknown environments by requesting and interpreting natural language-and-vision assistance when unaided navigation is difficult.
Method
The paper builds a photo-realistic HANNA simulator with ANNA assistants and trains a hierarchical memory-augmented agent using imitation learning that discourages repeated mistakes and models future progress.
Results
47% of tasks succeed in previously unseen environments, and the methods outperform competitive baselines by large margins.
Takeaways & Limitations
HANNA demonstrates that agents can learn to request and interpret simulated multimodal assistance across both seen and unseen environments.
Takeaways & Limitations
On TEST UNSEENALL, performance is mostly determined by memorization capability and is mostly insensitive to the help-request strategy.
Abstract
from arXiv · showhide
Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop "Help, Anna!" (HANNA), an interactive photo-realistic simulator in which an agent fulfills object-finding tasks by requesting and interpreting natural language-and-vision assistance. An agent solving tasks in a HANNA environment can leverage simulated human assistants, called ANNA (Automatic Natural Navigation Assistants), which, upon request, provide natural language and visual instructions to direct the agent towards the goals. To address the HANNA problem, we develop a memory-augmented neural agent that hierarchically models multiple levels of decision-making, and an imitation learning algorithm that teaches the agent to avoid repeating past mistakes while simultaneously predicting its own chances of making future progress. Empirically, our approach is able to ask for help more effectively than competitive baselines and, thus, attains higher task success rate on both previously seen and previously unseen environments. We publicly release code and data at https://github.com/khanhptnk/hanna . A video demo is available at https://youtu.be/18P94aaaLKg .
1 Introduction
HANNA frames object-finding as navigation with natural multimodal assistance that agents can request when progress stalls. The paper introduces a simulator, hierarchical agent, and imitation-learning approach for deciding when to seek and how to use help.
- HANNA models mobile agents navigating without maps to requested objects while interpreting first-person visual observations and requesting assistance.
- ANNA assistants provide reactive, partial-progress subtasks through natural-language instructions and images, rather than directly completing tasks.
- The simulator combines photo-realistic environments with simulated human assistance, modeling agents that ask for directions while finding objects.
- The agent uses a hierarchical memory-augmented architecture and imitation learning to navigate and decide when requesting assistance is worthwhile.
- 47% of tasks succeed in previously unseen environments, using training data from 51 environments and fewer than 9,000 language instructions.
2 Related work
Prior photo-realistic navigation tasks provide limited models of interactive human assistance. HANNA differs by combining requestable, multimodal, situation-adaptive help with less scripted interaction.
- Existing navigation tasks either do not let agents request help or provide only primitive, highly scripted assistance.
- VLN omits help requests, VNLA uses an always-available advisor with templated language, and CVDN limits assistance after trajectory deviations.
- HANNA assistants provide language-and-vision instructions that adapt to the agent’s current position and goal.
- The imitation-learning approach considers the agent’s past and future behavior when selecting beneficial and harmful actions.
3 The HANNA Simulator
The HANNA simulator embeds object-finding in photo-realistic indoor environments where agents can request route-based language-and-vision subtasks from localized assistants. These subtasks provide partial progress toward goals.
- Agents receive high-level object-finding requests and navigate indoor environments using only monocular first-person RGB observations.
- Assistance subtasks are guaranteed to move the agent closer to a goal but may not complete the main object-finding task.
- HANNA uses 68 Matterport3D residential environments represented as weighted navigation graphs with pose-dependent visual observations.
- ANNA assistants know scenes along selected paths and provide language instructions plus target-view images when agents request help nearby.
- Each assistance route has camera angles, a graph path, and an instruction; agents may enter it near its start and depart when the target view is reached.
- Reusing the Room-to-Room dataset supplies crowdsourced language instructions for constructing the route system.
4 Retrospective Curiosity-Encouraging Imitation Learning
The method trains navigation and help-request policies with imitation learning that discourages repeated mistakes and uses completed episodes to make future-aware help decisions. It also predicts reasons for requesting help.
- The agent maintains separate stochastic navigation and help-request policies over movement or stop actions and help-or-no-help decisions.
- The navigation teacher selects shortest-path reference actions toward goals or target images, while the help teacher requests assistance when agents are lost or uncertain and have not asked there before.
- A reason classifier predicts which lost, uncertainty, and repeat-request conditions support each help-request decision.
- The curiosity-encouraging loss lowers likelihoods of previously failed actions, discouraging repeated mistakes and encouraging unexplored actions.
- Retrospective teaching waits until an episode ends, then labels all time steps using observed future behavior to determine when help would improve progress.
5 Hierarchical Recurrent Architecture
The agent uses separate navigation and help-request networks with shared hierarchical components that encode language, task state, and episode history. Its intra-task state compares the current situation with similar past states before producing an action distribution.
- Architecture: The navigation and help-request policies are separate neural networks with text-encoding, inter-task, and intra-task components.The help-request network is mostly similar to the navigation network, but receives the navigation action distribution when computing state features.
- Architecture: The text-encoding component stores the hidden representation of the current language instruction in a text memory.
- Architecture: When the task changes because of a help request or route departure, the agent re-encodes the instruction and resets the inter-task state.
- Architecture: The intra-task module combines a tentative current state with weighted past states from nearly identical situations to represent the episode history.
- Architecture: The resulting state encodes context-sensitive dissimilarity from related past situations, supports curiosity-encouraging learning, and is passed through softmax to produce an action distribution.
6 Experimental Setup
HANNA evaluates object-finding tasks across seen and unseen environments using defined navigation, help-request, and task-efficiency baselines. Performance is measured by success, final-distance error, and SPL, which combines success with travel efficiency.
- Dataset: The dataset contains 289 object types and a 2,332-word instruction vocabulary, with shortest paths restricted to 5–15 locations.Tasks require approximately 9 to 32 meters of travel, based on an average edge length of 2.25 meters.
- Dataset: Evaluation includes SEENENV tasks in training environments and UNSEENALL tasks whose evaluation tasks and ANNA instructions were not encountered in those environments during training.
- Baselines: Navigation baselines include SHORTEST, RANDOMWALK, and FORWARD10, while help-request heuristics include NOASK, RANDOMASK, and ASKEVERY5.RANDOMASK requests help with probability 0.2, and ASKEVERY5 requests help after at least five walking steps.
- Metrics: Success rate measures solved tasks, navigation error measures final distance to the nearest goal, and SPL weights success by travel distance.
7 Results
HANNA remains challenging for heuristic agents, while assistance, language, learned help-request policies, and the proposed architecture improve performance across test settings. The curiosity-encouraging objective further reduces repeated mistakes and boosts success on unseen environments.
- Heuristic baselines achieve success rates below 7%, while the no-assistance agent succeeds on 17.21% of TEST SEENENV tasks and 8.10% of TEST UNSEENALL tasks.
- 71.16% and 39.35% success-rate gains over no help occur on TEST SEENENV and TEST UNSEENALL, respectively, when the agent leverages ANNA assistance.
- Language assistance improves success by +15.17% on TEST UNSEENALL versus +3.42% on TEST SEENENV.
- The learned help-request policy outperforms all baselines by at least 10.40% in TEST UNSEENALL success rate while predicting future progress and using agent uncertainty.
- The hierarchical recurrent model outperforms an LSTM-based encoder-decoder by +28.2% on TEST UNSEENALL.
- Curiosity-encouraging training reduces repeated looping and decisions, increasing TEST UNSEENALL success by +4.33% over training without that objective.
8 Conclusion
The paper presents a photo-realistic simulator for human assistance, imitation-learning techniques for requesting and interpreting assistance, and a hierarchical model for subtasks. It identifies more natural interaction, theoretical modeling, and real-world evaluation as future directions.
- HANNA is a photo-realistic simulator that mimics primary characteristics of real-life human assistance.
- The approach combines imitation learning for requesting and interpreting simulated assistance with a hierarchical neural network for representing subtasks.
- Future work includes more natural linguistic interaction, a theoretical framework for human assistance, and deployment and evaluation on real-world platforms.
A Proof of Lemma 1
The construction associates natural-language instructions with paths in an environment graph. It uses ancestor paths to obtain logarithmic instruction complexity between locations while requiring Θ(N log N) collected instructions.
- Θ(N log N) instructions are collected by storing forward and reverse instructions from each node to ancestors at exponentially increasing distances.
- A language-assisted path is a path associated with a natural-language instruction.
- O(log N) language-assisted paths suffice to traverse between a node and any ancestor under the construction.
- For arbitrary nodes, traversal proceeds through their least common ancestor, preserving O(log N) language-assisted paths.
B Proof of Lemma 2
The supplied passages describe retrospective help-request supervision and a memory-augmented architecture. The model combines navigation, visual, target-similarity, time, text, and attention-based memory features, with help-request training governed by configured objectives.
- Retrospective help-request teacher: The retrospective teacher uses the agent’s future trajectory and current navigation policy to determine whether requesting help would improve progress.
- Retrospective help-request teacher: When continuing navigation is expected to make progress, the teacher suggests doing nothing; otherwise it considers requesting help.
- Input features: Navigation actions combine panoramic visual features with orientation features, while the stop action maps to a zero vector.
- Input features: Target-similarity features compare current and target views using cosine similarity and retain the maximum similarity for each current view angle.
- Input features: Learned local and global time embeddings encode episode progress, generalize to unseen step counts, and support evaluation on longer episodes than training episodes.
- Memory architecture: The architecture uses Transformer-style attention and similarity-based attention to retrieve related past states for curiosity-encouraging training.
F Analysis
The analysis examines how curiosity-encouraging loss and help-request condition prediction affect seen and unseen environments. The results show a trade-off between memorization, generalization, and help-request behavior, alongside conservative but uneven condition prediction.
- Ablation study: Removing the curiosity-encouraging loss slightly improves memorization on VAL SEENENV but reduces generalization to unseen environments.
- Help-request behavior: +5.2% on VAL SEENENV and +6.4% on VAL UNSEENALL are the increases in uncertain wrong conditions without help-request condition prediction.
- Analysis: Jointly learning navigation and help-request policies is challenging because better navigation supplies fewer positive examples of situations requiring help.
- Help-request behavior: The final agent requests help at about 18% of time steps and follows a conservative help-request policy on TEST UNSEENALL.
- Help-request behavior: Condition prediction has high precision for lost and uncertain wrong conditions, at 76.9% and 86.6%, but recall remains below 50% for all conditions.
G Mini-batch Resetting
Mini-batch resetting can improve success rates by helping agents escape loops, but it makes evaluation nondeterministic when batch size exceeds one. Deterministic batch-size-one results are slightly lower while preserving the paper’s conclusions and qualitative claims.
- Mini-batch Resetting: Mini-batch resetting forces agents to forget past states, helping them escape looping situations and improving success rates by about 1-2%.The mechanism resets inter-task modules and local times for all agents in the same mini-batch.
- Mini-batch Resetting: Batch sizes greater than one make task behavior nondeterministic because agents share the mini-batch resetting mechanism.This dependency complicates direct comparison and evaluation of the methods.
- Mini-batch Resetting: Batch-size-one evaluation removes mini-batch-construction dependence, producing slightly lower results while retaining the conclusions and qualitative claims from the main evaluation.The paper recommends these deterministic results for future comparisons.
- Mini-batch Resetting: The reported test-split tables use batch size 1 for results with perfect assistance interpretation, different assistance types, and different help-request policies.These tables report success rates or results under deterministic evaluation.
- Mini-batch Resetting: The curiosity-encouraging-loss comparison evaluates navigation-mistake repetition and help-request repetition alongside model results on TEST UNSEENALL.The table compares the proposed model with and without the curiosity-encouraging loss and an LSTM encoder-decoder.