Source-linked AI summary
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
Xin Wang, Wenhan Xiong, Hongmin Wang, William Yang Wang
TL;DR
Vision-and-language navigation remains difficult because model-free reinforcement learning overlooks environment dynamics and generalizes poorly beyond commonly used synthetic settings. The paper proposes RPA, a hybrid model-free/model-based reinforcement learning approach with look-ahead planning through an environment model. RPA achieves the best reported results on the real-world Room-to-Room dataset and improves transfer to unseen environments.
Problem
Model-free reinforcement learning for vision-and-language navigation does not consider environment dynamics, requires substantial training data, and is commonly evaluated in synthetic rather than real-world environments.
Method
RPA combines model-free and model-based reinforcement learning, using an environment model to simulate future visual perceptions and support look-ahead planning.
Results
RPA significantly outperforms baselines and achieves the best results on the real-world Room-to-Room dataset, with stronger transfer to unseen environments than model-free methods.
Takeaways & Limitations
The approach demonstrates that combining model-free control with model-based look-ahead can address model-free reinforcement learning’s generalization issue in unseen scenes.
Takeaways & Limitations
The Room-to-Room dataset has limited action diversity because upward and downward camera movements are rare in its trajectories.
Abstract
from arXiv · showhide
Existing research studies on vision and language grounding for robot navigation focus on improving model-free deep reinforcement learning (DRL) models in synthetic environments. However, model-free DRL models do not consider the dynamics in the real-world environments, and they often fail to generalize to new scenes. In this paper, we take a radical approach to bridge the gap between synthetic studies and real-world practices---We propose a novel, planned-ahead hybrid reinforcement learning model that combines model-free and model-based reinforcement learning to solve a real-world vision-language navigation task. Our look-ahead module tightly integrates a look-ahead policy model with an environment model that predicts the next state and the reward. Experimental results suggest that our proposed method significantly outperforms the baselines and achieves the best on the real-world Room-to-Room dataset. Moreover, our scalable method is more generalizable when transferring to unseen environments.
1 Introduction
Vision-and-language navigation requires embodied agents to map changing visual observations and natural-language instructions to actions in realistic environments. RPA addresses limitations of model-free reinforcement learning by combining model-free control with model-based look-ahead planning and achieves the best reported R2R results with stronger transfer to unseen environments.
- Task: Vision-and-language navigation asks an embodied agent to follow natural-language instructions through realistic indoor environments using changing first-person visual observations.The agent must select actions while interacting with the environment rather than processing fixed visual and linguistic inputs.
- Motivation: Model-free reinforcement learning directly maps observations to actions but does not model environment dynamics, requires substantial data, and is commonly evaluated only in synthetic environments.These settings simplify noisy visual and linguistic perception compared with real-world navigation.
- Motivation: Model-based reinforcement learning can simulate future states and perform multi-step look-ahead planning, reducing reliance on trial-and-error in the real environment.The approach is motivated by humans’ ability to imagine future observations and plan before acting.
- Approach: RPA combines model-free and model-based deep reinforcement learning for vision-and-language navigation by using an environment model to simulate future visual perceptions.The agent selects actions from both current real observations and planned future observations.
- Results: RPA significantly outperforms baselines, achieves the best results on the real-world Room-to-Room dataset, and transfers better to unseen environments than model-free methods.The reported contribution explicitly targets the generalization problem of model-free reinforcement learning in unseen scenes.
2 Related Work
Prior vision-language navigation work connects language to navigation actions but largely evaluates simplified synthetic settings. Model-based reinforcement learning offers planning through learned environment models, while RPA adapts hybrid model-based and model-free reinforcement learning to real-world vision-language navigation.
- Vision, Language and Navigation: Vision-language grounding connects language semantics to physical environment properties, while vision-and-language navigation requires sequential interaction with the environment.Navigation is more task-driven than visual question answering because the agent must act over time.
- Vision, Language and Navigation: Earlier robot-navigation methods often rely on prior global maps or build maps online, whereas vision-and-language navigation uses natural language to specify an unannotated goal without requiring a global map.This makes the navigation setting more challenging than map-based approaches.
- Vision, Language and Navigation: Existing navigation methods often operate in synthetic environments with simple discrete observations or unrealistic top-down views.These design choices differ from realistic visual perception in embodied navigation.
- Model-based Reinforcement Learning: Model-based reinforcement learning learns environment models for planning but can suffer from mismatch between learned and real environments when the simulator is unavailable.Hybrid approaches combine model-free and model-based reinforcement learning to address this issue; RPA applies that combination to real-world vision-language navigation.
3 Method
The method combines a model-free recurrent policy with a model-based look-ahead path that predicts future states and rewards, then uses both paths for action selection. It trains the environment model first and the enhanced policy model second, with attention-based language grounding and reward-based supervision.
- Task Definition: The agent receives an initial pose and language instruction, then selects actions sequentially to reach the language-specified target position.The task evaluates navigation using success rate and final navigation error.
- Overview: The model-free path encodes language and visual state information with a recurrent policy, while the model-based path performs multiple look-ahead processes.The action predictor combines information from both paths to generate a probability distribution over actions.
- Look-Ahead Module: The look-ahead module rolls forward imagined actions through an environment model that predicts each next abstract state and reward.An LSTM encodes the predicted states and rewards along each trajectory, and multiple trajectory representations are concatenated for action prediction.
- Environment Model: The environment model projects the current state and action before separate transition and reward networks produce the next state and scalar reward.The projection function concatenates state and action features; the transition and reward functions are learnable neural networks.
- Recurrent Policy Model: The recurrent policy uses attention-based LSTM decoding to emphasize instruction words relevant to the current action.The decoder considers word features, environment state, previous action, and its previous hidden state; its output and context vector feed the action predictor.
- Learning: Training first learns and fixes the environment model, then learns the enhanced policy model with the look-ahead module.Policy learning combines supervised imitation of demonstration actions with reward supervision based on distance to the target.
4 Experiments
Experiments compare RPA with supervised and model-free reinforcement-learning baselines on R2R, evaluate environment-model learning and reward designs, and analyze an example trajectory. RPA achieves the best reported results, with especially large gains on unseen environments.
- Experimental Settings: The R2R experiments use validation and test splits, including seen and unseen environments, and report TL, NE, SR, and OSR.R2R contains training, seen-validation, unseen-validation, and test sets; unseen splits contain environments absent from training.
- Results and Analysis: RPA further boosts performance over XE and model-free RL, achieving the best results consistently across the reported metrics.The comparison includes Teacher-forcing, Student-forcing, XE, model-free RL, and RPA.
- Results and Analysis: 14.5%: RPA’s relative success rate improvement on Test over XE, compared with 6.7% on Val Seen and 15.5% on Val Unseen.The improvement is larger on unseen validation than on seen validation, while model-free RL provides only a small unseen-set boost.
- Learning Curves of the Environment Model: Around 500 iterations: the environment model’s transition and reward losses converge to stable points, although reward prediction is noisier.The reward curve is noisier because rewards are sparse and high variance within trajectory samples.
- Reward Analysis: Discounted reward functions outperform Global Distance and Success rewards, while adding success information further improves performance.The final model uses Discounted & Success reward, combining a binary success reward with the final action’s immediate reward.
- Case Study: The illustrated RPA trajectory contains 11 actions from starting position (1) to destination (11), with Up and Down appearing rarely.The observed action distribution indicates a limitation of the R2R dataset.
5 Conclusion
The paper presents RPA as a strong and scalable approach for vision-and-language navigation that improves performance on unseen scenes. It uses look-ahead simulation and imagined trajectories to address model-free reinforcement learning’s generalization issue.
- Conclusion: Experiments show that RPA achieves superior performance and tackles the generalization issue of model-free RL on unseen scenes.The conclusion also describes RPA as more scalable than model-free agents because it incorporates imagined trajectories.
A Error Analysis
The analysis identifies three recurring navigation errors: out-of-vocabulary words, ambiguous instructions in complex unseen scenes, and accumulated mistakes during trajectories.
- Out-of-vocabulary words: Agents fail to interpret out-of-vocabulary instruction words, such as “pottery,” causing incorrect visual identification and actions.The paper suggests external knowledge as a possible remedy.
- Ambiguous instructions: Ambiguous descriptions in complex unseen environments can cause agents to select the wrong visual referent and execute incorrect actions.In one example, the instruction refers to the larger painting, but paintings appear on every wall.
- Error accumulation: A single wrong action can change the agent’s view enough to trigger a sequence of further wrong decisions.After the scene changes, the original instruction may no longer be correct for the agent’s new situation.
B Network Architecture
The architecture combines language encoding, recurrent action prediction, environment simulation, trajectory encoding, and final action prediction for planned-ahead navigation.
- Language encoder: The language encoder uses word embeddings, an LSTM, a linear layer, and tanh to produce instruction features.Its embedding dimension is 256 and its LSTM hidden size is 512.
- Recurrent policy model: The recurrent policy uses action embeddings, an LSTM decoder, attention, and projection to represent or predict navigation actions.In the model-free path, it passes the decoder and context representation directly rather than applying the projection module.
- Environment model: The environment model projects visual and action inputs, predicts transitions, and estimates rewards.It contains separate projection, transition, and reward functions.
- Model-based path: The trajectory encoder summarizes predicted states and rewards, and concatenated look-ahead encodings form the model-based path representation.The trajectory encoder is an LSTM with size 256.
- Action predictor: The action predictor is a multilayer perceptron that maps the combined representation to probabilities over six actions.Its final layer has six outputs followed by SoftMax.
C Training Details
The reported training configuration uses validation-tuned hyperparameters, Adam optimization, regularization, gradient clipping, and discounted cumulative rewards.
- Configuration: Hyperparameters for RPA-agent and environment-model training are tuned on the validation set.The paper reports settings for both training procedures.
- RPA agent: RPA-agent training uses batch size 100, up to 40,000 iterations, Adam with learning rate 1e-4, gradient clipping at norm 5, weight decay 0.0005, and dropout 0.5.The discounted factor for cumulative reward is 0.95.
RPA Agent Hyperparameter Setting
Environment-model training uses a separate optimization configuration and a weighted transition-plus-reward loss that stabilizes after roughly 500 iterations.
- Environment model: Environment-model training uses batch size 100, Adam with learning rate 1e-5, weight decay 0.0005, and dropout 0.5.The final loss combines transition and reward losses with weights 1 and 0.001.
- Convergence: The transition and reward losses converge to a stable point after around 500 iterations.This convergence is reported for the environment-model training losses.
Environment Model Hyperparameter Setting
Figure 7 presents error cases involving an out-of-vocabulary word, a complex unseen environment, and an ambiguous instruction.
- One error case involves the out-of-vocabulary word pottery.
- Another error case combines a complex unseen environment with an ambiguous instruction.
- Figure 7 is referenced for the error-case examples, with further explanation in Section A.