Source-linked AI summary
Episodic Transformer for Vision-and-Language Navigation
Alexander Pashevich, Cordelia Schmid, Chen Sun
TL;DR
VLN agents must handle long compositional tasks and complex human instructions. The paper proposes E.T., which encodes language with the full visual-action episode history and uses synthetic instructions for pretraining and joint training. E.T. improves long-horizon task solving and reaches state-of-the-art ALFRED performance, with further gains from synthetic instructions.
Problem
VLN requires agents to execute highly compositional tasks and understand complex human instructions over long horizons.
Method
E.T. uses a multimodal transformer to encode language, full episode visual history, and previous actions, while synthetic instructions support pretraining and joint training.
Results
E.T. outperforms recurrent baselines on ALFRED, and synthetic instructions further improve performance, especially in unseen environments.
Takeaways & Limitations
Full-episode attention and synthetic intermediate representations together enable stronger performance on long-horizon, compositional VLN tasks.
Abstract
from arXiv · showhide
Interaction and navigation defined by natural language instructions in dynamic environments pose significant challenges for neural agents. This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. To improve training, we leverage synthetic instructions as an intermediate representation that decouples understanding the visual appearance of an environment from the variations of natural language instructions. We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance. Our approach sets a new state of the art on the challenging ALFRED benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test splits.
1. Introduction
VLN agents must execute long, compositional household tasks while grounding variable human instructions in limited egocentric observations. E.T. addresses these challenges with full-episode multimodal attention and synthetic instructions, achieving state-of-the-art ALFRED performance.
- VLN requires agents to ground human instructions in embodied perception and action while often executing long tasks from limited egocentric views.
- 53-action ALFRED examples require remembering a fireplace location across 31 timesteps and resolving object- and location-grounded coreferences.
- Recurrent architectures may struggle with long-term dependencies, motivating a transformer that combines camera observations, instructions, and previous actions across the entire episode.
- Synthetic instructions are proposed as an intermediate interface decoupling task specification from visual appearance and human-language variation.
- E.T. with synthetic instructions achieves a new state of the art on the challenging ALFRED benchmark.
2. Related work
Related work spans instruction-following agents, neural training methods, multimodal transformers, and semantic parsing. ALFRED extends vision-and-language navigation with longer sequences and object interaction, making it especially challenging.
- Instruction-following systems use structured commands, logic programs, natural language, target-state images, or combinations of these forms.
- VLN adds rich visual context, requiring agents to explore, perceive, and act from images or videos while following instructions.
- ALFRED is more challenging than other VLN datasets because episodes are longer, include object interaction, and require every step to succeed.
- Neural VLN agents are trained with imitation learning, reinforcement learning, or both, alongside auxiliary tasks intended to improve performance and generalization.
- Multimodal transformers unify language, vision, and action inputs, while semantic parsing converts natural language into machine-interpretable logic forms.
3. Method
E.T. predicts actions from language and the complete visual-action episode history using multimodal transformer encoders. Synthetic language supports encoder pretraining and joint training with natural-language demonstrations.
- 3.1. VLN background: VLN demonstrations pair a natural-language instruction with expert visual observations and expert action sequences, and the agent learns to approximate the expert policy.
- 3.1. VLN background: Unlike recurrent agents using a hidden state, full-episode agents receive all previous visual observations and actions directly when predicting the next action.
- 3.2. Episodic Transformer model: E.T. uses language, visual, action, and multimodal transformer encoders with temporal or positional encodings and causal attention over the episode history.
- 3.2. Episodic Transformer model: The visual encoder uses ResNet-50, the action encoder maps action types through a lookup table, and visual outputs feed a fully connected action predictor.
- 3.2. Episodic Transformer model: During training, E.T. predicts all demonstration actions and minimizes cross-entropy; during testing, it uses observations available up to the current timestep.
- 3.3. Synthetic language: Synthetic language is used for seq2seq encoder pretraining and for joint action training with natural-language demonstrations generated from expert path-planner annotations.
4. Results
Experiments on ALFRED show that full-episode attention improves performance on long-horizon tasks, while synthetic instructions strengthen training and generalization, especially in unseen environments.
- Comparison with recurrent models: 33.8% and 3.2% task success rates make E.T. relatively 45.6% and 33.3% better than the LSTM-based agent in seen and unseen environments.The comparison uses models trained with natural-language annotations only.
- Comparison with recurrent models: 6.5 average subgoals per task make long-term memory important for solving full tasks.E.T.'s advantage is larger for complete tasks than for individual subgoals.
- Comparison with recurrent models: Full episode observability and attention distinguish E.T. from recurrent models that explicitly observe only the last visual frame.The LSTM-based comparison controls for visual features and language encoder architecture.
- Training with synthetic annotations: 13.9% and 68.7% relative improvements in seen and unseen environments result from joint training with subgoal annotations.Speaker translation annotations improve over the no-joint-training baseline by 10.6% and 21.8%, respectively.
- Training with synthetic annotations: 37.8% and 228.1% overall relative improvements over the baseline E.T. model show that language-encoder pretraining and joint training are complementary.The gains are especially large in unseen environments.
5. Conclusion
E.T. combines full-episode multimodal history with synthetic-instruction training to improve vision-and-language navigation on ALFRED.
- E.T. is a transformer-based architecture for vision-and-language navigation.
- The model observes and encodes the full episode history of vision, language, and actions.
- Synthetic instructions support pretraining and joint training with human-annotated instructions.
- Training with synthetic instructions further improves performance in seen and especially unseen environments.
A. Appendix
The appendix adds analyses covering state-of-the-art comparisons, component-wise improvements, synthetic-data scale, and transformer attention visualizations.
- The appendix reports additional results and analysis beyond the main paper.
- It includes state-of-the-art comparisons using all evaluation metrics.
- It provides a complete breakdown of performance improvements and examines the impact of synthetic data size.
- It visualizes transformer attention maps.
A.1. Comparison with state of the art
The appendix compares E.T. with public-leaderboard methods across validation and test folds using full-task, subgoal, and path-length-weighted metrics.
- The comparison covers E.T. and state-of-the-art methods from the public leaderboard.
- Goal Cond. measures the average number of subgoal conditions completed for each task.
- Path-length-weighted scores weight each metric by the ratio of agent path length to expert path length.
- Success rates are reported on validation and test folds for full tasks and subgoal conditions.
A.2. Complete performance analysis
The appendix analyzes performance improvements by sequentially adding visual, masking, architectural, language-pretraining, and additional-data components to an LSTM baseline.
- The analysis begins with the LSTM-based baseline proposed by Shridhar et al.
- Object-detection visual features and pretrained MaskRCNN pixel-mask predictions significantly improve the original baseline.
- The component sequence then replaces the LSTM with E.T. and pretrains its language encoder on human-to-synthetic translation.
- Joint training with an additional 45K demonstrations achieves the state-of-the-art performance reported in Table A1.
A.3. Impact of synthetic demonstration size
Increasing synthetic demonstrations during joint training substantially improves E.T. performance up to 22K examples, while doubling them to 44K has only a minor additional effect.
- 22K synthetic demonstrations significantly improve performance over training with human annotations only.The improvement occurs during joint training with natural-language and synthetic instructions.
- 44K synthetic demonstrations provide only a very minor performance improvement beyond 22K.The authors nevertheless use 44K synthetic demonstrations in the main paper.
- Table A3 reports validation-fold success rates for joint training with different numbers of synthetic-instruction demonstrations.
A.4. Visualizing visual attention
E.T.’s multimodal transformer uses attention to revisit earlier visual observations, supporting task completion when objects, locations, or state changes must be remembered across long episodes.
- Attention rollout shows E.T. attending to previous visual frames while producing actions.The analysis averages attention across heads and recursively combines transformer-layer attention with skip connections.
- At timestep 19, E.T. attends to the microwave observed at timestep 8 to bring an apple there.
- At timestep 39, E.T. revisits frames from potato slicing at timesteps 17–18 to retrieve the sliced potato.
- When moving a second pan, E.T. attends to the earlier frame showing where the first pan was replaced.
- When retrieving a washed cloth, E.T. attends to the frame showing the washing action to track a visually subtle state change.
A.5. Visualizing language attention
Language-attention visualizations compare E.T. trained from scratch with a language-pretrained version, showing that pretraining can focus attention on instruction tokens needed for successful actions.
- Language-attention heatmaps compare E.T. trained from scratch with E.T. whose language encoder was pretrained.Red backgrounds indicate higher attention scores.
- The pretrained agent attends to “bat” and successfully finds the bat, unlike the non-pretrained agent.
- The pretrained agent attends to “refridgerator” and correctly cools the egg before heating it in a microwave.
- The pretrained agent focuses on “knife” and picks up the knife, whereas the non-pretrained agent picks up an incorrect fork.
- The qualitative examples indicate that language pretraining produces attention better aligned with human interpretation.
- The qualitative task examples include successful completion and failures involving object-reference confusion and navigation in unseen environments.