Source-linked AI summary
Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi
TL;DR
Visual navigation agents must generalize to unseen scenes, yet conventional models remain frozen during inference. The paper proposes SAVN, a meta-reinforcement learning method that learns a self-supervised interaction loss and adapts while navigating. In AI2-THOR, SAVN improves success rate and SPL over non-adaptive baselines and outperforms hand-crafted-loss and memory-augmented alternatives.
Problem
Visual navigation requires learning to adapt at both training and test time because generalization to unseen scenes is a central challenge.
Method
SAVN is a meta-reinforcement learning approach that learns a self-supervised interaction loss enabling adaptation during inference without explicit supervision.
Results
SAVN outperforms non-adaptive baselines in success rate and SPL, including 40.8 vs 33.0 success rate and 16.2 vs 14.7 SPL in AI2-THOR.
Takeaways & Limitations
The learned interaction loss supports more effective navigation than hand-crafted losses, while SAVN also outperforms a memory-augmented non-adaptive baseline.
Takeaways & Limitations
The authors note that producing a reliable function g may be difficult in non-synthetic environments, so g is used only in hand-crafted loss variations.
Abstract
from arXiv · showhide
Learning is an inherently continuous phenomenon. When humans learn a new task there is no explicit distinction between training and inference. As we learn a task, we keep learning about it while performing the task. What we learn and how we learn it varies during different stages of learning. Learning how to learn and adapt is a key property that enables us to generalize effortlessly to new settings. This is in contrast with conventional settings in machine learning where a trained model is frozen during inference. In this paper we study the problem of learning to learn at both training and test time in the context of visual navigation. A fundamental challenge in navigation is generalization to unseen scenes. In this paper we propose a self-adaptive visual navigation method (SAVN) which learns to adapt to new environments without any explicit supervision. Our solution is a meta-reinforcement learning approach where an agent learns a self-supervised interaction loss that encourages effective navigation. Our experiments, performed in the AI2-THOR framework, show major improvements in both success rate and SPL for visual navigation in novel scenes. Our code and data are available at: https://github.com/allenai/savn .
1. Introduction
The paper frames learning as continuous adaptation and applies this idea to visual navigation in unseen scenes. SAVN uses self-supervised interaction and meta-reinforcement learning to adapt during inference, improving navigation outcomes over non-adaptive approaches.
- Humans continue learning while performing tasks, whereas conventional deep learning models typically freeze during inference.
- Visual navigation must generalize to unseen scenes whose structures and object appearances were not observed during training.
- SAVN adapts during inference without explicit supervision by learning a self-supervised interaction loss for navigation.
- The agent modifies its network during both training and testing, unlike traditional reinforcement learning systems that freeze the network after training.
- 40.8 vs 33.0 success rate and 16.2 vs 14.7 SPL show SAVN outperforming the non-adaptive baseline in AI2-THOR.
2. Related Work
Prior navigation work includes map-based, end-to-end learning, target-driven, language-guided, and self-supervised approaches. This paper distinguishes SAVN by adapting dynamically to novel scenes through self-supervised meta-learning without labeled examples or an exploration phase.
- Deep Models for Navigation: Traditional navigation methods plan with environment maps, while learning-based methods perform localization, mapping, exploration, and semantic recognition end-to-end.
- Deep Models for Navigation: Existing approaches address target-driven navigation, joint mapping and planning, auxiliary navigation tasks, topological maps, and self-supervised deep reinforcement learning.
- Deep Models for Navigation: SAVN focuses on using meta-learning to navigate new scenes from only the target class label, without an exploration phase.
- Meta-learning: Unlike supervised meta-learning methods, SAVN adapts without supervision or labeled examples, using self-supervised interaction within the same domain.
- Meta-learning: Other meta-learning methods speed training through exploration or structured noise, whereas SAVN emphasizes self-supervised adaptation during a single visual navigation task.
- Meta-learning: Prior work considers adaptation to perturbations or learns objectives with different techniques, while SAVN combines visual navigation with self-supervised loss adaptation.
3. Adaptive Navigation
The section introduces adaptive visual navigation using meta-learning: an agent continually updates its policy from self-supervised interaction while navigating, rather than freezing after training.
- Training and Testing: During training, navigation gradients update the interaction-loss parameters, while test-time interaction gradients update the rest of the network with the loss parameters fixed.This separates learning the self-supervised objective from adapting the navigation network during inference.
- Task Definition: The navigation task uses egocentric RGB observations and a target object class, with success requiring a correct termination action near and visibly facing the target.Tasks are defined by a scene, target object class, and initial position, with disjoint training and testing scene sets.
- Learning to Learn: The method frames visual navigation as meta-learning, adapting the agent to new tasks through interaction during both training and inference.MAML motivates learning parameters that adapt quickly to novel tasks after gradient updates.
- Learning to Learn: SAVN learns a self-supervised interaction objective whose updates modify the policy network as the agent interacts with a scene.The objective can be learned or hand-crafted, but the paper’s main approach learns it for effective navigation.
- Learning to Learn How to Learn: The learned interaction loss uses temporal convolutions over previous hidden states and policies to produce a scalar objective for adaptation.The architecture uses two one-dimensional temporal-convolution layers, with 10×1 and 1×1 filters, followed by an ℓ2 norm.
- Learning to Learn How to Learn: The interaction loss is trained to align its gradients with supervised navigation gradients, enabling continued adaptation when navigation supervision is unavailable.The objective minimizes navigation loss while maximizing similarity between interaction and navigation gradients.
4. Experiments
Experiments in AI2-THOR evaluate SAVN against adaptive and non-adaptive baselines on unseen scenes using Success Rate and SPL. SAVN improves navigation effectiveness and efficiency, while analyses link its gains to fewer failed actions and interaction-gradient adaptation.
- Experimental setup: AI2-THOR experiments use 120 indoor scenes across kitchens, living rooms, bedrooms, and bathrooms, with separate training, validation, and testing scenes.Each room category contributes 20 training, 5 validation, and 5 testing scenes.
- Metrics: Success requires issuing Done when the target object is within 1 meter and visible; SPL measures success weighted by path length.Evaluation includes all trajectories and trajectories whose optimal path length satisfies L ≥5.
- Main results: The learned self-supervised loss outperforms all baselines by large margins in both success rate and SPL, improving about 8% absolute success and 1.5 SPL over non-adaptive A3C.The comparison includes random, nearest-neighbor, and non-adaptive baselines, plus hand-crafted self-supervised alternatives.
- Main results: Hand-crafted exploration losses improve success over baselines, but their SPL is less impressive than the learned loss.The authors hypothesize that minimizing these losses is less conducive to efficient navigation.
- Behavioral analysis: SAVN experiences significantly fewer failed actions as episodes progress and less frequently terminates or remains stuck than the non-adaptive baseline.A failed action is identified when two consecutive frames are sufficiently similar, typically after a collision.
- Ablation study: Using the learned objective with an added hand-crafted prediction loss outperforms the corresponding non-adaptive model by 3.3% on all trajectories and 4.8% for trajectories of length at least 5.SAVN also outperforms a memory-augmented baseline and remains superior when termination information is supplied by the environment.
5. Conclusions
SAVN adapts during both training and inference, using self-supervised interaction to improve visual navigation without explicit supervision.
- SAVN learns during both training and inference, unlike non-adaptive agents.
- The learned self-supervised interaction loss outperforms hand-crafted losses for navigation.
- SAVN outperforms non-adaptive baselines by a large margin and navigates more effectively than a memory-augmented non-adaptive baseline.
- The authors conjecture that self-supervised interaction-based adaptation may extend to other domains.