Source-linked AI summary
Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, Wolfram Burgard
TL;DR
The paper addresses sensor-based robot navigation without explicit localization, mapping, or planning, while seeking rapid adaptation across changing tasks. It introduces successor-feature deep reinforcement learning to transfer knowledge between related navigation problems. Experiments report faster learning after the first task while preserving the ability to solve earlier tasks.
Problem
The paper asks how robots can navigate from onboard sensory data without explicit localization, mapping, or planning while adapting quickly to changing goals and environments.
Method
The method extends successor-feature reinforcement learning to transfer learned representations across navigation tasks, including environmental and reward changes.
Results
The algorithm substantially reduces learning time after the first task and preserves the ability to solve earlier tasks after transfer.
Takeaways & Limitations
Successor features provide a compact way to retain encountered tasks while enabling fast transfer with minimal additional memory and computational requirements.
Abstract
from arXiv · showhide
In this paper we consider the problem of robot navigation in simple maze-like environments where the robot has to rely on its onboard sensors to perform the navigation task. In particular, we are interested in solutions to this problem that do not require localization, mapping or planning. Additionally, we require that our solution can quickly adapt to new situations (e.g., changing navigation goals and environments). To meet these criteria we frame this problem as a sequence of related reinforcement learning tasks. We propose a successor feature based deep reinforcement learning algorithm that can learn to transfer knowledge from previously mastered navigation tasks to new problem instances. Our algorithm substantially decreases the required learning time after the first task instance has been solved, which makes it easily adaptable to changing environments. We validate our method in both simulated and real robot experiments with a Robotino and compare it to a set of baseline methods including classical planning-based navigation.
I. INTRODUCTION
The paper targets sensor-based robot navigation without explicit localization, mapping, or planning, while requiring rapid adaptation across changing goals and environments. It proposes successor-feature deep reinforcement learning to transfer knowledge across related tasks and evaluates the approach in simulated and real robot settings.
- Autonomous navigation requires reaching a designated goal using only onboard sensor input, commonly through localization, mapping, planning, and control algorithms.
- The proposed approach frames navigation as reinforcement learning and seeks solutions without explicit localization, mapping, or path planning.
- Successor representation learning is used to support fast adaptation to new goals and environments while preserving solutions to earlier tasks.The method retains a compact representation of encountered tasks' Q functions with minimal additional computational costs.
- The method is evaluated in simulated and real-world maze environments using visual and depth inputs, with comparisons against planners, supervised imitation, and DQN transfer.
- Relations to existing work: DQN-based transfer is difficult because learned policy dynamics and reward associations are not clearly separated, complicating transfer while preserving the original policy.
- Relations to existing work: Successor-feature reinforcement learning partitions Q-learning into reward-predictive features and estimates of how those features evolve over time.
III. BACKGROUND
The paper formalizes navigation as a Markov Decision Process in which sensor observations and their recent history inform state, actions are selected under a policy, and rewards define value estimates. Bellman recursion supports Q-learning, SARSA, and greedy policy improvement.
- In the navigation MDP, the agent receives observations, selects actions, and obtains reward signals over a sequence of time steps.
- The state is represented by the current observation together with a history of recent observations, with history length H.
- The Q-function follows Bellman recursion through immediate reward plus discounted expected future Q-value under policy dynamics.
- Given a known Q-function, an improved policy greedily selects the action with maximum Q-value in each state.
- Deep neural networks provide powerful function approximators for applying these reinforcement-learning principles to control.
B. Successor feature reinforcement learning
Successor-feature reinforcement learning replaces direct black-box Q-function learning with a representation that separates reward prediction from feature evolution. The method learns features, reward weights, and successor features jointly or transfers them across tasks.
- Direct Q-function approximation creates a black-box representation that makes knowledge transfer between tasks challenging.
- The method approximates rewards as a linear combination of learned state features and reward weights.
- Successor features represent the discounted expected sequence of learned features under a policy from a given state and action.
- The feature mapping can be designed to represent the state, allowing successor features to use learned features and actions as inputs.
- The successor-feature formulation uses a Bellman equation in which the reward function is replaced by the learned feature representation.
- The reformulation separates learning into estimating feature expectations under policy dynamics and estimating reward obtainable in a state.
- For the optimal policy, the algorithm alternates stochastic gradient steps to learn the feature mapping, reward weights, and approximate successor-feature mapping.
- With stored successor features, transferring to a new task can require optimizing only the new task's successor-feature parameters or learning an additional reward-weight vector.
IV. TRANSFERRING SUCCESSOR FEATURES TO NEW
The paper uses successor features to transfer navigation knowledge across tasks while preserving earlier solutions, targeting both changed goals and changed maze environments. The method relies on shared feature structure and adds task-specific mappings or reward weights with limited extra storage.
- Task transfer: Successor features decouple task-specific reward estimation from estimating feature occurrences under policy dynamics, supporting transfer between related navigation tasks.This framework covers different goals in one maze and different environments with shared state-space similarities.
- Transfer procedure: The architecture uses a convolutional feature extractor, successor-feature networks, reward-weight regression, and task-specific mappings B_i between current and old task features.These components implement the alternating objectives for the current task and retained tasks.
- Task transfer: For tasks sharing an environment and transition dynamics, later tasks can reuse learned features and successor features while learning a new reward weight vector.Successor features may also need adaptation when policy-induced feature expectations change.
- Transfer procedure: The transfer procedure initializes the new task from the previous task, trains its features and successor features, preserves old feature relations with mappings B_i, and estimates old-task successor features under current-task features.Stored old successor-feature networks can alternatively support action selection for previous tasks.
- Memory and scope: The method has small overhead when successor features share one network or are small, requiring k−1 additional weight matrices and one reward vector per task.The shared-feature regression is expected to work well when tasks have shared structure; unrelated tasks may require more parameters and higher feature dimensionality.
- Intuition: The method’s intuition is that features can encode transferable spatial properties, while successor features capture their discounted accumulation under task-specific policy dynamics.Reward mappings can be relearned quickly or transferred directly when the reward relation remains appropriate.
A. Experimental setup
The experiments evaluate navigation transfer in simulated maze-like 3D environments using a noisy differential-drive robot with visual or depth observations. Training uses stochastic gradient optimization with shared minibatch settings and exploration alongside learning.
- Simulation environment: The simulated environment contains cubic obstacles and a target, with a Pioneer-3dx agent modeled by differential-drive dynamics and Gaussian control noise.The four discrete actions are standing still, turning left, turning right, and moving straight one meter.
- Rewards and episodes: The reward is −0.04 per step, −0.96 for obstacle collisions, and 1 for reaching the goal, forcing time-optimal behavior.Episodes begin from a random location and end when the goal is reached unless otherwise noted.
- Observations: Each step provides a forward-facing camera frame, and the state consists of the four most recently obtained observations.Visual observations are resized to 64 × 64 pixels.
- Optimization: Training uses ADAM with minibatches of 64, learning rates of 2.5 × 10−4 for supervised visual inputs and 2.5 × 10−5 for visual reinforcement learners.Depth-input reinforcement learning uses a learning rate of 5.0 × 10−5.
- Optimization: The study selects optimizer hyperparameters through a coarse grid search over learning rates from 1 × 10−6 to 1 × 10−3 and trains alongside exploration, processing one batch every four steps.The same minibatch size is used across the considered approaches.
B. Baseline method - supervised learning & DQN
The paper compares supervised action imitation and DQN with the proposed visual-navigation learner, using planner-derived labels and both scratch and transfer settings.
- Supervised learning: The supervised baseline predicts A∗ planner actions directly from visual input using a CNN.Its training data requires full localization, although evaluation does not.
- DQN: The DQN baseline is evaluated both from scratch and by fine-tuning a model trained on the base task.Transfer with fixed features is also evaluated for completeness.
- Supervised learning: The supervised learner uses 1.6e5 planner-labeled samples generated beforehand.The setup represents a best-case scenario for training a planner-imitating CNN in this domain.
- Evaluation protocol: Training comparisons scale supervised-learning steps so update counts match those of SF-RL and DQN.The reinforcement learners begin updating after 3e4 iterations and update every 4 steps.
C. Visual navigation in 3D mazes
The first experiment trains SF-RL, DQN, and the supervised learner on Map1 and periodically evaluates their current policies.
- Training protocol: All three learned methods are trained on the base maze Map1 before transfer experiments.Testing occurs every 10,000 training steps over 5,000 testing steps.
- Base environment: The supervised and reinforcement-learning models converge to performance comparable to the optimal A∗ planner on Map1.The supervised learner converges significantly faster because it receives optimal paths for starting positions covering the environment.
1) Base environment:
On the base environment and subsequent Map1-to-Map2 transfer, SF-RL is evaluated against planning, supervised learning, and DQN while retaining prior task knowledge.
- Transfer to different environment: Figure 6 compares average reward ± one standard deviation for A∗, supervised learning, DQN, and SF-RL across scratch and transfer conditions.The transfer conditions use Map1 and Map3 as source tasks.
- Transfer to different environment: SF-RL-Transfer reaches performance comparable to the optimal policy much faster than training from scratch.Its learning speed is comparable to the supervised learner trained on perfectly labeled actions.
- Transfer to different environment: SF-RL-Transfer adapts its features while preserving mappings to features learned for previous tasks.This differs from DQN-FixFeature, whose CNN features remain fixed during transfer.
- Evaluation: Table I reports final testing statistics over 50 random starting positions, with episode limits of 200 steps for Map1 and Map2.The table also uses a 500-step limit for Map3 and Map4.
- Transfer to different environment: When only environment dynamics change, transfer can retrain the successor-feature network while keeping the reward mapping fixed.With fixed or similar dynamics, adaptation can instead change the reward mapping alone or together with successor features.
D. More complicated transfer scenarios
The paper tests transfer between substantially different mazes and analyzes whether learned successor features encode robot pose information.
- More complicated transfer scenarios: Transfer from Map3 to Map4 changes object layouts substantially and moves the goal from an open area to a hidden corner.The results show a trend similar to the simpler maze transfer.
- More complicated transfer scenarios: After Map3-to-Map4 transfer, SF-RL-Transfer still solves the old task, whereas DQN-Finetune loses its Map3 policy.The difference is reported after the target and object locations change dramatically.
- Analysis of learned representation: The learned features can be regressed to recover robot poses from features extracted along optimal trajectories.The analysis compares ground-truth poses with predictions from a two-hidden-layer neural network on a held-out example.
- Realistic depth-input setting: Map5 is presented as a 3D maze in which the robot avoids colored object regions and navigates to a central traffic cone.The figure also compares simulated rendered depth images with real Kinect depth images at approximately matching poses.
- Analysis of learned representation: The analysis concludes that transition dynamics are encoded and that localization information can be reliably retrieved after training.This interpretation concerns the learned feature representation rather than an explicit localization module.
VI. REAL-WORLD EXPERIMENTS
Real-robot experiments transferred knowledge from simulated Map5 to real Map6 using depth inputs, substantially accelerating learning while achieving satisfactory performance after about 60,000 iterations.
- Real-world setup: The real-robot experiment collected depth images in Map6 with an onboard Kinect sensor, after changing object and target positions from simulated Map5.The setup used prerecorded images from explored locations to reduce prolonged training in the physical environment.
- Transfer comparison: SF-RL-Transfer used a base model trained on simulated Map5 and transferred it to real-world Map6, compared with SF-RL trained directly on Map6.This comparison is depicted in Fig. 8.
- Transfer comparison: Transfer produced a large learning speed-up despite simulated depth images lacking characteristic noise patterns in real Kinect data.The result compares transfer with training from scratch in the real environment.
- Results: 60,000 iterations yielded satisfactory performance, corresponding to approximately 8 hours of real experience at a 2 Hz collection rate.Learning in this experiment began after 10^4 steps.
- Results: After training on prerecorded images, the robot was tested in real environments including changed Maps 6 and 7.Map7 is mentioned as an additional changed environment but is not discussed in detail in the text.