Source-linked AI summary
Selective Experience Replay for Lifelong Learning
David Isele, Akansel Cosgun
TL;DR
Sequential learning with deep networks can forget earlier tasks because limited FIFO replay buffers lose prior experiences and training distributions drift. The paper augments FIFO replay with a selective long-term memory and compares four experience-selection strategies. Distribution matching consistently performs best across tested domains, while coverage maximization helps when less-trained tasks are more important.
Problem
Deep networks can forget previous tasks during sequential learning, while limited FIFO replay buffers lose earlier experiences and let the training distribution drift.
Method
The paper augments a short-term FIFO replay buffer with a selective long-term buffer and compares surprise, reward, distribution matching, and coverage maximization.
Results
Distribution Matching is consistently the top performer across tested domains, while Distribution Matching and Coverage Maximization prevent catastrophic forgetting in sequential intersection-task learning.
Takeaways & Limitations
Selective experience replay can prevent catastrophic forgetting when suitable selection algorithms are employed, with coverage maximization beneficial when less-trained tasks are more important.
Takeaways & Limitations
Distribution matching may underperform when task importance is not proportional to training time, particularly when less-trained tasks are more important.
Abstract
from arXiv · showhide
Deep reinforcement learning has emerged as a powerful tool for a variety of learning tasks, however deep nets typically exhibit forgetting when learning multiple tasks in sequence. To mitigate forgetting, we propose an experience replay process that augments the standard FIFO buffer and selectively stores experiences in a long-term memory. We explore four strategies for selecting which experiences will be stored: favoring surprise, favoring reward, matching the global training distribution, and maximizing coverage of the state space. We show that distribution matching successfully prevents catastrophic forgetting, and is consistently the best approach on all domains tested. While distribution matching has better and more consistent performance, we identify one case in which coverage maximization is beneficial - when tasks that receive less trained are more important. Overall, our results show that selective experience replay, when suitable selection algorithms are employed, can prevent catastrophic forgetting.
1 Introduction
Lifelong learning seeks systems that retain and reuse knowledge across sequential tasks, but deep networks can forget prior tasks as experience distributions shift. The paper proposes selective experience replay and compares strategies for preserving experiences under limited memory.
- Sequential task learning changes experience distributions and visited states, causing deep networks to lose performance on previous tasks.
- Deep reinforcement learning intensifies forgetting because FIFO replay buffers discard earlier or rarely occurring experiences.
- The paper develops online selective experience accumulation to retain prior experiences in long-term memory with limited storage.
- Four selection strategies are compared: surprise, reward, distribution matching, and coverage maximization.
- The paper evaluates selective replay across autonomous driving, grid-world navigation, and lifelong MNIST tasks.
2 Lifelong Machine Learning
Lifelong learning trains one system on sequentially encountered tasks while preserving prior knowledge. The paper frames limited-memory replay as a way to reduce bias from task-correlated experiences and the drift caused by FIFO buffers.
- Lifelong machine learning aims to preserve prior task knowledge and use it to improve learning on future tasks.
- The approach complements methods that preserve knowledge inside network parameters by selecting experiences for future training.
- Sequentially training a shared network creates task-correlated experiences relative to the global distribution.
- If prior tasks are forgotten, an estimator tracks the current task and never converges to the true global mean.
- With limited capacity, FIFO replay loses earlier experiences and causes the training distribution to drift over time.
- Selective long-term replay is proposed to preserve experiences and eliminate the non-stationary effects caused by FIFO replay.
3 Selective Experience Replay
Selective experience replay augments short-term FIFO storage with limited long-term episodic memory, using ranked selection to retain experiences that support learning across tasks. The paper compares surprise, reward, global distribution matching, and state-space coverage strategies, alongside a small FIFO buffer for incoming experiences.
- Long-term storage: Episodic memory stores a ranked subset of complete experiences online under resource constraints to preserve performance across tasks.It uses a priority queue, while the episodic memory remains complementary to the short-term FIFO buffer.
- Long-term storage: The selective replay process augments FIFO replay by preserving experiences in long-term memory and addressing the loss of earlier experiences under limited capacity.The short-term buffer continues refreshing so the network can train on incoming experiences.
- Storage trade-off: Training with long-term storage alone performs acceptably but does not match the performance of the combined long-term and FIFO approach.The authors report this result from an experiment described in the appendix.
- Selection strategies: Surprise ranks experiences by absolute TD error, while reward ranks them by the absolute value of their future discounted return.TD error is interpreted as the surprise of a transition; both strategies use priority queues for storage.
- Selection strategies: Global distribution matching uses reservoir sampling to maintain a random sample that approximately matches the distribution of all experiences seen so far.At time t, each experience has probability 1/t of being the maximum experience, regardless of when it was added.
- Selection strategies: Coverage maximization approximates a uniform visited-state distribution by replacing experiences with entries having the most neighbors within distance d.The method ranks samples by neighborhood density and can add entries in O(log(|E|)) time.
4 Experiments
The experiments evaluate selective experience replay in autonomous intersection handling and additionally test generalization in grid-world navigation and lifelong MNIST. The intersection study uses five tasks, a fixed forgetting-inducing order, and comparisons involving unlimited and limited FIFO buffers.
- Experimental domains: The primary domain is autonomous handling of unsigned intersections in the Sumo simulator, where tasks differ in difficulty and can require both transfer and adaptation.The driving setting includes competing objectives involving safety, goal-reaching time, and traffic disruption.
- Experimental domains: Five intersection tasks are evaluated: Right, Left, Left2, Forward, and Challenge, with the Challenge task crossing a six-lane intersection.The other tasks involve right turns, intersection crossing, or left turns across one or two lanes.
- Experimental setup: A random task ordering that demonstrates forgetting is fixed across experiments, and all experiments use dropout training.The paper notes that task order impacts learning and that dropout has been shown to help prevent forgetting.
- Experimental setup: The first experiment compares an unlimited-capacity FIFO buffer with a limited-memory FIFO buffer while sequentially training on five intersection tasks.Each task receives 10,000 training experiences, and success rate is evaluated periodically during training.
- Experimental domains: Additional tests use a four-room grid-world navigation task and a lifelong-learning variant of MNIST to assess results beyond the intersection domain and reinforcement learning.The MNIST agent sees two digits per task and must classify all 10 digits after five tasks.
5 Results
Limited-capacity FIFO replay forgets earlier tasks, whereas selective replay can preserve them; distribution matching is the most consistently effective strategy across the tested domains.
- 5.1 Baselines: Unlimited-capacity replay resists forgetting and can exceed single-task performance, establishing an upper bound for limited-memory methods.The network benefits from prior knowledge when training on new tasks.
- 5.1 Baselines: A FIFO buffer limited to 1000 experiences causes previous tasks to be forgotten as later tasks are trained.Training on the challenge task reduces performance on the single-lane tasks, and training on right harms other tasks.
- 5.2 Selective Replay Results: Surprise and reward selection exhibit catastrophic forgetting, making them poor strategies for retaining prior task knowledge.Surprise prioritizes uncertain or error-producing experiences, while reward-only storage ignores credit assignment.
- 5.2 Selective Replay Results: Distribution matching and coverage maximization prevent catastrophic forgetting, but distribution matching is slightly more stable and performs slightly better in this domain.Coverage favors rare events and outliers, whereas distribution matching better represents frequently visited regions.
- 5.3 Unbalanced Training: Coverage maximization retains better performance on a less-trained right task when challenge receives 12.5 times longer training.Both strategies perform about the same on challenge, while coverage retains more right-task experiences.
- 5.4 Grid World: In Grid World and MNIST, unselected replay rapidly destroys prior-task performance, while distribution matching generally preserves it best.Coverage helps in Grid World but performs more poorly on MNIST, likely because selecting an appropriate distance metric is difficult.
6 Conclusion
The paper retains experiences selectively in a long-term replay buffer alongside FIFO memory and compares four selection strategies. Distribution Matching and Coverage Maximization avoided catastrophic forgetting in sequential intersection tasks, while selective replay may also apply beyond lifelong learning.
- The method adds a long-term experience replay buffer to the short-term FIFO buffer and selects stored experiences using four strategies.The strategies are Surprise, Reward, Distribution Matching, and Coverage Maximization.
- In five sequential intersection tasks, Surprise and Reward still showed catastrophic forgetting, whereas Distribution Matching and Coverage Maximization did not.
- Selective experience replay can prevent catastrophic forgetting when suitable selection algorithms are used.
- The approach may also address limited storage and divergent behavior in single-task networks beyond lifelong learning.