Source-linked AI summary
Memory Replay with Data Compression for Continual Learning
Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, Jun Zhu
TL;DR
Continual learning must address catastrophic forgetting, while existing memory-replay buffers hold few original samples and incompletely characterize old data. MRDC compresses replay samples and uses DPPs to select compression quality without repetitive training. With suitable quality, compression can improve memory replay, and the method is validated in class-incremental learning and continual object detection.
Problem
Memory replay is effective for continual learning, but buffers of a few original samples cannot fully characterize the old-data distribution.
Source: §1 · p. 1Method
MRDC compresses old training samples to fit more in memory and uses DPPs to select compression quality without repetitive training.
Source: §1 · p. 2Results
With properly selected quality, JPEG can match or outperform recent strong approaches with less extra computation and can further improve their performance.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that MRDC provides a baseline and a direction for continual learning, with validation in class-incremental learning and semi-supervised continual object detection.
Source: §6 · p. 9The approach assumes compressed samples approximately follow the original data distribution, an assumption that may be violated when compression is too strong.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Continual learning needs to overcome catastrophic forgetting of the past. Memory replay of representative old training samples has been shown as an effective solution, and achieves the state-of-the-art (SOTA) performance. However, existing work is mainly built on a small memory buffer containing a few original data, which cannot fully characterize the old data distribution. In this work, we propose memory replay with data compression (MRDC) to reduce the storage cost of old training samples and thus increase their amount that can be stored in the memory buffer. Observing that the trade-off between the quality and quantity of compressed data is highly nontrivial for the efficacy of memory replay, we propose a novel method based on determinantal point processes (DPPs) to efficiently determine an appropriate compression quality for currently-arrived training samples. In this way, using a naive data compression algorithm with a properly selected quality can largely boost recent strong baselines by saving more compressed data in a limited storage space. We extensively validate this across several benchmarks of class-incremental learning and in a realistic scenario of object detection for autonomous driving.
1 INTRODUCTION
Memory replay is an effective continual-learning approach, but small buffers of original samples can represent old data distributions poorly. MRDC uses compression to store more old samples and DPPs to select compression quality without repeated training.
1 INTRODUCTION
- Memory replay of representative old samples is effective and achieves state-of-the-art performance, but existing approaches typically use small buffers of original data.The paper argues that low storage efficiency means these buffers lose information about the old-data distribution; other approaches may add computation through parameter learning or feature distillation.Source: §1 · p. 1
- The authors draw inspiration from biological memory, which encodes old experiences compactly for replay and flexible retrieval.They connect this analogy to maintaining information in limited storage.Source: §1 · p. 1
- MRDC reduces old-sample storage costs so more training samples fit in a limited memory buffer.The paper presents this as a computationally efficient way to increase replay capacity.Source: §1 · p. 2
- DPPs select compression quality without repetitive training, addressing the costly and impractical search for a suitable quality.The quality choice balances the quantity and quality of replay data.Source: §1 · p. 2
- With a properly selected quality, JPEG can match or outperform recent strong approaches with less extra computation and can further improve their performance.The introduction describes these comparisons for ImageNet-sub in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORK
Prior continual-learning work addresses forgetting through regularization, architecture changes, or replay, while memory-replay research improves buffer construction and use. Other approaches compress or generate remembered data, and compression methods include hand-crafted and learned techniques.
2 RELATED WORK
- Continual-learning methods use regularization, dedicated architecture subspaces, or replay; memory replay generally performs best in class-incremental learning and other reported scenarios.The cited scenarios include audio, few-shot, semi-supervised, and unsupervised continual learning.Source: §2 · p. 2
- Memory-replay studies improve either buffer construction, as with Mnemonics and TPCIL, or buffer use, as with methods addressing old/new-class imbalance.iCaRL introduced a general memory-replay protocol.Source: §2 · p. 2
- A naive compression-quality grid search requires retraining with different qualities, making it costly and less applicable when old data cannot be revisited or future data are unavailable.This limitation is noted for empirical quality selection.Source: §2 · p. 2
- Generative replay and data compression seek more efficient retention, while feature-replay methods face limits from a forgetting feature extractor or a frozen extractor.The passage says generative-model learning is challenging, computationally costly, and limited to relatively simple domains; REMIND freezes most of its extractor after the initial phase.Source: §2 · p. 3
- Compression includes lossless methods, which reconstruct data perfectly but have limited compression rates, and lossy methods, which permit higher rates by degrading data.The passage names JPEG, WebP, and JPEG2000 as hand-crafted lossy approaches, alongside neural methods based on rate-distortion optimization.Source: §2 · p. 3
3 CONTINUAL LEARNING PRELIMINARIES
In class-incremental learning, a network learns tasks in sequence and must distinguish all previously seen classes at test time. Memory replay stores selected old samples in a small buffer for later tasks, though limited storage constrains how much of the old-data distribution it can retain.
3 CONTINUAL LEARNING PRELIMINARIES
- Class-incremental learning evaluates all classes seen so far together, requiring distinctions across tasks.Tasks provide task-specific training data, and previously seen training datasets are unavailable when learning a new task.Source: §3 · p. 3
- Replay selects representative old training samples for a small memory buffer and reuses them while learning a new task to help prevent forgetting.For classification, mean-of-feature selection chooses samples nearest each class's feature mean.Source: §3 · p. 3
- Because storage is limited, the buffer holds far fewer original samples than the training set, making it less effective at remembering the old-data distribution.The passage notes that prior efforts to exploit the buffer, such as alleviating class imbalance, do not remove this limitation.Source: §3 · p. 3
4 METHOD
MRDC uses feature-based DPP optimization to select compression qualities that balance memory-buffer quantity with fidelity to the old data distribution, matching grid-search choices while reducing computational cost by more than 100 times.
4.1 MEMORY REPLAY WITH DATA COMPRESSION
- MRDC compresses stored old training samples to increase their quantity in a limited memory buffer, aiming to better recover the old data distribution.The quantity–quality trade-off matters because excessively low quality can violate the assumption that compressed samples approximately follow the original distribution.Source: §4.1 · pp. 3–4
- An appropriate JPEG quality substantially outperforms original-data replay, while qualities that are too high or low hurt performance and the best quality varies by replay approach.The best-performing quality remains consistent across different numbers of incremental-phase splits.Source: §4.1 · p. 4, Figure 2 · p. 4
4.2 QUALITY-QUANTITY TRADE-OFF
- As compressed-data quantity rises and quality falls, compressed features initially track the original subset, but many low-quality samples become out-of-distribution as performance declines.The analysis visualizes feature distributions with t-SNE and reports that the compressed subset area eventually becomes much larger than the original subset area.Source: §4.2 · p. 5, Figure 3 · p. 5
- The method formulates quality selection as conditional DPP sampling and relaxes its optimization to maximize stored subset size while constraining feature-volume consistency.It searches a finite set of candidate qualities; five candidates were found to be a good choice in the reported experiments.Source: §4.2 · pp. 5–6
4.3 VALIDATE OUR METHOD WITH GRID SEARCH RESULTS
- Unlike grid search, which requires repetitive training and may depend on unavailable past or future data, MRDC selects quality using feature volumes without repetitive training.The supplied passage reports that grid search found best ImageNet-sub JPEG qualities of 50 for LUCIR and 75 for PODNet.Source: §4.3 · p. 6
- The method selects qualities 50 for LUCIR and 75 for PODNet, matching grid search while reducing computational cost by more than 100 times.With ϵ = 0.5, these are the smallest qualities satisfying |Rq − 1| < ϵ. Whether this condition holds generally agrees across each incremental phase and the average over phases; more dynamic cases are explored in Appendix D.4.Source: §4.3 · p. 7
5 EXPERIMENT
The experiments evaluate MRDC in class-incremental learning on large-scale images and in semi-supervised continual object detection. They compare it with memory-replay baselines under constrained storage and report accuracy and storage-space effects.
5.1 CLASS-INCREMENTAL LEARNING
- The class-incremental evaluation covers CUB-200-2011, ImageNet-full, and ImageNet-sub, with ImageNet-sub split into 5, 10, or 25 incremental phases.The evaluation reports averaged incremental accuracy with single-head evaluation; the default buffer budget is equivalent to 20 original images per class unless specified otherwise.Source: §5.1 · p. 7
- With the same extra storage, MRDC matches or outperforms AANets and DDE on the same backbones and further boosts their performance by a large margin.Table 1 reports averaged incremental accuracy for classification tasks.Source: §5.1 · p. 7, Table 1 · p. 8
- Across storage budgets equivalent to 10, 20, 40, or 80 original images per class, MRDC improves LUCIR and PODNet by a similar margin, generally more for larger numbers of phases.Figure 7 presents the effects of these storage budgets on ImageNet-sub.Source: Figure 7 · p. 8, §5.1 · p. 9
5.2 LARGE-SCALE OBJECT DETECTION
- On SODA10M, the selected compression quality is best in grid search, and MRDC generally yields several times the finetuning performance of replaying original data.The detection experiment uses five phases, each with 1K labeled and 20K unlabeled examples, and a buffer equivalent to 100 original images per phase.Source: §5.2 · p. 9, Figure 8 · p. 9, Table 2 · p. 9
6 CONCLUSION
The conclusion presents compression with a properly selected quality as a way to improve memory replay within limited storage. It also identifies further work on compression, distribution regularization, and DPP-based optimization in replay.
6 CONCLUSION
- The authors conclude that properly selected compression quality can improve memory replay by storing more compressed samples in limited space.They describe the approach as an under-explored baseline and a promising direction for continual learning.Source: §6 · p. 9
- Future work could develop adaptive compression, regularize distribution changes from compression, or apply DPP analysis to replay choices such as prototype selection.The authors also suggest that their approach to saving a batch of data to describe its distribution may motivate applications in data compression and data selection.Source: §6 · pp. 9–10
ETHIC STATEMENT
The statement describes MRDC as a way to reduce memory-replay storage requirements, while noting that negative consequences are not obvious at the current stage.
ETHIC STATEMENT
- MRDC reduces memory-replay storage requirements and may facilitate large-scale applications such as autonomous driving.The statement says negative consequences at the current stage are not obvious.Source: ETHIC STATEMENT · p. 10
A TRADE-OFF BETWEEN QUALITY AND QUANTITY
Under limited storage, compression quality trades off against the number of samples that fit in memory. Lower quality saves space but tends to distort data and hurt continual-learning performance, while semantic information is described as hardly affected.
A TRADE-OFF BETWEEN QUALITY AND QUANTITY
- With storage fixed, reducing compression quality allows more compressed data in the memory buffer, and vice versa.Figure 9 plots this relationship for ImageNet and SODA10M; quality 100 denotes original, uncompressed data.Source: A TRADE-OFF BETWEEN QUALITY AND QUANTITY · p. 14, Figure 9 · p. 14
- Lower compression quality tends to distort data and hurt continual-learning performance, although the semantic information is described as hardly affected.The paper presents compressed images at different JPEG qualities in Figure 10.Source: A TRADE-OFF BETWEEN QUALITY AND QUANTITY · p. 14, Figure 10 · p. 14
- Figure 9 plots the quality–quantity relationship under storage limits equivalent to 20 original images per class for ImageNet and 50 per phase for SODA10M.The plotted quality-100 setting refers to original data without compression.Source: Figure 9 · p. 14
B IMPLEMENTATION DETAIL
The section gives training and evaluation setups for class-incremental learning and object detection, then reviews DPPs’ subset, diversity, and computational properties.
B.1 CLASS-INCREMENTAL LEARNING
- Class-incremental experiments train ResNet-18 for 90 epochs and evaluate averaged incremental accuracy; ImageNet superclasses vary in semantic similarity.The similarity study uses 10 adjacent classes per superclass, constructing 10 adjacent or remote superclasses.Source: §B.1 · p. 15
B.2 LARGE-SCALE OBJECT DETECTION
- Object detection uses Faster-RCNN with FPN and a ResNet-50 backbone, training Pseudo Labeling and Unbiased Teacher for 10K iterations per incremental phase.Performance is evaluated with AP, AP50, and AP75.Source: §B.2 · p. 15
C.1 DETERMINANTAL POINT PROCESSES (DPPS) PRELIMINARIES
- DPPs use kernel-based subset probabilities; large pairwise similarity lowers the likelihood of co-selection, encouraging diverse subsets while balancing diversity against item quality.With a dot-product kernel, the determinant has a geometric interpretation as a squared volume.Source: §C.1 · pp. 15–16
- DPP computation can remain costly: evaluating the denominator sums over terms, and computing each determinant takes O(k^3) time.Sparse storage is possible, but determinant computation remains prohibitive unless sparsity is extreme.Source: §C.1 · p. 17
C.2 MODELING OUR CASE AS A CONDITIONAL N mb
The paper models selection of compressed training data as a conditional DPP whose set size varies with compression quality. It then casts its two optimization goals in terms of subset volume and a feature-volume ratio.
C.2 MODELING OUR CASE AS A CONDITIONAL N mb
- Unlike standard k-DPPs with fixed set size, the proposed conditional N_mb^q-DPP determines the subset cardinality through compression quality while seeking a maximum-volume subset.Its input is the training dataset, and the selected subset represents it in feature space.Source: §C.2 · p. 17
C.3 THE FIRST GOAL
- The first goal—maximizing the conditional probability of the selected subset—is equivalent to maximizing its cardinality because that probability increases with set size.The paper states that the probability is maximized when all n items are selected.Source: §C.3 · pp. 18–19
C.4 THE SECOND GOAL
- Under Assumption 1, optimizing compression quality can be converted to minimizing |Rq − 1|; experiments indicate this discrepancy is minimized without compression.The volumes of compressed and source feature subsets are empirically reported to match only at q = sup Q.Source: §C.4 · pp. 19, 21
D.1 T-SNE VISUALIZATION OF NORMALIZED FEATURES
These analyses examine MRDC’s compression-quality choices and memory-budget effects. They report reduced forgetting and stronger fixed-budget performance, while showing that dynamic quality can suffer from data imbalance.
D.1 T-SNE VISUALIZATION OF NORMALIZED FEATURES
- In the feature visualization, increasing the original-subset quantity improves coverage of training data, while lower quality with more compressed samples distorts features out of distribution.Figure 12 visualizes original subsets and all training data, with five latest-task classes distinguished by color.Source: §D.1 · p. 22, Figure 12 · p. 22
D.2 Rq FOR 5-, 10- AND 25-PHASE IMAGENET-SUB
- For LUCIR and PODNet, the selected qualities are 50 and 75, respectively, and remain the same across 5-, 10-, and 25-phase ImageNet-sub splits.These selections agree with grid-search results.Source: Figure 13 · p. 22, §D.2 · p. 23
D.3 AANETS AND DDE
- For LUCIR with AANets or DDE, selected qualities are consistent across incremental phases and match LUCIR’s quality.The analysis uses 5-phase ImageNet-sub.Source: §D.3 · p. 23, Figure 14 · p. 23
D.4 STATIC VS DYNAMIC QUALITY
- Dynamic quality and quantity cause severe data imbalance, and perform far worse than static qualities of 50, 75, or 90.The authors identify alleviating imbalance in highly dynamic scenarios as future work.Source: §D.4 · p. 24
E.1 COMPUTATIONAL COST
- The computational-cost subsection compares LUCIR, LUCIR with AANets, LUCIR with DDE, and LUCIR with MRDC on CUB-200-2011 and ImageNet-sub.Each baseline is run on one Tesla V100.Source: §E.1 · p. 24, Table 3 · p. 24
E.2 STORAGE SPACE
- The storage-space analysis reports averaged incremental accuracy on ImageNet-sub for memory budgets equivalent to 10, 20, 40, and 80 original images per class.LUCIR and PODNet results are reproduced from their officially released codes.Source: Table 4 · p. 24, §E.2 · p. 24
E.3 AVERAGED FORGETTING
- Averaged forgetting of initial-phase classes is largely alleviated by MRDC on each backbone.The evaluation uses storage equivalent to 20 original images per class.Source: §E.3 · p. 24, Table 5 · p. 25
E.4 FIXED MEMORY BUDGET
- Under fixed budgets of 1000 or 2000 original images, MRDC substantially boosts each evaluated backbone on ImageNet-sub.The comparison covers LUCIR, LUCIR with AANets, LUCIR with DDE, and LUCIR with MRDC.Source: Table 6 · p. 25, §E.4 · p. 25
E.5 LESS COMPRESSED SAMPLES
On ImageNet-sub, LUCIR with quality-50 compression can use more samples than the original-image replay baseline within equivalent storage, while retaining a performance advantage at lower sample counts.
E.5 LESS COMPRESSED SAMPLES
- 85 quality-50 compressed images outperform 20 original images; even 40 compressed images outperform 20 originals while saving 52.94% average memory.Reducing the compressed-sample count from 85 to 70, 55, and 40 also lowers accuracy.Source: §E.5 · p. 25, Table 7 · p. 25