Source-linked AI summary
Latent Replay for Real-Time Continual Learning
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, Davide Maltoni
TL;DR
Continual learning at the edge is constrained by catastrophic forgetting and limited computation and storage. The paper introduces Latent Replay, which stores intermediate activations and slows lower-layer learning to keep them usable. Combined with continual-learning methods, it reaches state-of-the-art performance on CORe50 and OpenLORIS and supports near-real-time smartphone learning.
Problem
Edge continual learning must handle catastrophic forgetting while operating within limited computation and storage.
Method
Latent Replay stores activations at an intermediate layer and slows learning below that layer while leaving upper layers free to learn.
Results
Latent Replay combined with continual-learning methods achieves state-of-the-art performance on CORe50 and OpenLORIS and enables near-real-time smartphone learning.
Takeaways & Limitations
The computation-storage-accuracy trade-off can be selected for the application and available resources, including CPU-only edge devices.
Takeaways & Limitations
For applications with thousands of classes, storing latent activations may make external memory an issue.
Abstract
from arXiv · showhide
Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named "Latent Replay" where, instead of storing a portion of past data in the input space, we store activations volumes at some intermediate layer. This can significantly reduce the computation and storage required by native rehearsal. To keep the representation stable and the stored activations valid we propose to slow-down learning at all the layers below the latent replay one, leaving the layers above free to learn at full pace. In our experiments we show that Latent Replay, combined with existing continual learning techniques, achieves state-of-the-art performance on complex video benchmarks such as CORe50 NICv2 (with nearly 400 small and highly non-i.i.d. batches) and OpenLORIS. Finally, we demonstrate the feasibility of nearly real-time continual learning on the edge through the deployment of the proposed technique on a smartphone device.
1. Introduction
The paper targets continual learning on resource-constrained edge devices, where catastrophic forgetting, computation, and storage limit adaptation. It introduces Latent Replay to reduce these costs while narrowing the accuracy gap to cumulative training.
- Edge training is desirable for privacy, connectivity, and rapid adaptation, but model memory and computation often make it infeasible.
- CORe50 NICv2 uses small, highly correlated, non-i.i.d. batches while incrementally introducing instances and classes.In NICv2-391, each batch contains 300 frames from a short video of one moving object.
- Existing rehearsal-free approaches performed well but retained an accuracy gap of about 20% versus cumulative training.
- AR1* could not effectively adapt representation layers in the most challenging NICv2-391 setting.
- The paper proposes efficient continual learning intended to reduce the cumulative-training gap and enable nearly real-time edge training.
- Latent Replay injects old data at an intermediate layer, avoiding the full computation of repeatedly replaying input patterns.
- Experiments report state-of-the-art performance on CORe50 and OpenLORIS, alongside near-real-time incremental learning on an Android smartphone.
2. Related Works
Related work covers continual learning for incremental object recognition, rehearsal methods, streaming approaches, and edge training. The paper positions Latent Replay against these approaches in challenging robotic-vision settings.
- CORe50 NICv2 addresses single-task incremental learning of new instances and classes for robotic vision.
- CWR* freezes all but the final layer after the first batch, whereas AR1* enables end-to-end continual training.
- Pattern replay retains representative old information while new concepts are learned, with iCaRL providing a class-balanced herding strategy.
- Streaming continual learning updates models one pattern at a time and can be applied to the NIC setting.
- Deep Streaming LDA uses a fixed deep feature extractor and was evaluated on NICv2 alongside other techniques.
- Prior edge training required minutes on a Jetson TX2 and used few large batches, preventing direct comparison with this classification study.
3. Native Rehearsal
Native rehearsal augments continual-learning batches with stored patterns, improving accuracy and narrowing the gap to cumulative training. Larger memories help, but 1,500 patterns appear to offer a practical trade-off on this dataset.
- 3. Native Rehearsal: Native rehearsal adds stored patterns to each training batch through a simple external-memory procedure.The memory is updated by replacing a random subset with randomly selected patterns from the current batch.
- 3. Native Rehearsal: 1.27% of the total training set in a 1,500-pattern memory substantially improves CWR* and AR1* accuracy and reduces their gap to the ∼85% cumulative upper bound.The comparison uses MobileNetV11 on CORe50 NICv2 – 391.
- 3. Native Rehearsal: Increasing RMsize from 500 to 3,000 patterns improves accuracy for all evaluated algorithms, with diminishing gains between 1,500 and 3,000.The tested memory sizes are 500, 1,000, 1,500, and 3,000 patterns.
- 3. Native Rehearsal: AR1*free performs slightly better than AR1* with sufficient rehearsal, but performs worse with RMsize = 500 or without rehearsal.AR1*free removes Synaptic Intelligence protection from lower-layer weights.
- 3. Native Rehearsal: 5% worse than the cumulative upper bound, AR1*free with 3,000 patterns is the best Figure 4 combination.The authors note that better rehearsal-memory parametrization and exploitation could further reduce this gap.
4. Latent Replay
Latent Replay stores intermediate activations instead of raw past inputs and slows learning below the replay layer to preserve their validity. Layers above that point remain free to adapt at full pace.
- 4. Latent Replay: Latent Replay stores activation volumes at an intermediate layer rather than raw input patterns in external memory.The intermediate location is called the Latent Replay layer.
- 4. Latent Replay: Learning is slowed in all layers below the Latent Replay layer so stored activations remain valid while higher layers learn at full pace.The approach addresses activation aging caused by changes in representation layers.
- 4. Latent Replay: When lower layers are completely frozen, Latent Replay is functionally equivalent to input rehearsal while requiring less computation and storage.The savings arise because fewer patterns traverse the entire network and network representations typically compress information.
5. Experiments and Results
Experiments evaluate Latent Replay with continual-learning methods across CORe50 NICv2-391 and OpenLORIS, emphasizing computation–storage–accuracy trade-offs and benchmark performance. Results identify layer placement and activation storage as key design choices, with strong efficiency and competitive accuracy.
- Experimental setup: Experiments use MobileNetV1 with CWR*, AR1*, AR1*free, and LWF, comparing Latent Replay against other continual-learning approaches on CORe50 and OpenLORIS.For CWR*, replay uses pool6; AR1* and AR1*free allow the replay layer to vary, while AR1*free disables Synaptic Intelligence regularization.
- Experimental setup: Replay mini-batches preserve a fixed ratio of current and stored patterns, so only the current-pattern portion traverses the lower network layers.With 300 current patterns and 1500 stored patterns, a mini-batch of 128 contains 21 current and 107 replay patterns.
- Latent Replay design: Blocking representation-layer weight changes after the first batch while adapting batch-normalization moments provides the chosen learning slow-down strategy.This permits skipping the lower-network backward pass for native patterns because batch-normalization moment updates require only forward computation.
- Computation, storage, and accuracy trade-off: Pushing the replay layer downward improves accuracy until conv5_4/dw, after which performance saturates; the remaining approximately 4% gap to native rehearsal is attributed to activation aging.An intermediate method that feeds replay patterns from the input and stops backpropagation at conv5_4 reaches accuracy close to native rehearsal.
- Computation, storage, and accuracy trade-off: 32% computation, 66% storage, and a 5.07% accuracy drop make conv5_4/dw an efficiency–accuracy trade-off, while pool6 costs 0.027% and improves accuracy about 4%.The pool6 comparison is approximately 60% accuracy with replay versus approximately 56% without rehearsal; the 5.07% figure is the accuracy drop relative to native rehearsal for conv5_4/dw.
- Activation storage: Activation storage may become problematic with thousands of classes, although sparsification, quantization, and encoding can reduce storage with little accuracy reduction.The CORe50 example stores 1,500 conv5_4/dw activations requiring 48 MB; preliminary sparsification experiments target that layer using an L1 loss term.
- Benchmark comparisons: AR1*free with conv5_4/dw is reported as a practical efficiency–efficacy trade-off, reaching approximately 13% distance from the cumulative upper bound; iCaRL required 14 days versus less than 1 hour for other techniques.On OpenLORIS, the proposed approach ranked slightly below the top entry while offering significantly better inference time, replay memory, and model size; training efficiency was not measured by the challenge criteria.
6. Real-World Deployment on Embedded Devices
The CORe Android app demonstrates near-real-time continual learning on a smartphone. It supports updating existing classes and adding new ones using Latent Replay with CPU-only inference and training.
- Deployment: The Android CORe app demonstrates continual learning at the edge on a smartphone.The app is designed for incremental learning of new objects or poses and new classes.
- Inference: The app classifies framed objects at about 5 fps using CPU-only inference without hardware acceleration.Its interface returns the top three categories and highlights the most likely class with a green frame.
- Training: A 20-second video captured at 5 fps yields 100 frames, and continual learning completes in less than 1 second after acquisition.Training is triggered through the application interface for existing or initially empty classes.
- Implementation: The smartphone implementation uses MobileNetV1 with Latent Replay at pool6 and an external memory of 500 patterns.The Android implementation uses customized Caffe, C++ low-level code, and a Java interface.
7. Conclusions
Latent replay supports continual learning of new classes and instances from small, non-i.i.d. batches while keeping accuracy close to the cumulative upper bound. Its resource trade-off also enables continual learning on GPU-free edge devices, demonstrated through an Android application.
- Latent replay continually learns new classes and new instances of known classes from small, non-i.i.d. batches.
- About 5% accuracy gaps from the cumulative upper bound were achieved in some cases.
- The computation-storage-accuracy trade-off can be adjusted to the target application and available resources.
- An Android application demonstrated continual learning from short videos on edge devices without GPUs.
A. Implementation and Experiments Details
The experiments averaged test-accuracy curves over five runs, varying the order of encountered batches. They used the same experimental setup as the referenced benchmark and a customized Caffe environment with a single GPU.
- 5 different runs were averaged to obtain each test accuracy curve.
- Each run differed by the order of the encountered batches under the experimental setup from reference [15].
- Experiments ran in an Ubuntu 16.04 Docker environment using customized Caffe and a single GPU.
B. Hyperparameters
The experiments documented shared naming and hyperparameter choices, with selected differences between rehearsal types and AR1* variants. Latent rehearsal used a higher BatchReNorm moving-average update rate than native rehearsal.
- The experiments used the naming scheme from reference [17].
- AR1* and AR1*free used a higher learning rate for the CWR layer as described in reference [15].
- Different BatchReNorm moving-average update rates were selected for native rehearsal and latent replay to optimize each rehearsal type.
- A higher update rate was better suited for the latent version.
- Except for the update rate, native and latent rehearsal-based experiments used the same hyperparameters.
C. Model Architecture and Memory Trade-off
The study assessed computation, accuracy, and memory trade-offs by running AR1*free with different latent replay layers. The model was based on MobileNetV1 with Batch Norm replaced by Batch ReNorm.
- AR1*free was evaluated with different latent replay layers to assess accuracy, computation, and memory usage trade-offs.
- The model was based on MobileNetV1, with Batch Norm layers replaced by Batch ReNorm layers.
- The architecture details included pattern size and the number of weights per layer.
D. Android Application Setup and Performance
The Android deployment uses a OnePlus 6 reference smartphone and a customized arm64-v8a Caffe stack with OpenBLAS. Performance profiling reports inference and training times alongside peak RAM, while the experimental setup fixes selected hyperparameters across runs.
- The CORe Android application was tested on a OnePlus 6 smartphone without additional accelerators.
- Step times, CPU usage, and memory consumption may vary greatly with hardware, operating system, and background processes.
- The experiments used customized arm64-v8a Caffe compiled with OpenBLAS as the BLAS library.
- Hyperparameters were selected on run 0 and fixed for runs 1–4, except iCaRL, which was trained only on run 0 because of its approximately 14-day runtime.
- The model architecture table reports neurons, weights, and operations for each layer to assess accuracy, computation, and memory trade-offs.
- Profiling reports inference and training times plus peak RAM usage, with training times averaged across five incremental training sessions.