Recent Advances of Continual Learning in Computer Vision: An Overview

1 Introduction

• Retraining from scratch to adapt to daily new images and videos is time-consuming and extremely inefficient.The paper motivates models that learn and update sequentially during inference.Source: §1 · p. 1
• Sequential updates can cause catastrophic forgetting across categories and concept drift within a category, creating a stability-plasticity dilemma.Models must retain previously learned knowledge while remaining able to learn new knowledge.Source: §1 · p. 1
• The overview covers recent continual-learning methods across computer vision, including subareas beyond image classification.It contrasts its coverage with earlier overviews focused on image classification [14] [16].Source: §1 · p. 2
• It organizes methods into regularization, knowledge distillation, memory, generative replay, parameter isolation, and combinations of these techniques.The overview also discusses computer-vision subareas where continual learning may help but remains insufficiently investigated.Source: §1 · p. 2

2 Continual Learning: Problem Definition

• A task stream T = {T1, T2, ...} contains tasks with class-specific data instances and labels, and Nn counts classes covered after task Tn.Mn denotes the model after learning Tn.Source: §2 · p. 2
• Each update Pn maps the previous model Mn−1 and task Tn to Mn; earlier data are generally inaccessible except for instances and labels stored in memory.The objective is good performance on the new task without affecting previous tasks.Source: §2 · p. 2
• Task-, domain-, and class-incremental learning differ in whether inference receives a task ID and which classes it must distinguish.Task-incremental learning requires a task ID; domain- and class-incremental learning do not, while class-incremental learning distinguishes classes across tasks.Source: §2 · p. 3

3 Evaluation Metrics

• Average accuracy Aq measures performance after q tasks have been learned.It is defined using accuracy on each task's test set after continual learning.Source: §3 · p. 3
• Average forgetting Fq measures forgotten knowledge across the first q−1 tasks.For task p, forgetting is the difference between its maximum knowledge during learning and the knowledge remaining after q tasks.Source: §3 · p. 3
• Intransigence Iq measures how much continual learning prevents learning a new task compared with batch learning.The batch-learning comparator is accuracy on task q when batch learning is used for the q tasks.Source: §3 · p. 3
• Backward transfer BWTq measures how continual learning on task q influences performance on previously learned tasks.The section defines it using accuracies on earlier tasks after q tasks and after their own learning.Source: §3 · p. 3
• Forward transfer FWTq measures how continual learning on task q potentially influences performance on future tasks.The section defines bp as accuracy on task p at random initialization.Source: §3 · p. 3

• Methods are grouped as regularization-, knowledge distillation-, memory-, generative replay-, and parameter isolation-based, with some combining categories’ complementary strengths.Source: §4 · p. 3

4.1.1 Regularizing Loss Function

• EWC [27] adds a Fisher-information-weighted quadratic loss penalty on important weights, guiding updates toward low-loss regions shared across tasks.Source: §4.1.1 · p. 4, Figure 3 · p. 5
• EWC assumes a diagonal Fisher matrix and incurs linearly increasing cost across tasks; online EWC and EWC++ reduce this burden with a single penalty or Fisher matrix.Source: §4.1.1 · p. 4
• Other loss-based methods estimate parameter importance through synapses, output sensitivity, or asymmetric loss approximations, and may widen shared low-loss regions or balance stability and plasticity.Source: §4.1.1 · p. 5

4.1.2 Regularizing Gradient

• Gradient-regularization methods preserve prior-task performance by projecting new-task updates into subspaces that minimally interfere with earlier tasks.Source: §4.1.2 · p. 6
• Orthogonal-gradient approaches have been adapted to prompt tuning and spiking networks, where Hebbian projections avoid matrix inversion and singular value decomposition.Source: §4.1.2 · p. 6

4.1.3 Regularizing Learning Rate

• Learning-rate methods reduce rates for important parameters or tune learning rate and batch size to control local-minimum geometry; another approach regularizes through sparse Gaussian-process approximation.Source: §4.1.3 · p. 6

4.2 Knowledge Distillation Based Methods

• Continual distillation transfers knowledge from models trained on previous tasks to models learning new tasks to consolidate earlier knowledge.Source: §4.2 · p. 6

4.2.1 Distillation in the Presence of Sufficient Data Instances

• For sufficient-data classification, LwF [63] uses the previous model’s responses on new-task data as targets for old-task classifiers and ground truth for the new classifier.Source: §4.2.1 · p. 6, Figure 4 · p. 7
• LwF [63] may perform poorly when task data distributions differ substantially; later variants distill attention, intermediate representations, uncertainty, or causal effects.Source: §4.2.1 · p. 7
• Other classification methods address data imbalance, concept drift, or selective knowledge transfer through balanced losses, weight alignment, exemplar updates, and task- or layer-specific transfer.Source: §4.2.1 · p. 7
• In semantic segmentation, distillation methods address background distribution shift and preserve spatial relations; related work also covers person re-identification and compatibility-aware transfer.Source: §4.2.1 · pp. 7–8
• Distillation has also been adapted to object detection, conditional image generation, and captioning, with methods targeting detector outputs, aligned generated images, and captioning features.Source: §4.2.1 · p. 8
• A sufficient-data approach selects unlabeled instances similar to prior tasks and distills knowledge from previous tasks jointly rather than separately.Source: §4.2.1 · pp. 8–9

4.2.2 Distillation in the Presence of Limited Data Instances

• Limited-data work includes few-shot methods using anchor-task alignment, word embeddings, meta-learned task-adaptive features, or language-model knowledge, plus federated learning with global and task-specific parameters.Source: §4.2.2 · p. 9

4.3 Memory Based Methods

• Memory-based methods store prior data instances or related information in a buffer and replay them during new-task learning to mitigate catastrophic forgetting.Source: §4.3 · p. 9

4.3.1 Retraining on Memory Buffer

• Retraining methods include iCARL, which trains on stored and new instances and classifies by nearest exemplar mean, and ER [104], which uses reservoir sampling for streaming data.Source: §4.3.1 · p. 9
• Reservoir sampling can lose information from underrepresented tasks, prompting alternatives such as constraint-based selection, class-balancing sampling, and weighted task coresets.Source: §4.3.1 · pp. 9–10
• Other retraining approaches store compressed examples or activation volumes, and may add balanced sampling, targeted retrieval, bias correction, or task-specific statistics and anchors.Source: §4.3.1 · pp. 10–11
• Memory-based retraining extends to segmentation, detection, reasoning, generation, and compression, using strategies such as image patches, selective retrieval, and memory management.Source: §4.3.1 · p. 11

4.3.2 Optimizing on Memory Buffer

• Optimization-based methods use stored examples to constrain updates without retraining on them; GEM [25], for example, prevents updates from increasing episodic-memory losses.Source: §4.3.2 · p. 11
• Other optimization approaches preserve shared gradients or feature-space topology, or store task embeddings, feature descriptors, prototypes, and task-conditioned priors.Source: §4.3.2 · pp. 11–12
• GDumb builds a balanced memory buffer and trains from scratch on it at inference, while few-shot segmentation methods store class-level embeddings to support new classes.Source: §4.3.2 · p. 12

4.4 Generative Replay Based Methods

• Generative replay replaces a memory buffer with a generative module that reproduces information related to previous tasks.The review distinguishes generating previous data instances from generating those instances alongside their latent representations.Source: §4.4 · p. 12

4.4.1 Generating Previous Data Instances

• Most methods generate previous data instances with labels; classification approaches include GAN replay, pseudo-rehearsal, and other generative models.GANs performed best among the compared generative models, but all struggled with more complex datasets; generative replay can be computationally heavy, especially when combined with distillation.Source: §4.4.1 · p. 12
• Other approaches generate previous-task data with diffusion models or compensate for semantic drift using adversarially generated instances.Some methods also use unlabelled data and classifier pseudo-labels to update the generator.Source: §4.4.1 · pp. 12–13
• Generative replay has also been applied to class-conditional image generation, NeRFs, and temporal action segmentation.The temporal action method adds a temporal coherence variable to help generate temporally coherent videos.Source: §4.4.1 · p. 13

4.4.2 Generating Previous Data Instances along with Latent Representations

• Some methods generate latent representations alongside prior data, while another uses VAE latent features for frame prediction without using generated instances.The latent representations can support consolidation or representation learning; frame-prediction features can be integrated with other sensory information using a Markov Jump Particle Filter.Source: §4.4.2 · p. 13

4.5 Parameter Isolation Based Methods

• Parameter isolation assigns different model parameters to different tasks to prevent later tasks from interfering with previously learned knowledge.Some methods dynamically change the architecture, while others use fixed architectures.Source: §4.5 · p. 13, §4.5.1 · p. 13, §4.5.2 · p. 15

4.5.1 Dynamic Network Architectures

• Dynamic architectures include growing subnetworks, selecting experts, and searching or composing modules as new tasks arrive.Examples expand networks when prior knowledge is insufficient, or reuse existing modules and create new ones based on a data-driven prior; PICLE uses probabilistic search to assess module compositions.Source: §4.5.1 · pp. 13–14
• Other dynamic methods separate shared and task-specific features or parameters, including tree-structured decomposition, task-specific modulators, and prompt-based approaches.Prompt-based work includes task-specific subspaces and methods that route inputs to adapters or update existing prompts versus learning new ones.Source: §4.5.1 · pp. 14–15
• Dynamic parameter isolation also extends beyond classification to image generation, action recognition, multimodal reasoning, translation, and other vision tasks.Examples include expandable expert groups, selected action-recognition blocks, and model expansion when trainable parameters are insufficient.Source: §4.5.1 · p. 15

4.5.2 Fixed Network Architectures

• Fixed-architecture methods freeze parameters on selected paths or attention masks for earlier tasks while training remaining parameters for new tasks.PathNet selects paths through tournament selection; its later-task procedure freezes prior selected paths and reinitializes and retrains the rest.Source: §4.5.2 · pp. 15–16, Figure 5 · p. 16
• Other fixed-architecture approaches use meta-learning masks, pruning, inference-time parameter generation, or task-specific activation of kernels and vocabulary masks.PackNet trains the whole network for each task and prunes weights to free parameters; MARK uses a common knowledge base and trainable masks.Source: §4.5.2 · p. 16

4.6.1 Combination of Regularization and Memory Based Methods

• Regularization-and-memory methods combine constraints on model learning with stored exemplars, prototypes, or inducing points.Examples include VCL, which combines variational inference with coreset data summarization, and functional regularization using stored inducing points or decision-boundary data.Source: §4.6.1 · p. 17

4.6.2 Combination of Knowledge Distillation and Memory Based Methods

• Distillation-and-memory methods pair knowledge preservation with selected previous-task data, with online approaches also addressing overfitting, underfitting, or limited plasticity.Applications include video, fake-media detection, object detection, embodied-agent learning, and multimodal medical data.Source: §4.6.2 · pp. 17–18

4.6.3 Combination of Knowledge Distillation and Generative Replay Based Methods

• Distillation combined with generative replay uses generated prior-task data to support learning, including in federated learning, semantic segmentation, and object detection.Approaches use generated images or features with distillation; some also address data imbalance or biased features.Source: §4.6.3 · p. 18

4.6.4 Combination of Memory and Generative Replay Based Methods

• Memory-and-generative-replay methods train with stored and generated instances, or use stored statistics or compressed embeddings to generate pseudo-data.EEC compresses stored embeddings into centroids and covariance matrices when memory fills; other methods use preparatory generated data or feature statistics for image-style alignment.Source: §4.6.4 · pp. 18–19

4.6.5 Combination of Generative Replay and Parameter Isolation Based Methods

• Generative replay combined with parameter isolation includes methods that pair replay with HAT or expand representations and synthesize prior-task prototypes.CURL addresses unsupervised continual learning without task labels, task boundaries, or class labels; EASE uses task-specific adapters and a subspace ensemble.Source: §4.6.5 · p. 19

4.6.6 Other Combinations

• Other combinations include regularization with distillation or parameter isolation, and methods combining regularization, distillation, and memory or replay.DER combines regularization, knowledge distillation, and memory for sequential data without a clear task boundary; continual adversarial defense combines replay, self-distillation, and regularization.Source: §4.6.6 · p. 19

4.7 Other Methods

• Meta-learning approaches initialize or adapt models across tasks, control knowledge transfer, or separate task-agnostic features from task-specific classifiers.iTAML uses a two-phase process to predict the task and then adapt, updating task-specific parameters separately to mitigate data imbalance.Source: §4.7 · pp. 19–20
• Generic frameworks can combine with continual-learning methods through additional classifiers, compositional components, primitive reuse, or unlearn-relearn updates.MUC trains additional classifiers for out-of-distribution data and can further mitigate forgetting when combined with MAS or LwF.Source: §4.7 · p. 20
• Other classification methods include repetition-based environments, sparse subnetworks, representation alignment, stable-and-plastic blocks, and coarse-to-fine label settings.The review notes that continual learning with repetition was shown to be important in mitigating catastrophic forgetting.Source: §4.7 · p. 20
• Additional applications include continual semantic segmentation, graph learning, model-based reinforcement learning, vision-language benchmarks, test-time adaptation, video streams, and visual-inertial odometry.In a single video stream, one work observed that using momentum during optimization seemed unhelpful.Source: §4.7 · p. 21

5.1 Image Classification (Online Continual Learning)

• Yan et al. [246] consistently leads average accuracy, while the state-of-the-art method for average forgetting varies across online-learning settings and datasets.The comparison uses average accuracy and average forgetting on CIFAR-100 and Tiny-ImageNet; each data instance is passed to the model only once.Source: §5.1 · p. 21, Table 1 · p. 22

5.2 Image Classification (Few-Shot Class Incremental Learning)

• Zou et al. [287] consistently outperforms other methods on both accuracy and performance dropping rate in few-shot class-incremental learning.The comparison uses CUB200, with 100 base classes and 100 new classes divided into ten 10-way, 5-shot tasks.Source: Table 2 · p. 22, §5.2 · p. 22

5.3 Semantic Segmentation

• Across the 100-50, 100-10, and 100-5 semantic-segmentation settings, the state-of-the-art method varies.The comparisons report mIoU for base classes, new classes, all classes, and the average across learning steps.Source: §5.3 · pp. 22–23, Table 3 · p. 23

5.4 Object Detection

• The most recent method [261] consistently outperforms other class-incremental object-detection methods across the reported metrics.The comparison covers MS COCO 2017 under 40 + 40 and 70 + 10 base/new-class settings, using AP across IoU thresholds and object sizes.Source: §5.4 · p. 23

5.5 Person Re-identification

• On average, the most recent person re-identification method consistently outperforms other methods, although the leading method varies across datasets.The comparison uses five datasets and two training orders.Source: §5.5 · p. 23

6 Discussion

• Most continual learning methods still focus on image classification, and simply adapting them to other vision problems may not yield satisfactory performance.Source: §6 · pp. 23–24
• The discussion includes method comparisons for class-incremental object detection measured in AP (%) and continual person re-identification.Source: Table 4 · p. 24, Table 5 · p. 24
• Classification methods do not always transfer well: they can even encourage forgetting in segmentation [77], while selective retrieval helps analogical reasoning but is less effective in image classification [130] [26] [125].Source: §6 · p. 24
• Most studies develop problem-specific algorithms, leaving continuous learning of knowledge shared across related vision problems, such as face recognition and anti-spoofing, open for exploration.Source: §6 · p. 24
• Many methods assume fully supervised tasks without class overlap, unlike real-world streams that may be unlabeled and share classes across time.Source: §6 · p. 24
• Many studies emphasize accuracy without evaluating computational complexity or resource assumptions, factors relevant to lightweight-device applications.Source: §6 · p. 24

7 Conclusion

• The overview synthesizes continual learning techniques and applications across computer vision, compares methods on different tasks, and discusses future research directions.Source: §7 · p. 25

8 Author contributions

• Haoxuan Qu handled conceptualization, visualization, and the original draft.Source: §8 · p. 25
• Hossein Rahmani and Li Xu contributed to supervision and review and editing, respectively; both are listed for review and editing.Source: §8 · p. 25
• Bryan Williams contributed to review and editing, while Jun Liu handled conceptualization, project administration, supervision, and review and editing.Source: §8 · p. 25