PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Continual Learning with Growing Self-Organizing Maps and Synthetic Replay

Pujan Thapa, Alexander Ororbia, Travis Desell

arXiv:2608.27662v1cs.LG

TL;DR

Continual learning needs to absorb changing class streams without task boundaries while avoiding catastrophic forgetting and raw-data storage. This paper uses surprisal-growing GSOMs with per-unit distributional statistics and encoder-decoder replay, achieving competitive results across benchmarks while exposing scalability limits for high-dimensional data without strong representations.

  • Problem

    Continual learning must preserve prior knowledge under changing streams, but class-incremental and task-free settings make forgetting and replay especially challenging.

  • Method

    The framework combines surprisal-driven GSOM growth with per-unit mean, variance, and covariance statistics for exemplar-free synthetic replay, optionally operating through encoder-decoder latent spaces.

  • Results

    The framework remains competitive with existing continual-learning schemes across MNIST, CIFAR-10, CIFAR-100, TinyImageNet, and MiniImageNet, including strong reported results in split benchmarks.

  • Takeaways & Limitations

    Topology-preserving statistical memory provides an unsupervised, adaptive replay approach that avoids storing raw exemplars and supports interpretable continual learning.

  • Takeaways & Limitations

    Pixel-space mean-variance replay produces low-quality samples, while full covariance is infeasible for high-dimensional images, requiring lower-dimensional latent representations.

Abstract

from arXiv · show

This work presents a generative continual learning framework based on growing self-organizing maps (GSOMs) that are augmented with learned distributional statistics as well as encoder-decoder models for class-incremental learning. The proposed approach enables exemplar-free replay using distributional statistical memory, which eliminates the need to store raw data. Each GSOM unit maintains its own mean, variance, and covariance estimates, which are subsequently used to generate synthetic samples for replay; in encoder-decoder configurations, these samples are then decoded back into the input space (via ancestral sampling) for subsequent training. Our method is fully unsupervised, as it does not rely on explicit task boundaries or class labels during training. Results across multiple benchmarks show that the proposed approach achieves performance competitive even with supervised state-of-the-art memory-based methods while consistently outperforming memory-free approaches. In several settings, our framework matches or exceeds existing baselines, particularly in challenging single-class incremental scenarios. We also provide baseline results for single-class incremental TinyImageNet and MiniImageNet, offering a useful reference for future work. This work highlights the effectiveness of an unsupervised, adaptive, topology-driven neural form of statistical replay as a scalable, flexible approach to continual learning.

1 Introduction

Continual learning must incorporate changing data streams while preserving prior knowledge, with catastrophic forgetting especially difficult in task-free class-incremental settings. The paper addresses these challenges using an unsupervised, adaptive GSOM framework with surprisal-driven growth and statistical replay.

  • Continual learning systems must learn from changing data streams while retaining previously acquired knowledge.
  • Class-incremental learning is challenging because models must discriminate among previously seen classes without task-identification information.
  • Many benchmarks assume explicit task boundaries, although real-world streams often have unknown or ill-defined boundaries.
  • Task-free continual learning presents data continuously without explicit task boundaries or task identities.
  • The proposed GSOM framework combines adaptive topology, surprisal-driven growth, and statistical generative replay for unsupervised task-free learning.

2 Methodology

The method combines topology-preserving GSOMs, online distributional statistics, and synthetic replay for task-free continual learning. Surprisal guides local growth, while latent-space encoder-decoder configurations address the computational limits of replay in high-dimensional image data.

  • Self-Organizing Maps: SOMs project high-dimensional data onto a two-dimensional grid while preserving topological relationships through BMU-centered local updates.
  • Growing Self-Organizing Maps: GSOMs expand their map when existing representations insufficiently capture local data variance, allocating capacity to undermodeled regions.
  • Surprisal-Driven Growth: Surprisal-driven growth expands the topology locally when an incoming sample is poorly explained by the current representation.
  • Synthetic Replay: At each batch, incoming data is combined with replayed samples, and synthetic samples drawn from active BMU statistics are decoded for subsequent training.
  • Task-Free Learning: The framework remains unsupervised during representation learning and GSOM updates, using labels only post-hoc for evaluation.
  • Distributional Statistics: Each GSOM unit stores mean, variance, and covariance statistics that support both generative replay and surprisal-based growth decisions.
  • Latent-Space Replay: Full covariance modeling is moved to a lower-dimensional latent space because pixel-space covariance is computationally prohibitive for high-dimensional images.
  • Structured Memory: The GSOM stores statistical summaries rather than exemplars, so memory scales with the number of units instead of dataset size.

3 Results

Across standard split and single-class incremental benchmarks, GSOM-based methods achieve competitive or superior results while training without labels, task identities, or explicit boundaries. Performance is strongest with pretrained representations, whereas models trained from scratch struggle on high-dimensional datasets.

  • Evaluation protocol: Experiments cover MNIST, CIFAR-10, CIFAR-100, TinyImageNet, and MiniImageNet under streaming and split protocols, averaged over 20 random class sequences.Evaluation includes sequential class presentation and disjoint class-group splits.
  • Standard split benchmarks: Without pretrained representations, global VAE-GSOM reaches 50.11% on Split CIFAR-10, 13.81% on Split CIFAR-100, and over 92.11% on Split-MNIST.These results are reported against regularization, replay, and probabilistic baselines.
  • Standard split benchmarks: With pretrained ResNet-18 features, GSOM + ResNet-18 (Global FT) achieves 97.21% on Split-MNIST, 58.1% on Split CIFAR-10, 35.67% on Split CIFAR-100, and 33.60% on Split MiniImageNet.The method is reported as competitive or superior across these benchmarks despite unsupervised training.
  • Representation quality: CLIP embeddings further raise performance to up to 81.92% on Split CIFAR-10 and 65.01% on Split CIFAR-100.The reported improvements indicate that stronger feature spaces benefit GSOM-based replay.
  • Single-class incremental learning: In single-class incremental learning, CLIP-based GSOM reaches up to 79.21% on CIFAR-10 and 63.87% on CIFAR-100 without stored exemplars or task identifiers.The VAE-based variants achieve 51.12% on CIFAR-10 and 11.22% on CIFAR-100 without pretrained representations.
  • Single-class incremental learning: For SCIL, GSOM + CLIP achieves 42.61% on TinyImageNet and 34.57% on MiniImageNet, while from-scratch VAE variants achieve 6%.These benchmarks involve 200 sequential TinyImageNet classes and 100 sequential MiniImageNet classes.

4 Conclusion and Future Work

The framework combines surprisal-driven GSOM growth with generative replay to support task-free continual learning without storing raw data. The authors report broad benchmark competitiveness while identifying generative-quality and distributional-modeling limitations.

  • The GSOM adapts its topology to evolving data distributions, while unit-level statistics generate synthetic replay samples without storing raw data.Encoder–decoder integration supports both low- and high-dimensional settings.
  • Per-BMU encoder–decoder models train unstably and produce lower generative quality than global models because each unit has limited data.The limitation is attributed to localized data scarcity.
  • Gaussian distributional assumptions may inadequately represent complex multimodal distributions, reducing synthetic-replay fidelity.The stated consequence concerns the fidelity of generated replay samples.
  • The framework combines GSOM structure, probabilistic growth, and generative replay for task-free continual learning.The conclusion presents this combination as a promising direction for scalable, task-agnostic continual learning.
  • Future work targets parallel GSOM updates, parameter-efficient per-BMU fine-tuning, purely online learning, and more expressive non-Gaussian or mixture-based replay.These directions address scalability, representation quality, deployment constraints, and replay expressiveness.

C GSOM Error Accumulation and Redistribution

GSOM error accumulates at the BMU and supports structural growth. When interior expansion is blocked, redistribution transfers growth pressure toward feasible boundary locations while conserving total error.

  • Each unit accumulates quantization error by adding the squared Euclidean distance between an input z and its BMU weight wb.The update occurs whenever the unit is selected as the Best Matching Unit.
  • Boundary growth resets the accumulated error at the unit that triggered expansion.
  • When an interior unit exceeds threshold without a free four-neighbourhood position, its accumulated error is redistributed to neighbouring units.This follows the standard GSOM redistribution rule.
  • Redistribution conserves total error: the BMU retains half, and the other half is divided equally among its four neighbours.The mechanism propagates growth pressure toward nearby boundary units where expansion is feasible.
  • Per-BMU encoder–decoder models use localized samples for region-specific specialization but require additional memory.This contrasts with a global model trained across the full dataset.

E Experimental Settings

The experiments evaluate batch-wise GSOM continual-learning configurations with optional global or per-BMU encoder–decoder models. Settings vary growth, replay sampling, initialization, pretrained embeddings, and bias-corrected Gaussian generation.

  • The batch-wise procedure combines each incoming batch with replay, updates the optional global encoder–decoder and GSOM, then rebuilds replay from active BMU statistics.Replay samples are generated for subsequent batches.
  • Exponential moving averages maintain per-unit mean, variance, and covariance statistics, while BMUs are selected using Euclidean distance.Other distance metrics were explored but did not replace the default in the supplied description.
  • GSOM growth is constrained by Igrow and Gmax, limiting growth iterations and units added per iteration to control expansion.The map is fixed after the permitted growth iterations within each batch.
  • Larger Igrow and Gmax improve representational capacity and accuracy but increase computational and memory costs.The reported experiments use conservative settings such as Igrow = 5 and Gmax = 4.
  • Pretrained encoder embeddings, including CLIP ViT-B/32 and ResNet-18, are kept frozen by default to preserve learned representations.Optional fine-tuning was also considered.
  • Synthetic latent samples are generated from each unit’s running mean and covariance after diagonal regularization and eigenvalue clamping.The procedure samples from a positive-definite covariance estimate.

F Ablation Studies

The ablations examine task awareness, adaptive capacity, growth criteria, and bias correction. GSOM trades some performance against fixed SOM for task-free operation and data-dependent map size, while combined growth and bias correction improve results.

  • Fixed SOM versus GSOM: Fixed SOM generally achieves higher accuracy, whereas GSOM provides a fully task-free alternative with adaptive capacity and competitive performance.Fixed SOM uses replay at known class boundaries, while GSOM replays continuously without task information.
  • Capacity and topology: GSOM dynamically adapts map size, using 357–376 units for MNIST and up to 1508 units for CLIP-based settings.The fixed SOM uses a constant 40 × 40 topology with 1600 units across datasets.
  • Growth sensitivity: Increasing growth iterations and per-iteration additions expands maps and improves accuracy, but excessive growth produces disproportionately large topologies.The parameters Igrow and Gmax control the capacity–performance trade-off.
  • Growth criterion: The combined quantization-error and Mahalanobis-surprisal criterion achieves the strongest overall growth results across datasets.Quantization error acts as a stabilizing gate, while surprisal enables more adaptive topology expansion.
  • Bias correction: Bias correction consistently outperforms non-corrected running statistics across datasets and improves the reliability of estimates and training stability.The reported gains are modest but consistent; uncorrected estimates can poorly condition variance and covariance during early training.

G Generative Capability of the Proposed Method

The model generates replay samples directly from GSOM statistics for MNIST and through latent-space sampling followed by VAE decoding for other datasets.

  • Latent-space generation: For other datasets, the VAE–SOM model samples latent vectors from per-unit full-covariance Gaussians and decodes them into reconstructed images.The latent covariance is sampled using eigen-decomposition before decoding with the VAE decoder.

H Class Level Analysis

Confusion matrices show that replay preserves strong class separability on simpler datasets but produces increasingly distributed predictions as dataset complexity increases.

  • Confusion matrices: MNIST predictions are almost entirely diagonal, indicating strong class separability and effective retention with minimal interference.The corresponding row-normalized matrix displays prediction distributions for each true class.
  • Confusion matrices: CIFAR-10 retains dominant diagonals but shows broader confusion among visually similar classes.This reflects partial overlap in the learned representations.
  • Confusion matrices: CIFAR-100 and MiniImageNet distribute predictions more broadly across related groups, with structured cross-group confusion becoming more evident.Larger datasets use coarse class groupings to improve interpretability.
  • Confusion matrices: TinyImageNet has a highly diffuse matrix with weak diagonal dominance and significant mixing across groups, although residual structure remains.The matrix is not entirely random despite the substantial class mixing.
  • Confusion matrices: Overall, prediction confidence shifts from concentrated outputs to distributed outputs as task difficulty increases.This pattern exposes both replay robustness and limitations in complex, fine-grained scenarios.

I Continual Learning Metrics

Continual-learning metrics reveal mild forgetting on CIFAR-10 and substantially stronger forgetting on CIFAR-100, with little forward transfer in the unsupervised single-head setting.

  • Evaluation measures: The analysis uses per-task accuracy matrices, FWT, BWT, forgetting, and row-normalized confusion matrices to characterize continual learning.The confusion matrices summarize prediction distributions for each true class across datasets.
  • Per-task accuracy: CIFAR-10 shows relatively mild catastrophic forgetting, whereas CIFAR-100 exhibits substantially stronger degradation across earlier tasks.The per-task accuracy matrices use a globally trained VAE with GSOM.
  • Transfer and forgetting: FWT is consistently negative and approximately matches per-class baseline accuracy because unseen classes have no prior capability.The single-head unsupervised model has no prior capability on unseen classes before learning each task.
  • Transfer and forgetting: BWT is negative for early tasks but approaches zero and sometimes becomes slightly positive for later tasks.This pattern indicates degradation initially, followed by reduced interference and occasional minor stabilization.

J Encoder Decoder architectures

The experiments use encoder-decoder configurations to reconstruct images from latent representations, alongside global-VAE evaluations on CIFAR-100. The reported analyses include task accuracy, transfer, backward transfer, and forgetting.

  • Evaluation: CIFAR-100 global-VAE evaluation reports a task accuracy matrix across 10 sequential tasks.The evaluated latent dimension is 32 × 2 × 2.
  • Evaluation: CIFAR-100 global-VAE evaluation also reports Forward Transfer, Backward Transfer, and Forgetting for each task.These measures characterize transfer and retention across the sequential task stream.
  • VAE architecture: The CIFAR-10/100 VAE uses residual downsampling and upsampling blocks with ELU activations and Tanh output.The architecture accepts 3×32×32 inputs.

K Visualization of Learned Representations

The visualizations show GSOM representations becoming organized and stable for MNIST, remaining semantically recognizable but more diverse for CIFAR-10, and becoming fragmented for CIFAR-100. They track decoded or unit representations across sequential tasks and classes.

  • MNIST: MNIST GSOM representations quickly form coherent, class-specific prototypes that remain relatively stable across tasks.The corresponding figure visualizes GSOM unit vectors after every single task.
  • CIFAR-10: CIFAR-10 decoded representations are more diverse but retain recognizable semantic structure across classes.The visualization follows decoded GSOM unit vectors after each class.
  • CIFAR-100: CIFAR-100 representations become more fragmented and less visually distinct, reflecting greater intra-class variability and task difficulty.The visualization covers decoded GSOM unit vectors after every 10 classes.
Loading 2608.27662v1…