Source-linked AI summary
An Optimal Transport-driven Approach for Cultivating Latent Space in Online Incremental Learning
Quyen Tran, Hai Nguyen, Hoang Phan, Quan Dao, Linh Ngo, Khoat Than, Dinh Phung, Dimitris Metaxas, Trung Le
TL;DR
Online Class Incremental Learning must model multimodal, shifting class streams despite limited replay and changing latent representations. The paper introduces OTC, combining online Optimal Transport mixture modeling with Dynamic Preservation and MMOT-based testing. The method incrementally updates class distributions and is reported effective on benchmark datasets, with improved forgetting control and performance.
Problem
OCIL methods based on single prototypes or fixed GMM centroids may not represent multimodal class streams or adapt to latent feature shifts.
Method
OTC uses MMOT to incrementally update GMM centroids and covariance matrices, while Dynamic Preservation strengthens representation learning and MMOT supports testing.
Results
Benchmark experiments demonstrate OTC’s effectiveness, including improved performance and strong control of forgetting of previously observed tasks.
Takeaways & Limitations
MMOT provides an online way to characterize complex class data, while Dynamic Preservation helps retain old knowledge in OCIL.
Takeaways & Limitations
The computational-complexity analysis is stated in terms of latent dimensionality d, centroids per class K, and mini-batch size B.
Abstract
from arXiv · showhide
In online incremental learning, data continuously arrives with substantial distributional shifts, creating a significant challenge because previous samples have limited replay value when learning a new task. Prior research has typically relied on either a single adaptive centroid or multiple fixed centroids to represent each class in the latent space. However, such methods struggle when class data streams are inherently multimodal and require continual centroid updates. To overcome this, we introduce an online Mixture Model learning framework grounded in Optimal Transport theory (MMOT), where centroids evolve incrementally with new data. This approach offers two main advantages: (i) it provides a more precise characterization of complex data streams, and (ii) it enables improved class similarity estimation for unseen samples during inference through MMOT-derived centroids. Furthermore, to strengthen representation learning and mitigate catastrophic forgetting, we design a Dynamic Preservation strategy that regulates the latent space and maintains class separability over time. Experimental evaluations on benchmark datasets confirm the superior effectiveness of our proposed method.
1. Introduction
Online Class Incremental Learning must adapt to changing data distributions with single-iteration updates and no task IDs during inference, while limiting catastrophic forgetting. OTC addresses multimodal class streams with an Optimal Transport-based online mixture model, Dynamic Preservation, and memory-buffer selection.
- OCIL faces dynamically changing distributions, single-iteration updates for arriving batches, and unavailable task IDs during inference.
- Single class centroids may miss multimodal data, whereas OTC uses multiple adaptive centroids learned from training samples.The motivation figure contrasts one adaptive centroid per class with four adaptive centroids per class.
- OTC applies Dynamic Preservation and memory-buffer selection to strengthen class discrimination and mitigate forgetting of previously observed tasks.The framework pulls same-class representations closer and pushes different-class representations farther apart.
- MMOT combines Optimal Transport with Gaussian Mixture Models to incrementally characterize complex incoming streams in OCIL.The framework uses an online formulation based on the entropic dual form of OT and GMMs.
- Benchmark experiments demonstrate that OTC effectively addresses streaming-data multimodality and achieves strong control of forgetting.
2. Related work
Related work addresses continual learning through regularization, architecture, and memory, while OCIL methods emphasize replay selection and representation learning. Prior GMM approaches retain fixed centroids or rely on costly EM updates, motivating OTC’s online gradient-based alternative.
- Continual learning methods commonly use regularization, task-specific architectures, or memory to preserve knowledge from earlier tasks.Memory-based methods store past data or generate pseudo-samples from previous history.
- Prior OT-GMM work mainly computes distances between given mixture models, whereas learning GMM parameters through OT remains an inverse problem.
- OCIL research focuses on selecting meaningful replay samples or learning representations through contrastive, generative, distillation, and related strategies.
- Existing GMM-based methods use EM and keep learned centroids fixed, limiting adaptation when latent features shift with new data.OTC instead updates centroids through gradient descent, avoiding EM’s multiple iterations at each learning step.
3. Background
Optimal Transport measures the minimum expected cost over couplings between distributions, and Wasserstein distance formalizes this transport problem. Entropic regularization adds a KL-divergence penalty and yields a dual formulation for machine-learning use.
- Wasserstein distance minimizes expected transport cost over couplings whose marginals are the two input distributions.
- The transport distributions operate on a domain Ω⊆R^d with a non-negative continuous cost or metric d(x, y).
- The background specifies the coupling constraint through Γ(Q, P), the set of couplings admitting Q and P as marginals.
- Entropic regularization augments the transport objective with ε times the KL divergence between a coupling and the independent coupling Q ⊗ P.ε is the regularization rate, while Q ⊗ P represents independent coupling.
- Fenchel-Rockafellar duality provides an entropic regularized dual form that enables OT applications in machine learning and deep learning.
4. Proposed Method
OTC combines an online Optimal Transport-based mixture model with dynamic preservation to represent multimodal class streams and maintain separable latent representations. Its centroids and covariance matrices update incrementally from current and replayed class batches, while learned centroids constrain representation learning.
- Online learning: At each online step, new samples and replayed old samples are encoded into latent representations for continual updating.The system receives new-class batches, retrieves old-class data from a memory buffer, and processes both through the feature extractor.
- MMOT framework: The method replaces EM-based mixture updates with cheaper gradient-descent steps, allowing centroids to adapt as the latent space shifts with new data.The paper identifies EM’s multiple iterations as costly for online learning and emphasizes flexible centroid updates.
- Online learning: MMOT incrementally updates mixing proportions, centroids, and covariance matrices using gradient descent on each class’s streamed data.The optimization uses current or replayed batches and updates the parameters after a few Kantorovich-network steps.
- MMOT framework: Gumbel-Softmax provides differentiable mixture-component sampling, while the entropic dual form of Optimal Transport makes the Wasserstein objective suitable for online optimization.The temperature parameter and Gumbel noise enable continuous relaxation of categorical component assignments.
- MMOT framework: OTC uses an MMOT framework to characterize each class with multiple adaptive centroids rather than a single prototype.The approach learns Gaussian-mixture parameters by minimizing Wasserstein distance between empirical and modeled latent distributions.
- Dynamic preservation: Dynamic preservation pulls class representations toward their respective MMOT centroids, making same-class features closer and different-class features more separate.This objective uses the centroids of each class’s learned mixture rather than one prototype to constrain the latent space.
5. Experiment
Experiments across four OCIL benchmarks compare OTC with nine baselines using accuracy and forgetting metrics. OTC generally improves accuracy, especially with limited memory and harder datasets, while centroid studies show that adaptive multi-centroid representations and centroid-based replay selection improve performance.
- Experimental setup: OTC is evaluated on Tiny-ImageNet, CIFAR-100, CIFAR10, and MNIST against nine baselines using Final Average Accuracy and Final Forgetting Measure.The reported evaluations use benchmark datasets, multiple memory sizes, and the two stated metrics.
- Performance comparison: Up to 2% higher final average accuracy is reported for OTC over baselines, with the strongest improvement for each dataset occurring at the smallest memory size.This result is reported in Table 1 across three challenging datasets and varying memory sizes.
- Performance comparison: OTC outperforms baselines by up to 2% on CIFAR-100 and 13% on Tiny-ImageNet, while remaining slightly better than BiC+AC on CIFAR10 with M = 1k.The Tiny-ImageNet result concerns a sequence of 20 tasks.
- Performance comparison: OTC consistently ranks among the top two methods for lowest average forgetting on CIFAR10 and CIFAR-100, but forgets more than CoPE on Tiny-ImageNet.The authors attribute CoPE’s low Tiny-ImageNet forgetting to poor initial performance, which leaves less accuracy to lose.
- Ablation study: Increasing centroids per class improves CIFAR10 accuracy up to a memory-dependent threshold; with M = 200, more than 3 centroids degrades performance, while M = 1K performs best with 4.With M = 1K, increasing the count to 5 slightly degrades model quality.
- Ablation study: Centroid-based replay selection consistently outperforms random sampling and improves the diversity of the episode memory.The ablation supports using MMOT centroids to incrementally characterize latent-space data when selecting replay samples.
6. Conclusion
The work presents MMOT for Online Class Incremental Learning and combines it with Dynamic Preservation and MMOT-based testing to retain knowledge and improve performance.
- MMOT incrementally characterizes data complexity in Online Class Incremental Learning.
- Dynamic Preservation enhances the model’s ability to retain old knowledge.
- The MMOT-based testing strategy improves performance.
- Experiments on benchmark datasets demonstrate the method’s effectiveness for online learning.
7. Background
A Gaussian Mixture Model represents a probability distribution with Gaussian components, and Wasserstein-based formulations provide established ways to compare or optimize such models.
- A Gaussian Mixture Model is a probability distribution composed of several Gaussian components.
- For a GMM with K components, the mixture weights sum to 1, and GMM_d(K) contains measures representable with at most K components.
- GMMs are fundamental in statistical problems and are applied to image segmentation, anomaly detection, and keystroke recognition.
- Prior work introduced optimization formulations for Wasserstein distances between GMMs and related Orlicz–Wasserstein distances.
8. Compare EM and MMOT for OCIL
MMOT and EM both scale linearly with the number of mixture components, but MMOT avoids EM’s iterative inner loop and dense responsibility storage, making it better suited to continual feature drift and streaming OCIL data.
- MMOT update: MMOT updates mixture parameters using reparameterized sampling, Gumbel–Softmax, and gradients of an entropic OT dual objective.
- MMOT update: The MMOT update samples mixture components and relaxed weights, then evaluates latent-space distances using diagonal-covariance operations.
- EM comparison: Classical EM evaluates all K component likelihoods for B points at O(BKd) per iteration and repeats this for I_EM iterations until convergence.
- Complexity comparison: MMOT avoids the inner-loop factor I_EM and the B×K responsibility tensor while retaining linear scaling in K.
- Complexity comparison: MMOT uses O(T_ϕB + BKd + SBd) time and O(Bd + Kd) memory per class and batch.
- OCIL suitability: When EM requires more than a few iterations, MMOT becomes asymptotically cheaper in computation and memory and better suited to streaming, non-stationary OCIL data.
9. Implementation Details
The experiments use four benchmark datasets with different task partitions, specified model and batch settings, average accuracy and forgetting metrics, and repeated runs across class orders and random seeds.
- Datasets: Tiny-ImageNet has 200 classes and is divided into 100 tasks with two classes per task.
- Datasets: CIFAR100 has 100 classes and is split into 10 tasks with 10 classes per task.
- Datasets: CIFAR10 has 10 classes and is divided into five nonoverlapping tasks with two classes per task.
- Datasets: MNIST is split into five disjoint tasks, each containing two handwritten-digit classes.
- Training setup: Streaming inputs use batch size 10, buffer samples use batch size 64, and experiments apply data augmentation.
- Model setup: MNIST uses a two-hidden-layer 400-unit MLP, while the other datasets use a slim ResNet-18.
- Metrics: Average accuracy measures mean test accuracy across tasks, while average forgetting measures the gap between peak and final task accuracy.
- Robustness: Multiple runs vary class sequences, with means and standard deviations reported across orderings and random seeds.
10. Additional Experiments
Additional experiments show that OTC outperforms strong baselines on MNIST and remains superior in offline class-incremental learning.
- On MNIST, OTC is compared with GSA, MOSE, and BiC+AC in Table 4.
- 2.4% higher average accuracy and 1.6% less forgetting are reported for OTC on MNIST against these baselines.
- More than 6% is the largest reported gap between OTC and the strongest considered baseline across offline class-incremental learning cases.Table 5 reports average accuracy in the offline CIL setting.