PaperlayerView exact revision on arXiv

Source-linked AI summary

ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning

Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, Jun Zhu

arXiv:2101.00407v2cs.LGcs.AIstat.ML

TL;DR

Continual learning commonly assumes incoming data are labeled, although real applications may provide only partially labeled incremental data, and existing methods do not effectively exploit the unlabeled portion. ORDisCo jointly learns a classifier and conditional GAN, using online generator replay and discriminator consistency. It significantly improves classification and conditional image generation over strong baselines on the evaluated SSCL benchmarks.

  • Problem

    Existing continual-learning strategies may not effectively capture and exploit massive incremental unlabeled data, which lack annotations.

    Source: §3.2 · p. 4

  • Method

    ORDisCo jointly learns a classifier and conditional GAN, replaying generated samples online and stabilizing discriminator parameters important for old unlabeled data and their pseudo-labels.

    Source: §1 · pp. 1–2

  • Results

    ORDisCo significantly improves classification and conditional image generation over strong baselines on SSCL benchmarks including SVHN, CIFAR10, and Tiny-ImageNet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations show that ORDisCo can capture and exploit unlabeled data in classification and conditional generation across the evaluated benchmarks.

    Source: §6 · p. 8

    The paper leaves a systematic study of alternative unlabeled losses for future work.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Continual learning usually assumes the incoming data are fully labeled, which might not be applicable in real applications. In this work, we consider semi-supervised continual learning (SSCL) that incrementally learns from partially labeled data. Observing that existing continual learning methods lack the ability to continually exploit the unlabeled data, we propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN), which continually passes the learned data distribution to the classifier. In particular, ORDisCo replays data sampled from the conditional generator to the classifier in an online manner, exploiting unlabeled data in a time- and storage-efficient way. Further, to explicitly overcome the catastrophic forgetting of unlabeled data, we selectively stabilize parameters of the discriminator that are important for discriminating the pairs of old unlabeled data and their pseudo-labels predicted by the classifier. We extensively evaluate ORDisCo on various semi-supervised learning benchmark datasets for SSCL, and show that ORDisCo achieves significant performance improvement on SVHN, CIFAR10 and Tiny-ImageNet, compared to strong baselines.

1. Introduction

The introduction frames semi-supervised continual learning (SSCL) as learning from incrementally arriving, partially labeled data, a setting existing continual-learning methods do not effectively exploit. ORDisCo couples a classifier with a conditional GAN and online replay, and significantly outperforms strong baselines on the evaluated benchmarks.

1. Introduction

  • SSCL addresses incremental learning from partially labeled data, which the paper describes as common in applications such as face recognition.In the face-recognition example, incoming user data often lack true labels unless the user provides a password for verification.Source: §1 · p. 1
  • Existing continual-learning strategies may not effectively exploit unlabeled data, a challenge identified as catastrophic forgetting of unlabeled data.The introduction reports that joint training of a strong semi-supervised classifier significantly outperforms the best existing continual-learning strategies on the same classifier.Source: §1 · p. 1
  • ORDisCo jointly learns a semi-supervised classifier and a conditional GAN, with the generator capturing the joint distribution of partially labeled data.The classifier also supplies pseudo-labels for unlabeled data to improve conditional-generator training.Source: §1 · pp. 1–2
  • ORDisCo replays conditional-generator samples to the classifier online and selectively stabilizes discriminator parameters for old unlabeled data and their classifier-predicted pseudo-labels.The paper describes online replay as time- and storage-efficient.Source: §1 · p. 2
  • ORDisCo significantly improves both classification and conditional image generation over strong baselines in SSCL.The evaluations use SVHN, CIFAR10, and Tiny-ImageNet benchmarks.Source: §1 · p. 2

2. Related Work

Related work spans supervised continual-learning strategies, generative replay, limited-supervision continual learning, and semi-supervised learning. The paper distinguishes SSCL by its incremental partially labeled data setting, including both new instances and new classes.

2. Related Work

  • Continual-learning methods include weight regularization and replay; other approaches address memory imbalance or buffer selection.Generative replay aims to recover learned data distributions, but conditional generation relies heavily on labeled data and offline saving adds time and storage costs.Source: §2 · p. 2
  • Few-shot continual learning [40] [3] and unsupervised continual learning use limited labeled data or large unlabeled datasets, while SSCL covers both new instances and new classes from partially labeled data.The paper contrasts SSCL with, which stacks generators and discriminators, whereas its setting assumes relatively constant model size.Source: §2 · p. 2
  • Semi-supervised learning methods include pseudo-labeling, consistency regularization [13], VAT, and graph Laplacian regularization.GAN-based SSL includes two-network approaches [38] [23] [30], Triple-GAN, and ISL-GAN; the paper notes these methods consider static rather than dynamic data distributions.Source: §2 · pp. 2–3

3. Semi-supervised Continual Learning and Its Challenges

Semi-supervised continual learning (SSCL) learns from incremental data with few labels and many unlabeled examples, but existing continual-learning strategies struggle to exploit the unlabeled distribution. The section formulates this setting and studies the resulting forgetting problem.

3.1. Problem Formulation

  • SSCL provides small amounts of labeled data and large amounts of unlabeled data in incremental semi-supervised batches.Each batch contains labeled and unlabeled sub-batches, and performance is evaluated after each batch.Source: §3.1 · p. 3

3.2. A Systematic Study of Existing CL Methods

  • The study adapts weight-regularization and memory-replay strategies to SSCL and evaluates them on SVHN batches with three labels per class.The training set is split into 30 batches; memory replay uses uniform sampling because labels are insufficient for sample selection.Source: §3.2 · p. 3
  • JT exceeds SMB by around 9.70% on average, a gap attributed to catastrophic forgetting of unlabeled data.SMB replays supervised data, while JT trains jointly on all training samples seen.Source: Figure 1 · p. 3
  • Sequential training significantly underperforms joint training, and weight regularization does not resolve the SSCL problem.The authors hypothesize that parameter-change regularization limits learning from incremental data.Source: §3.2 · p. 4
  • Even a 10,000-image unlabeled memory buffer poorly covers the training distribution, while ORDisCo’s generated samples are reported to achieve better results with less storage.The buffer stores around 10% of the dataset, yet its coverage remains limited.Source: §3.2 · p. 4, Figure 2 · p. 4

4. Method

ORDisCo jointly trains a classifier with a conditional GAN so each can use information from partially labeled incremental data. It combines online generative replay with selective discriminator stabilization to address forgetting of unlabeled data.

4.1. Conditional Generation on Incremental Semisupervised Data

  • The classifier, generator, and discriminator learn together: the classifier supplies pseudo-labels for conditional generation, while the generator and discriminator recover incremental-data distributions.The framework describes the generator and classifier as helping each other through this joint training.Source: §4.1 · pp. 4–5
  • The classifier uses cross-entropy on labeled data, including a supervised memory buffer, and a consistency-regularization loss for unlabeled data.The consistency loss uses an exponential moving average of the classifier and two random noises; other unlabeled-loss options are left for future study.Source: §4.1 · p. 5

4.2. Improving Continual Learning of Unlabeled Data

  • After five SVHN batches, ORDisCo's conditional samples have better label consistency than a conditional-GAN baseline using the same supervised-memory replay.Each sample row has a fixed label; the caption attributes the comparison to the two proposed strategies.Source: Figure 4 · p. 5
  • ORDisCo replays conditional samples to the classifier online, using them as labeled and unlabeled data while avoiding the time and storage costs of offline replay.The samples come from the generator rather than from saved old models or offline-sampled data.Source: §4.2 · p. 5, Table 1 · p. 5
  • The discriminator selectively penalizes parameter changes according to their importance for preserving consistency on old unlabeled data and classifier-predicted pseudo-labels.Importance is estimated from gradient norms on the squared l2-norm for unlabeled data-label pairs and averaged across batches.Source: §4.2 · p. 6

5. Experiment

The experiments evaluate ORDisCo and baselines across semi-supervised continual-learning settings, including incremental instances, new classes, and conditional generation. They report stronger classification and generation results for ORDisCo, with replay and discrimination-consistency regularization contributing to performance.

5.1. Experiment Setup

  • The benchmarks use incremental batches with few labels; New Instance splits datasets into batches, while New Class presents two new classes in each of five binary tasks.Evaluation uses all classes seen so far without access to task labels.Source: §5.1 · p. 6

5.2. SSCL of New Instance

  • The reported performance is measured after each semi-supervised batch is trained to convergence, and strong results on limited data indicate ORDisCo can learn a usable model before more data arrive.The evaluation includes 30 incremental batches for the reported SVHN and CIFAR10 settings.Source: §5.2 · p. 7
  • At the final batch, ORDisCo improves over SMB by 12.20% on SVHN-1 and 5.95% on CIFAR10-5, and outperforms SMB and SMB+UMB on the reported Tiny-ImageNet settings.The Tiny-ImageNet comparisons cover 10- and 20-class images at 32×32 and 10-class images at 64×64.Source: Figure 5 · p. 7, §5.2 · p. 7

5.3. Ablation Study and Analysis

  • Both online semi-supervised generative replay and discrimination-consistency regularization substantially improve SSCL, with the regularizer more effective when labels and batches are limited.A jointly trained classifier also predicts ORDisCo's conditional samples better than samples from a sequentially trained conditional GAN with SMB.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4. SSCL of New Class

  • ORDisCo significantly outperforms SMB, SMB+UMB, and SMB+UMB+UC in New Class SSCL through more effective use of unlabeled data.The setting uses five binary classification tasks and keeps the total number of labels equal to New Instance.Source: §5.4 · p. 8

5.5. Conditional Generation in SSCL

  • From the sixth incremental batch, ORDisCo generates conditional samples described as high-quality and significantly better than the SMB baseline.The figure shows samples for ‘0’ in SVHN-3 and ‘horse’ in CIFAR10-13.Source: Figure 6 · p. 8, §5.5 · p. 8

6. Conclusion

The paper analyzes semi-supervised continual learning and proposes a classifier–conditional-GAN framework for partially labeled data. Its evaluations support use in classification and conditional generation, while broader plug-and-play extensions remain future work.

  • The paper provides a systematic analysis of semi-supervised continual learning, a realistic setting described as challenging and not extensively studied.Source: §6 · p. 8
  • The proposed method continually learns a conditional GAN together with a classifier from partially labeled data.Source: §6 · p. 8
  • Evaluations on various benchmarks support capturing and exploiting unlabeled data in classification and conditional generation; extending the framework to varied semi-supervised approaches remains future work.The paper also identifies implementation on Mindspore as further work.Source: §6 · p. 8

A. A Systematic Analysis of CL Methods

The analysis finds that existing continual-learning strategies leave a gap to joint training in semi-supervised settings, particularly because they do not continually exploit unlabeled data. Supervised replay helps, but unsupervised buffers do not reliably close that gap.

  • Sequential training underperforms joint training across four semi-supervised classifiers; weight regularization does not resolve the gap, while supervised replay alleviates forgetting but leaves an unlabeled-data gap.Source: A. A Systematic Analysis of CL Methods · p. 12, Figure 7 · p. 12
  • An unsupervised buffer of 10,000 images does not further improve performance, while buffers of 500 or 1,000 images cause severe overfitting.The 10,000-image buffer is twice the size of a typical generator used in ORDisCo.Source: A. A Systematic Analysis of CL Methods · p. 12, Figure 8 · p. 12
  • Replaying all labeled data outperforms selecting a fixed 200-sample supervised buffer by mean of feature, especially as more labeled data arrive.The comparison motivates replaying all labeled data in this study; selecting labeled samples effectively in SSCL is left as future work.Source: A. A Systematic Analysis of CL Methods · p. 12, Table 6 · p. 12
  • Generated samples cover the training-data distribution substantially better than an unsupervised memory buffer of comparable size.The buffer is around 10% of the training data, comparable to a typical generator's storage cost.Source: A. A Systematic Analysis of CL Methods · pp. 12–13

B. Implementation of SSL Classifiers and CL Methods

The implementation uses jointly trained SSL classifiers to validate baseline implementations and searches training epochs and regularization strengths for continual-learning baselines.

  • SSL classifiers are jointly trained for 200,000 iterations, and their SVHN performance with 1,000 labels is compared with published results.The implementations achieve comparable performance; slight variance may reflect different random seeds.Source: B. Implementation of SSL Classifiers… · p. 13
  • Sequential training uses 500 epochs per batch, selected as the best average among 100, 200, 500, and 1,000 epochs.Source: B. Implementation of SSL Classifiers… · p. 13
  • For EWC, SI, and MAS, the study searches regularization strengths and additionally tunes training epochs among 250, 500, and 750.The reported results use the best-performing trial for each method.Source: B. Implementation of SSL Classifiers… · p. 13

C. SSCL of New Instance on SVHN-3 and CIFAR10-13

ORDisCo performs better than other baselines on new-instance SSCL, especially with fewer incremental batches. Ablations support online generative replay and discrimination-consistency regularization, while a larger unsupervised memory buffer only slightly improves SSCL and remains inferior to ORDisCo.

  • ORDisCo achieves much better performance than other baselines on new-instance SSCL, particularly with fewer incremental batches, and quickly learns a usable model from limited data.The reported performance is summarized for SVHN-3 and CIFAR10-13; Fig. 9 presents averaged CIFAR10 accuracy.Source: C. SSCL of New Instance on SVHN-3 and… · p. 13, Figure 9 · p. 13
  • Online semi-supervised generative replay substantially improves SSCL, while discrimination-consistency regularization is more effective with fewer incremental batches and labels.The ablation caption specifies the regularization advantage for 15 batches.Source: C. SSCL of New Instance on SVHN-3 and… · p. 13, Table 9 · p. 14
  • A 4× larger UMB, with or without the unified classifier, only slightly improves SSCL and still significantly underperforms ORDisCo.The larger UMB contains around 20,000 images, compared with around 5,000 in the original UMB.Source: C. SSCL of New Instance on SVHN-3 and… · p. 14, Table 10 · p. 14

D. Conditional Samples

The section presents conditional samples generated by ORDisCo after incremental learning on SVHN-3 and CIFAR10-13.

  • Figure 10 shows ORDisCo conditional samples learned from incrementally provided partially labeled data, presenting its conditional samples in the setting of incremental learning.Source: D. Conditional Samples · p. 14

E. Complexity Analysis of Generative Replay

ORDisCo’s online generative replay has constant storage and linear time complexity, unlike offline replay strategies with higher time or storage costs. With a fixed 200-image supervised memory buffer, ORDisCo also outperforms the baselines.

E. Complexity Analysis of Generative Replay

  • ORDisCo’s online replay uses O(1) storage and O(B) time, versus Strategy 1’s O(B) storage and time and Strategy 2’s O(1) storage and O(B^2) time.If inference is required after each batch, Strategy 1’s time complexity is O(B^2); Strategy 2 also incurs more storage costs than ORDisCo.Source: E. Complexity Analysis of Generative… · pp. 14–15, Figure 11 · p. 15

F. Selecting SMB of a Fixed Size

  • With a fixed 200-image supervised memory buffer, ORDisCo substantially outperforms the baselines, while the smaller buffer causes more severe overfitting in SSCL.The buffer contains 20 images per class and is selected using the mean-of-feature approach.Source: Figure 12 · p. 15, F. Selecting SMB of a Fixed Size · p. 15

E. Complexity Analysis of Generative Replay

G. Hyperparameters of ORDisCo

ORDisCo uses α = 0.5 to balance discriminator predictions for fake data-label pairs from the generator and classifier, and sets λ = 0.001 to address forgetting of unlabeled data.

  • ORDisCo sets α = 0.5 to balance discriminator predictions for generator- and classifier-produced fake data-label pairs, and λ = 0.001 to address forgetting of unlabeled data.The authors grid-searched λ over 0.1, 0.01, and 0.001 based on training error.Source: G. Hyperparameters of ORDisCo · p. 15

Related papers

Loading 2101.00407v2…