PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Consistency Regularization for Semi-Supervised Learning

Yue Fan, Anna Kukleva, Bernt Schiele

arXiv:2112.05825v1cs.CV

TL;DR

Consistency regularization promotes invariant predictions under augmentation, but the paper questions whether invariance is suitable for perceptually diverse strong augmentations. FeatDistLoss enforces same-class consistency at the classifier level while increasing feature distances between differently augmented views to encourage equivariant representations. CR-Match reports state-of-the-art results across CIFAR-10, CIFAR-100, SVHN, STL-10, and Mini-ImageNet; on CIFAR-100 with 4 annotated examples per class, its 39.45% error rate is 4.83% better than the second best 44.28%.

  • Problem

    Consistency regularization promotes invariant predictions under augmentation, but the paper questions whether invariance is suitable for perceptually diverse strong augmentations.

    Source: §1 · p. 1

  • Method

    FeatDistLoss enforces same-class consistency at the classifier level while increasing feature distances between differently augmented views to encourage equivariant representations.

    Source: §3.1 · p. 5, §5 · p. 13

  • Results

    CR-Match reports state-of-the-art results across CIFAR-10, CIFAR-100, SVHN, STL-10, and Mini-ImageNet; on CIFAR-100 with 4 annotated examples per class, its 39.45% error rate is 4.83% better than the second best 44.28%.

    Source: §1 · p. 2, §5 · p. 13

  • Takeaways & Limitations

    CR-Match defines a new state-of-the-art on CIFAR-10, CIFAR-100, SVHN, STL-10, and Mini-ImageNet; on CIFAR-100 with 4 annotated examples per class, its error rate is 39.45%, 4.83% better than the second best (44.28%).

    Source: §5 · p. 13

Abstract

from arXiv · show

Consistency regularization is one of the most widely-used techniques for semi-supervised learning (SSL). Generally, the aim is to train a model that is invariant to various data augmentations. In this paper, we revisit this idea and find that enforcing invariance by decreasing distances between features from differently augmented images leads to improved performance. However, encouraging equivariance instead, by increasing the feature distance, further improves performance. To this end, we propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss, that imposes consistency and equivariance on the classifier and the feature level, respectively. Experimental results show that our model defines a new state of the art for various datasets and settings and outperforms previous work by a significant margin, particularly in low data regimes. Extensive experiments are conducted to analyze the method, and the code will be published.

1 Introduction

Because labeled data can be unavailable or costly, semi-supervised learning uses unlabeled data; this paper questions whether invariance to strong augmentations is always desirable. It proposes FeatDistLoss and reports that encouraging equivariance can improve generalization, especially with few labels.

1 Introduction

  • Semi-supervised learning methods use unlabeled data because large-scale annotated datasets are needed for state-of-the-art performance and labels may be unavailable or costly.Source: §1 · p. 1
  • Consistency regularization trains predictions to remain invariant to input perturbations, but the paper questions this goal for perceptually diverse strong augmentations.The paper proposes improving augmentation-based consistency regularization with FeatDistLoss.Source: §1 · p. 1
  • FeatDistLoss regulates distances between features of differently augmented versions of the same class to encourage invariance or equivariance.Equivariance pushes same-class representations farther apart, covering more feature space, whereas invariance makes them more compact.Source: §1 · pp. 1–2
  • The paper reports that increasing equivariance between differently augmented views can improve performance, especially when few labels are available per class.Source: §1 · p. 2
  • CR-Match combines FeatDistLoss with other techniques and reports new state-of-the-art results across standard SSL benchmarks, particularly in low-data regimes.The benchmarks named include CIFAR-10, CIFAR-100, SVHN, STL-10, and Mini-Imagenet.Source: §1 · p. 2

2 Related Work

Related work spans consistency regularization and other ways to exploit unlabeled data, including augmentation, pseudo-labeling, generative models, graph methods, and self-supervision. The paper distinguishes its feature-space notion of equivariance from equivariant architectures that preserve transformations predictably.

2 Related Work

  • Consistency regularization seeks consistent predictions for perturbed inputs, while SSL also includes pseudo-labeling, generative models, and graph-based methods.Prior perturbation approaches include model averaging, random max-pooling, dropout, advanced augmentation, and MixUp.Source: §2 · pp. 2–3
  • Weak and strong augmentation can add noise to unlabeled data, while pseudo-labeling can assign labels from weak views and train on strongly augmented images.Source: §2 · p. 3
  • Some SSL methods use self-supervised pretraining, whereas others add an auxiliary rotation-prediction loss during SSL training.The cited pretraining methods normally use several training phases and involve many hyperparameters.Source: §2 · p. 3
  • Here, equivariance means differently augmented versions of one image occupy different feature-space points despite sharing a semantic label, unlike architectures that transform features predictably.The paper says object location or orientation is more predictable when features are pushed apart.Source: §2 · p. 3

3 CR-Match

CR-Match extends consistency regularization with FeatDistLoss, pairing same-label predictions across augmentations with feature-level control of invariance or equivariance. Its unlabeled-data objective combines this loss with confidence thresholding and rotation prediction alongside supervised training.

3.1 Feature Distance Loss

  • FeatDistLoss enforces same-label predictions across augmentations while either pulling their features together or pushing them apart.It applies a feature-distance term and a pseudo-label cross-entropy term; feature distances are computed in a projected low-dimensional space.Source: §3.1 · p. 5
  • Only unlabeled images whose weak-view prediction exceeds a confidence threshold contribute to the unlabeled loss.The confidence thresholding mechanism provides a curriculum for balancing labeled and unlabeled losses.Source: §3.1 · p. 6
  • Increasing feature distance consistently yields better generalization than decreasing it, and the authors attribute this to equivariance between differently augmented views.The final model uses strong- and weak-view features while maintaining the same semantic label.Source: §3.1 · pp. 5–6

3.2 Overall CR-Match

  • CR-Match combines supervised training, unlabeled FeatDistLoss, and a self-supervised rotation-prediction loss.The unlabeled loss regularizes features from weakly and strongly augmented images, while rotation prediction supplies an additional supervisory signal.Source: §3.2 · pp. 6–7

3.3 Implementation Details

  • CR-Match uses weak random cropping and mirroring, and strong augmentation combining RandAugment and CutOut.At each step, two strong-augmentation operations are sampled with randomly selected magnitudes.Source: §3.3 · p. 7

4 Experimental Results

CR-Match improves on prior methods across the reported benchmark settings, with especially large gains in low-label regimes. The analyses examine FeatDistLoss design choices and find advantages for increasing feature distances and using weak–strong augmentation pairs.

4.1 Main Results

  • With 4 labeled examples per class on CIFAR-100, CR-Match beats FixMatch by 9.40% and the second-best method by 4.83% in absolute error rate.The paper also reports state-of-the-art results across its settings on CIFAR-10, CIFAR-100, SVHN, STL-10, and Mini-ImageNet.Source: §4.1 · pp. 8–9, Table 1 · p. 8, Table 2 · p. 8

4.2 Ablation Study

  • Across CIFAR-10, CIFAR-100, SVHN, and Mini-ImageNet ablations, RotNet and FeatDistLoss both contribute, but their combination is not always best.The combination outperforms either individual loss on Mini-ImageNet, CIFAR-100, and SVHN; on CIFAR-10, FeatDistLoss alone outperforms the combination.Source: Table 3 · p. 9, §4.2 · p. 10

4.3 Influence of Feature Distance Loss

  • Increasing feature distances outperforms decreasing them, while both approaches improve on classifier-level consistency alone.The authors conjecture that decreasing distances can discard augmentation information such as rotation or translation, whereas increasing distances encourages equivariance.Source: §4.3 · p. 10, Table 4 · p. 11
  • CR-Equiv features make augmentation transformations more predictable than CR-Inv features, including for translation, scaling, rotation, and color jittering.For distinguishing strongly from weakly augmented features, the reported test errors are 0.27% for CR-Equiv and 46.18% for CR-Inv.Source: §4.3 · p. 11, Table 4 · p. 11
  • Weak–strong augmentation pairs outperform weak–weak and strong–strong pairs for both CR-Inv and CR-Equiv, and CR-Equiv generalizes better in all three settings.With weak–strong pairs, CR-Equiv outperforms CR-Inv by 1.46%.Source: §4.3 · pp. 11–12, Table 5 · p. 12
  • Removing the linear projection or replacing it with an MLP worsens error, as does removing the confidence threshold for applying FeatDistLoss.The reported error rises from 45.52% to 48.37% without the linear layer, to 47.52% with an MLP, and to 46.94% without the threshold.Source: §4.3 · p. 12, Table 6 · p. 12

5 Conclusion

FeatDistLoss combines classifier-level consistency with feature-level equivariance, increasing distances between features from differently augmented images. The resulting framework reports state-of-the-art performance across five benchmarks, with particularly large gains in low-data settings.

5 Conclusion

  • FeatDistLoss imposes classifier-level consistency while increasing feature distances between differently augmented images, encouraging more equivariant representations.The paper links this representation to improved classification boundaries and a more robust model.Source: §5 · p. 13
  • The framework reports new state-of-the-art results on CIFAR-10, CIFAR-100, SVHN, STL-10, and Mini-ImageNet.Source: §5 · p. 13
  • On CIFAR-100 with 4 annotated examples per class, the error rate is 39.45%, versus 44.28% for the second-best method.Source: §5 · p. 13

Related papers

Loading 2112.05825v1…