PaperlayerView exact revision on arXiv

Source-linked AI summary

Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning

Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, Eric Xing

arXiv:2003.05438v5cs.CVcs.LGeess.IV

TL;DR

Unsupervised methods can overlook soft similarities between positive and negative pairs when input transformations are not reflected in their labels. Un-Mix couples image mixtures with label reassignment, and the paper reports consistent accuracy gains across methods and datasets. The authors also report improved generalization and transferability, with an additional forward pass as a limitation.

  • Problem

    Unsupervised methods often focus on frameworks and losses while overlooking input–label-space correlations that could represent fine-grained soft similarities between pairs.

    Source: §1 · p. 1

  • Method

    Un-Mix mixes input images and reassigns labels to jointly adjust input and loss spaces in unsupervised learning.

    Source: §1 · p. 2

  • Results

    Un-Mix yields consistent 1∼3% accuracy improvement across unsupervised methods and datasets, including CIFAR-10/100, STL-10, Tiny ImageNet, and ImageNet.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports better representation generalization and transferability across evaluated methods and downstream tasks, including object detection.

    Source: §5 · p. 7

    The method requires one additional forward pass for mixed images, with no extra back-propagation.

    Source: §4.6 · p. 7

Abstract

from arXiv · show

The recently advanced unsupervised learning approaches use the siamese-like framework to compare two "views" from the same image for learning representations. Making the two views distinctive is a core to guarantee that unsupervised methods can learn meaningful information. However, such frameworks are sometimes fragile on overfitting if the augmentations used for generating two views are not strong enough, causing the over-confident issue on the training data. This drawback hinders the model from learning subtle variance and fine-grained information. To address this, in this work we aim to involve the distance concept on label space in the unsupervised learning and let the model be aware of the soft degree of similarity between positive or negative pairs through mixing the input data space, to further work collaboratively for the input and loss spaces. Despite its conceptual simplicity, we show empirically that with the solution -- Unsupervised image mixtures (Un-Mix), we can learn subtler, more robust and generalized representations from the transformed input and corresponding new label space. Extensive experiments are conducted on CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet and standard ImageNet with popular unsupervised methods SimCLR, BYOL, MoCo V1&V2, SwAV, etc. Our proposed image mixture and label assignment strategy can obtain consistent improvement by 1~3% following exactly the same hyperparameters and training procedures of the base methods. Code is publicly available at https://github.com/szq0214/Un-Mix.

1. Introduction

Unsupervised methods often emphasize training frameworks and losses while overlooking how input and label spaces can jointly encode soft similarities. Un-Mix couples image mixtures with label reassignment, and the paper reports consistent accuracy gains across methods and datasets.

1. Introduction

  • Prior unsupervised methods focus on training frameworks and loss formulations, overlooking input–loss-space correlations that could encode fine-grained soft similarities between pairs.The paper describes contrastive learning as potentially over-confident, drawing an analogy to supervised learning with InfoNCE's pair classification and label smoothing.Source: §1 · p. 1
  • The paper frames label smoothing as encouraging equal probability for incorrect classes, which can explore more latent patterns and reduce overfitting.It connects this observation to the possibility that contrastive methods using InfoNCE can also encounter over-confidence.Source: §1 · p. 1
  • Unlike augmentation that changes input-space distance while leaving loss labels unchanged, image mixtures adjust semantic distance and unsupervised labels together.The paper says the degree of change is controllable and that the resulting representations have fewer variance directions across semantically similar classes.Source: §1 · p. 2
  • The paper reports that mixture-learned representations improve downstream object detection, including a 0.6% AP50 gain over baseline MoCo V2 for a 200-epoch model.That model is also reported to outperform the MoCo V2 800-epoch model.Source: §1 · p. 2
  • The proposed method obtains consistent accuracy improvement by 1∼3% across SimCLR, BYOL, MoCo V1&V2, SwAV, and other base approaches on CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet, and ImageNet.Source: §1 · p. 2

2. Related Work

Related work spans unsupervised feature learning, supervised prediction smoothing, and concurrent image-mixture approaches. The paper distinguishes Un-Mix by its self-mixture strategy within a minibatch and its intended integration into existing unsupervised frameworks.

2. Related Work

  • Unsupervised visual learning has used reconstruction objectives, handcrafted pretext tasks, contrastive feature learning, and clustering-based methods.Examples include autoencoders, colorization, jigsaw solving, momentum contrast, and clustering [Caron et al. 2018, 2020].Source: §2 · p. 2
  • Supervised label smoothing and hidden-state interpolation encourage softer predictions; prior work reports flatter representations, generalization, and robustness benefits.The cited studies concern supervised learning [Verma et al. 2019; M¨uller, Kornblith, and Hinton 2019; Shen et al. 2021].Source: §2 · p. 3
  • Concurrent i-Mix and MixCo also use image mixtures in unsupervised learning, while Un-Mix uses self-mixture within a minibatch for simpler integration.The paper says their mechanisms and implementations generally differ [Lee et al. 2021; Kim et al. 2020].Source: §2 · p. 3

3. Our Approach

Un-Mix mixes input images and adjusts unsupervised loss distances to represent degrees of similarity between samples. It supports both two-view and single-view mixture paradigms, with the single-view strategy used as the main approach.

3.1. Paradigms of Mixtures

  • Mixing both views preserves their similarity but can be unhelpful on ImageNet; mixing only one view is the main strategy, uses one additional forward pass, and yields more accuracy gain experimentally.The two-view strategy is effective on smaller datasets such as CIFAR and STL-10, but barely helpful on large-scale ImageNet.Source: §3.1 · p. 3

3.2. Image Mixture Strategies

  • Mixup interpolates two images, while Cutmix replaces selected image regions; the mixture ratio determines the softened distance assigned to positive pairs.In each iteration, one mixture operation is randomly chosen for all current samples with a predefined probability; with a memory bank, negative-pair distances remain unchanged.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Loss Functions

  • Pairing the first and last images, second and penultimate, and so on gives consistent soft distances across the minibatch and permits a weighted batch loss.For InfoNCE, the paper relates mutual information to a lower bound log(N) − L_N and states that adding mixtures enlarges N and tightens the bound.Source: §3.3 · pp. 4–5

3. Our Approach

  • The approach adds an extra loss from mixed-image pairs to the base method’s original loss, with the final objective combining original and mixed predictions.The base loss may be InfoNCE or an ℓ2 distance; mixed-pair losses measure fit between mixed samples and other views.Source: §3.3 · p. 4

4. Experiments

The experiments evaluate Un-Mix across unsupervised representation-learning benchmarks and downstream detection, using linear evaluation and transfer tasks.

4. Experiments

  • The experiments assess representation quality through linear evaluation and transferability through object detection on PASCAL VOC and COCO.The section evaluates unsupervised pretraining across several datasets and methods.Source: §4 · p. 5

4.1. Datasets

  • The evaluation spans CIFAR-10/100, STL-10, Tiny ImageNet, and ImageNet, ranging from 10 to 1,000 classes.The datasets differ in image size and labeled or unlabeled training data.Source: §4.1 · p. 5

4.2. Baseline Approaches

  • The tested baselines include contrastive methods MoCo V1/V2 and SimCLR, non-contrastive BYOL, clustering-based SwAV, and Whitening.The paper describes the respective approaches as contrastive, online/target prediction, clustering, and latent-space whitening.Source: §4.2 · p. 5

4.3. Implementation Details in Pre-training

  • Experiments use ResNet-18 outside ImageNet and ResNet-50 on ImageNet, with baseline-matched settings; mixture γ is 1.0 and P is 0.5 for non-ImageNet and 0 for ImageNet.The stated objective is to isolate mixture effects rather than optimize baseline hyperparameters.Source: §4.3 · p. 5

4.4. Linear Classification

  • Un-Mix yields 1 ∼3% gains on CIFAR-10/100, STL-10, and Tiny ImageNet; ImageNet MoCo V2 improves 1.1%, or 2.3% with multi-scale training.The gains use baseline hyperparameters; the paper notes that loss values are usually larger than the baselines, except for unstable BYOL.Source: §4.4 · p. 6

4.5. Downstream Tasks

  • The proposed mixtures consistently improve over the baseline on PASCAL VOC (trainval07+12 to test2007) and COCO (train2017 to val2017), with significant improvements reported on both datasets.All VOC models are fine-tuned for 24k iterations; COCO training uses 180K iterations and the Detectron2 coco R 50 C4 2× default schedule.Source: §4.5 · p. 7

4.6. Visualization and Analysis

  • Using 128-dimensional last-convolution-layer ResNet-18 features, t-SNE shows that mixture-model embeddings have more separate clusters than baseline MoCo, especially for classes 9, 8, and 1.The method’s information gain comes from mixed representations and mixture ratios, rather than longer training.Source: §4.6 · p. 7

5. Conclusion

The conclusion presents image mixtures with label reassignment as a strategy for unsupervised frameworks. The authors report improved representation capability across frameworks and transfer tasks, with implementation described as simple and its extra cost as reasonable.

5. Conclusion

  • The paper proposes image mixtures with corresponding label reassignment to flatten inputs and predictions across unsupervised frameworks.The conclusion says the strategy was investigated across various architectures.Source: §5 · p. 7
  • Experiments across SimCLR, BYOL, MoCo V1/V2, and downstream object detection show better generalization, transferability, and robustness across pretext tasks and frameworks.The conclusion includes contrastive and non-contrastive learning, with or without memory banks, and multi-scale training.Source: §5 · p. 7
  • The authors describe the method as simple to implement and as incurring reasonable extra cost.Source: §5 · p. 7

Appendix

The appendix lists supporting material on losses, implementation, training, mixture strategies, pseudocode, and visualization.

Appendix

  • Appendix sections A and B cover comparisons of loss functions and implementation details for unsupervised pretraining and linear evaluation.These sections correspond to image-mixture strategies, pretraining, and linear classification in the main paper.Source: Appendix · p. 10
  • Sections C and D discuss multi-scale training and comparisons of branch-mixture strategies.Section D compares both-branch and single-branch mixtures.Source: Appendix · p. 10
  • Sections E and F provide PyTorch-like mixture pseudocode and visualizations of ResNet-50 layers on ImageNet.Source: Appendix · p. 10

A. Comparison of Different Loss Functions

On standard ImageNet, combining mixed and original image pairs with Un-Mix gives the highest reported accuracy among the compared loss settings. Mixed pairs alone are close to the MoCo V2 baseline.

A. Comparison of Different Loss Functions

B. Implementation Details in Un-/self-supervised Pre-training and Linear Evaluation

The experiments use dataset-specific pretraining schedules and separate linear-evaluation protocols for non-ImageNet datasets and ImageNet.

B. Implementation Details in Un-/self-supervised Pre-training and Linear Evaluation

  • Non-ImageNet pretraining runs for 1,000 epochs on CIFAR-10, CIFAR-100, and Tiny ImageNet, and 2,000 epochs on STL-10.Learning rates are 3×10^-3 for CIFAR-10/100 and 2×10^-3 for Tiny ImageNet and STL-10; all use Adam and batch size 512.Source: B. Implementation Details in… · p. 10
  • Non-ImageNet linear evaluation runs for 500 epochs with Adam and no data augmentation, with the learning rate decaying from 10^-2 to 10^-6.Weight decay is 5 × 10^-6.Source: B. Implementation Details in… · p. 10
  • ImageNet linear evaluation uses 100 epochs, an initial learning rate of 30, and weight decay of 0.The learning rate is multiplied by 0.1 at epochs 60 and 80.Source: B. Implementation Details in… · p. 10

C. Enabling Multi-scale Training

The paper extends Un-Mix to multi-scale inputs, aggregating losses across scales while adapting memory-bank use to the different input sizes.

C. Enabling Multi-scale Training

  • Multi-scale training compares differently sized crops of an image to capture patterns from scenes or objects.The exploration is motivated by prior work on comparing random crops.Source: C. Enabling Multi-scale Training · p. 10
  • The mixture-strategy illustration distinguishes mixing both branches from mixing only IA, and frameworks with or without memory banks or negative pairs.Source: Figure 10 · p. 11
  • Top-1 classification comparisons cover ResNet-18 linear and 5-nearest-neighbor classifiers across datasets and loss functions.The table caption identifies the evaluation scope but provides no outcome values in the supplied passage.Source: Table 7 · p. 11
  • Different input sizes yield latent features of the same dimension after the MLP projection head, and losses from all scales are aggregated.The proposed extension uses a separate memory bank for each input size; sharing one bank performed poorly, which the authors conjecture is due to easy negative samples.Source: C. Enabling Multi-scale Training · p. 11

D. Comparison of Both Branch Mixtures and Single Branch Mixture

The comparison favors single-branch mixtures as the simpler design overall, although both mixture variants usually outperform the baseline. The pseudocode describes the central mixture implementation, with memory-bank-specific additions when needed.

D. Comparison of Both Branch Mixtures and Single Branch Mixture

  • Both-branch mixtures outperform single-branch mixtures for SimCLR across datasets, but underperform them with the other three baseline approaches.The authors conjecture that SimCLR benefits from increased negative-pair composition, while the other approaches do not use negative pairs; both variants are still better than baseline in most cases.Source: D. Comparison of Both Branch Mixtures… · pp. 11–12
  • Single-branch mixtures are described as the superior design because they require no additional mixture-ratio information, while both variants usually beat baseline results.Source: D. Comparison of Both Branch Mixtures… · p. 12
  • Un-Mix weight distributions have a wider value range and fewer elements near zero than baseline MoCo V2 in the reported comparison.The figure compares ResNet-50 layer weights on ImageNet.Source: D. Comparison of Both Branch Mixtures… · p. 12, Figure 12 · p. 12

E. A PyTorch-like [Paszke et al. 2019] Code for Our Mixture Strategy

  • The pseudocode gives the central mixture implementation; frameworks with memory banks require additional, non-core code to handle the bank mechanism.The passage says the central implementation remains the same.Source: E. A PyTorch-like [Paszke et al. 2019]… · p. 12

F. Visualizations of All Convolutional Layers

This section visualizes weight distributions across all convolutional layers of an ImageNet ResNet-50, comparing baseline MoCo V2 with the mixture-trained model.

F. Visualizations of All Convolutional Layers

  • Fig. 12 compares weight distributions from all ResNet-50 convolutional layers on ImageNet for baseline MoCo V2 and the mixture-trained model.The passage begins to describe an observed distribution difference but ends before stating it.Source: F. Visualizations of All Convolutional… · p. 12

Related papers

Loading 2003.05438v5…