PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Domain-Agnostic Contrastive Learning

Vikas Verma, Minh-Thang Luong, Kenji Kawaguchi, Hieu Pham, Quoc V. Le

arXiv:2011.04419v2cs.LGcs.AIstat.ML

TL;DR

Contrastive learning often relies on domain-specific augmentations that may be unavailable in some modalities. DACL uses Mixup-noise at the input or hidden-state level to create contrastive examples, and the paper reports results across tabular data, images, and graphs, including improvements when combined with image-specific augmentation.

  • Problem

    Contrastive learning needs similar and dissimilar examples, but domain-specific augmentations may not be readily available for modalities such as graphs or tabular data.

    Source: §1 · pp. 1–2

  • Method

    DACL creates positive and negative examples with Mixup-noise, mixing samples at the input or hidden-state level and extending the procedure with additional Mixup forms.

    Source: §1 · p. 2

  • Results

    Across the reported domains, DACL is a viable option where augmentations are unavailable; Mixup-noise also improves over the SimCLR baseline on CIFAR10, CIFAR100, and ImageNet.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The paper presents Mixup-noise as a domain-agnostic alternative and reports that it can complement image-specific augmentations in visual representation learning.

    Source: §1 · p. 2

    The theoretical analysis omits the projection head and other forms of Mixup-noise.

    Source: §4 · p. 4

Abstract

from arXiv · show

Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a novel domain-agnostic approach to contrastive learning, named DACL, that is applicable to domains where invariances, and thus, data augmentation techniques, are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning. Finally, we theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach.

1. Introduction

Contrastive learning needs similar and dissimilar examples, but domain-specific augmentations may be unavailable outside familiar modalities; DACL uses Mixup-noise to construct them and is evaluated across domains.

1. Introduction

  • Domain-specific semantic-preserving augmentations such as flipping, rotation, jittering, and cropping may be unavailable for graph or tabular data.Source: §1 · p. 2
  • DACL uses Mixup-noise to create similar and dissimilar examples by mixing samples at the input or hidden-state level.The paper also analyzes why Mixup-noise works better than Gaussian-noise.Source: §1 · p. 2
  • The paper proposes that geometric- and binary-Mixup can further improve DACL, and extends hidden-state mixing to domains with non-fixed topology, such as graphs.Source: §1 · p. 2
  • DACL's Mixup-noise augmentation complements image-specific augmentations, improving over the SimCLR baseline on CIFAR10, CIFAR100, and ImageNet.Source: §1 · p. 2

2. Contrastive Learning : Problem Definition

Contrastive learning trains representations by distinguishing semantically similar examples from dissimilar ones, and this paper uses InfoNCE to express that objective.

2. Contrastive Learning : Problem Definition

  • Contrastive learning aims to bring an anchor and its positive sample closer in embedding space than the anchor and its negative sample.Source: §2 · p. 2
  • The anchor, positive, and negative are respectively a reference sample, a semantically similar sample, and a dissimilar sample.Source: §2 · p. 2
  • The paper uses InfoNCE, which maximizes similarity for a positive pair and minimizes similarity for negative pairs.Source: §2 · p. 2

3. Domain-Agnostic Contrastive Learning with Mixup

DACL constructs positive examples by interpolating samples, either in input space or in fixed-length hidden representations, and uses these examples in contrastive learning.

3. Domain-Agnostic Contrastive Learning with Mixup

  • For input-space Mixup, DACL interpolates an anchor with a randomly chosen sample, choosing λ so the positive remains closer to the anchor.Other positive samples in the training batch serve as negatives, as in SimCLR.Source: §3 · p. 2
  • For sequences, trees, and graphs with non-fixed topology, DACL mixes fixed-length hidden representations because input-space Mixup is not feasible.Source: §3 · p. 2
  • Mixup-noise perturbs a sample toward another sample in the data distribution, unlike independent Gaussian-noise; the authors argue this can prevent a trivial averaging transformation.Source: §3 · pp. 2–3
  • The DACL procedure follows SimCLR's encoder, projection-head, and normalized temperature-scaled InfoNCE setup, with hidden-state mixing substituted for input mixing when needed.Source: §3 · p. 3

3.1. Additional Forms of Mixup-Based Noise

  • DACL+ randomly selects Linear-, Geometric-, or Binary-Mixup to create both positive samples for each input.Geometric-Mixup uses a weighted-geometric mean; Binary-Mixup swaps input elements using a binary mask.Source: §3.1 · p. 3

4. Theoretical Analysis

The section analyzes Mixup-noise and Gaussian-noise contrastive learning for binary classification, relating contrastive optimization to classification loss and comparing their induced regularization.

4. Theoretical Analysis

  • The contrastive loss relates to classification loss at a perturbed positive sample, with an error term that increases as the negative example is more likely to share the positive example’s label.The relationship is analyzed for binary classification and compared across Mixup-noise and Gaussian-noise.Source: §4 · pp. 3–4
  • Under Theorem 2’s assumptions, contrastive training approximately minimizes classification loss at the original input with additional gradient regularization when a training example is correctly classified.The stated assumptions hold for linear models and feedforward ReLU/max-pooling networks without skip connections; the paper says this partially explains benefits of training after correct classification.Source: §4 · pp. 4–5
  • Mixup-noise adds regularization on the gradient in the input-covariance direction, while the Gaussian-noise version does not.The analysis identifies this as an implicit regularization effect that can reduce model-class complexity along the data manifold.Source: §4 · p. 5
  • When the input distribution lies on a low-dimensional manifold, the analysis shows Mixup-noise has an advantage over Gaussian-noise and can reduce model-class complexity along that manifold.The paper characterizes the regularization as data-adaptive and says it can promote generalization.Source: §4 · pp. 4–5
  • The analysis also applies when positive samples mix hidden representations, yielding implicit regularization along the hidden-layer data manifold.The paper presents input-space or hidden-space Mixup-noise as promoting generalization in the corresponding data manifold.Source: §4 · p. 5

5. Experiments

The experiments evaluate DACL on tabular, image, and graph data using linear evaluation. DACL outperforms Gaussian-noise methods on tabular and image data, complements SimCLR across image datasets, and closely matches InfoGraph on graph datasets.

5. Experiments

  • The experiments evaluate learned representations with a linear classifier trained on a frozen encoder, using test accuracy as a proxy for representation quality.The projection head is discarded for linear evaluation.Source: §5 · p. 5

5.1. Tabular Data

  • On tabular proxy datasets, DACL beats Gaussian-noise contrastive learning, DACL+ improves further, and a linear classifier on DACL representations outperforms full-network supervised training.Fashion-MNIST and CIFAR-10 are permuted and flattened into vectors for these experiments.Source: §5.1 · p. 6, Table 1 · p. 6

5.2. Image Data

  • DACL outperforms Gaussian-noise contrastive learning on CIFAR-10/100 and ImageNet; DACL+ improves further, and combining DACL with SimCLR improves SimCLR across all datasets.DACL alone falls short of image-augmentation methods such as SimCLR.Source: §5.2 · pp. 6–7, Table 2 · p. 7, Table 3 · p. 7
  • The ImageNet SimCLR reproduction differs from Chen et al. (2020b) because these experiments use 100 pretraining epochs rather than 1000.The image experiments use CIFAR-10, CIFAR-100, and ImageNet benchmarks.Source: §5.2 · p. 6

5.3. Graph-Structured Data

  • On six graph-classification benchmarks, DACL matches InfoGraph within standard deviations and has higher mean accuracy on four datasets, despite using no domain knowledge to formulate its contrastive loss.Experiments apply Mixup-noise to encoder outputs; intermediate-layer experiments are left for future work.Source: §5.3 · p. 7

6. Related Work

Related work surveys domain-specific pretext tasks and contrastive sample-construction strategies, then distinguishes DACL from prior Mixup-based approaches through its theory, variants, and cross-domain evaluation.

6. Related Work

  • Pretext tasks include language, vision, and graph objectives, most of which are domain-specific; language modeling is noted as an exception spanning NLP and vision.Examples include word prediction, image rotation prediction, and graph-context prediction.Source: §6 · p. 8
  • Contrastive methods based on domain-specific augmentation, local/global context, or sequence ordering have limits on graphs or tabular data where those structures may be unavailable [Chopra et al., 2005; Hadsell et al., 2006; Ye et al., 2019; He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Hjelm et al., 2019; Sun et al., 2020; Veličković et al., 2019; Bachman et al., 2019; Trinh et al., 2019; Oord et al., 2018; Hénaff et al., 2019].Domain-augmentation methods are described as inapplicable where semantic-preserving augmentation is absent; context methods require local/global context, and sequential methods require ordered data.Source: §6 · p. 8
  • Unlike prior Mixup-based contrastive methods, this work analyzes Mixup-noise theoretically, proposes complementary forms, and evaluates across domains.The paper also notes that Mixup-based contrastive methods avoid the additional gradient computation required by an adversarial-direction method.Source: §6 · pp. 8–9

7. Discussion and Future Work

The discussion presents DACL as a viable option where domain-specific augmentations are unavailable, while distinguishing its outcomes across tabular, graph, and image tasks. It also identifies limits in the theoretical setting and directions for extending the experiments.

7. Discussion and Future Work

  • For tabular data, DACL and DACL+ achieve better test accuracy than fully supervised neural-network training.The discussion frames this as a result for tabular data.Source: §7 · p. 9
  • For graph classification, DACL is on par with the mutual-information maximization method of Sun et al. (2020).This is the discussion's comparison with a recently proposed graph contrastive-learning method.Source: §7 · p. 9
  • On image datasets, DACL falls short of methods using image-specific augmentations, but its Mixup-noise can complement those augmentations.The cited examples include random cropping, horizontal flipping, and color distortions.Source: §7 · p. 9
  • The theoretical analysis is limited to binary classification; the examined Mixup-noise forms interpolate between two samples.Suggested future work includes extending the analysis to multiclass settings and testing mixtures of more than two samples or a learned mixing policy.Source: §7 · p. 9

A. Additional Discussion on Theoretical Analysis

The appendix explains how the theoretical distribution can represent finite empirical samples or include augmentations and pair-generation choices. It also connects gradient regularization in ReLU networks to path-weight norms used in generalization bounds.

A. Additional Discussion on Theoretical Analysis

  • For a finite training set of n samples, the theorem's distribution D can be instantiated as the empirical measure; it can also incorporate additional augmentations and positive/negative-pair generation choices.The appendix states that D is otherwise arbitrary.Source: A. Additional Discussion on… · p. 13
  • For ReLU networks, the Jacobian is expressed through layerwise weight and activation-derivative products, linking the gradient norm to a sum of path weights.The text says regularizing ∥∇f(x)∥ tends to promote generalization because it corresponds to the path-weight norm used in generalization error bounds (Kawaguchi et al., 2017).Source: A. Additional Discussion on… · p. 13

B. Proof

  • The proof discussion notes that, for each input, examples generated from other original data points are treated as negatives, while those generated from that input are positives.The negative-to-positive ratio increases with the number of original data points, and the proofs apply for any number of original points.Source: B. Proof · p. 13

B.1. Proof of Theorem 1

The proof rewrites the contrastive loss using the difference between positive- and negative-pair similarities, and proceeds from an idealized loss–classification relationship to a practical setting.

B.1. Proof of Theorem 1

  • The contrastive loss is rewritten as a softplus-style expression in the difference between positive- and negative-pair similarities.The displayed rewrite uses the difference sim[h(x+), h(x++)] − sim[h(x+), h(x−)].Source: §B.1 · p. 14
  • The proof first establishes the contrastive-loss and classification-loss relationship under an ideal situation, then moves to the practical situation.The practical-case transition follows the ideal-case relationship.Source: §B.1 · pp. 14, 16
  • The ideal-case setup takes two perturbed versions of x as positives and a perturbed sample from a different-label example as the negative, using a normalized inner-product similarity.The stated conditions include y ≠ ȳ.Source: §B.1 · p. 15

B.2. Proof of Theorem 2

The proof relates contrastive pre-training to classification loss with regularization, then analyzes Mixup and Gaussian-noise cases under stated assumptions.

B.2. Proof of Theorem 2

  • Contrastive pre-training is related to minimizing standard classification loss while regularizing loss changes along the perturbation direction δ(x, ˜x).The argument begins from the loss-level relationship in Lemma 6.Source: §B.2 · p. 18
  • The proof next analyzes the model level and introduces Lemma 7 under existence of ∇f(x) and ∇2f(x), with f(x) = ∇f(x)⊤x and ∇2f(x) = 0.For ψ, the supplied passage states ψ′(·) = ψ(·)(1 − ψ(·)) > 0.Source: §B.2 · p. 18
  • The Mixup case is related to standard classification loss plus a regularization term on ∥∇f(x)∥.Lemma 8 assumes a fixed x, differentiability conditions, f(x) = ∇f(x)⊤x, ∇2f(x) = 0, E˜x[˜x] = 0, and a label-dependent sign condition.Source: §B.2 · p. 19
  • The Gaussian-noise case sets δ(x, ˜x) = ˜x ∼ N(0, σ2I); Lemmas 8–9 are then combined to obtain Theorem 2.Lemma 9 repeats the fixed-point and differentiability conditions stated in its passage.Source: §B.2 · pp. 20–21

B.3. Proof of Theorem 3

The proof invokes a standard probability bound and proceeds by bounding the Rademacher complexity of F(mix).

B.3. Proof of Theorem 3

  • The proof applies a standard result from Bartlett & Mendelson (2002) to obtain a bound holding with probability at least 1 − δ.It then bounds the Rademacher complexity Rn(F(mix)) to establish the desired statement.Source: §B.3 · p. 21

C. Best Hyperparameter Values for Various Experiments

This section presents best hyperparameter values for tabular, CIFAR10/100, and ImageNet experiments, and reports ranges in which the method works well.

C. Best Hyperparameter Values for Various Experiments

  • The method works well across α ∈ [0.6, 0.9] and ρ ∈ [0.1, 0.5].The section points to Tables 5–7 for the best values used in the experiments.Source: C. Best Hyperparameter Values for… · p. 22
  • Table 5 presents best hyperparameter values for the tabular-data experiments.The table is associated with the experiments reported in Table 1.Source: Table 5 · p. 22
  • Table 6 presents best hyperparameter values for the CIFAR10/100 experiments.The table is associated with the experiments reported in Table 2.Source: Table 6 · p. 23
  • Table 7 presents best hyperparameter values for the ImageNet experiments.The table is associated with the experiments reported in Table 3.Source: Table 7 · p. 23

Related papers

Loading 2011.04419v2…