PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving out-of-distribution generalization via multi-task self-supervised pretraining

Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Keskar, Richard Socher

arXiv:2003.13525v1cs.CVcs.LG

TL;DR

Domain generalization asks whether models can perform on target distributions unseen during training, where supervised pretraining may not transfer well. The paper combines self-supervised pretext tasks, including a new Gabor-response reconstruction task, then fine-tunes on source-domain labels. Self-supervised features match or exceed supervised features, especially under larger domain shifts.

  • Problem

    Domain generalization seeks models that perform on target domains drawn from distributions different from the source domains, while common ImageNet-based evaluations may involve domains with substantial similarity to ImageNet.

    Source: §1 · pp. 1–2

  • Method

    The method pretrains a shared feature extractor on multiple self-supervised tasks, including Gabor filter-response reconstruction, then fine-tunes it with supervised source-domain data.

    Source: §3.2 · p. 4, §3.3 · p. 5

  • Results

    Self-supervised features, especially combined pretext tasks, match or exceed fully supervised features for domain generalization; on PACS, the three-task combination surpasses the supervised pretrained baseline by 2% on average.

    Source: §1 · p. 2, PACS · p. 10, §5 · p. 15

  • Takeaways & Limitations

    Self-supervised features show improved localization and can be combined with domain-generalization methods such as IRM for further performance improvement.

    Source: §1 · p. 2, §4.4 · p. 13, §5 · p. 15

    Comparisons with other domain-generalization methods are not direct because they use different stopping criteria, architectures, or loss functions.

    Source: Baselines · p. 9

Abstract

from arXiv · show

Self-supervised feature representations have been shown to be useful for supervised classification, few-shot learning, and adversarial robustness. We show that features obtained using self-supervised learning are comparable to, or better than, supervised learning for domain generalization in computer vision. We introduce a new self-supervised pretext task of predicting responses to Gabor filter banks and demonstrate that multi-task learning of compatible pretext tasks improves domain generalization performance as compared to training individual tasks alone. Features learnt through self-supervision obtain better generalization to unseen domains when compared to their supervised counterpart when there is a larger domain shift between training and test distributions and even show better localization ability for objects of interest. Self-supervised feature representations can also be combined with other domain generalization methods to further boost performance.

1 Introduction

Domain generalization asks models to perform on target domains drawn from distributions different from the source domains, without relying on target-domain training data. The paper argues that multi-task self-supervised features can match or exceed supervised features, particularly under larger shifts.

1 Introduction

  • Domain generalization targets performance on a different-distribution target domain, requiring representations that generalize to unseen domains.Unlike the ERM assumption of identically distributed training and test data, domain generalization addresses shifted or unseen distributions.Source: §1 · p. 1
  • Standard VLCS and PACS evaluations can be relatively easy because their domains resemble ImageNet; supervised pretraining may also encode shape and texture biases that hinder transfer.VLCS and PACS contain fewer than 11,000 samples each, and share class-label similarities with ImageNet.Source: §1 · p. 2
  • Self-supervised learning trains encoders on unlabeled data through pretext tasks, and its features have matched supervised performance on several vision tasks.Prior work also reports improved adversarial robustness and out-of-distribution detection for difficult near-distribution examples.Source: §1 · p. 2
  • Self-supervised features match or exceed fully supervised features for domain generalization, with stronger results on LabelMe and Sketch and improved object localization.The paper also reports that self-supervised representations can be combined with other domain-generalization methods.Source: §1 · p. 2

2 Related work

Related work spans self-supervised representation learning and methods for generalization across distribution shifts. This paper positions selected combinations of self-supervised tasks as a route to comparable or better domain-generalization performance than supervised learning.

2 Related work

  • Prior self-supervised research combines pretext tasks or uses them to improve robustness and few-shot learning, but these examples address other downstream tasks.Doersch and Zisserman report gains from combined tasks on classification, detection, and depth prediction.Source: §2 · p. 3
  • Domain adaptation uses unlabeled target samples, whereas domain generalization learns representations without target-domain samples during training.Prior domain-generalization approaches include invariant representations, training-time domain shifts, and data augmentation.Source: §2 · p. 3
  • The paper's contribution is to show that selected combinations of self-supervised tasks can match or outperform supervised learning in domain generalization.The authors distinguish this approach from Carlucci et al., who combine supervised learning with an auxiliary jigsaw task.Source: §2 · p. 3

3 Methods

The method pretrains a shared feature extractor on self-supervised tasks, then fine-tunes it with source-domain labels for a target domain with a shared label space. Its pretext tasks include a new Gabor-response reconstruction task alongside Rotation and DeepCluster.

3.1 Problem Setting

  • Training and test domains share the label space, but the target distribution is distinct and supplies no training examples.The setup is homogeneous domain generalization, not unsupervised domain adaptation.Source: §3.1 · pp. 3–4

3.2 Self-Supervised Learning for Domain Generalization

  • The feature extractor is shared across self-supervised tasks, then fine-tuned with supervised ERM over source domains; its parameters update in both stages.The figure depicts multi-task pretraining followed by supervised fine-tuning.Source: §3.2 · pp. 4–5, Figure 1 · p. 4

3.3 Pretext Tasks

  • The new Gabor task reconstructs filter-bank responses to target low-level visual information, complementing Rotation and DeepCluster tasks at different feature levels.Gabor filters capture edges and texture; the paper characterizes Rotation as low/mid-level and DeepCluster as high-level.Source: §3.3 · p. 5

Gabor Filter Response Reconstruction:

  • The Gabor encoder-decoder reconstructs seven-direction filter responses using thresholded grayscale targets and average pixel-wise binary cross-entropy.The input image is subtracted from the filter-bank response before grayscale conversion and binary thresholding.Source: Gabor Filter Response Reconstruction · p. 5, Figure 2 · p. 5

Rotation:

  • Rotation trains a model to predict one of four input-image angles: 0°, 90°, 180°, or 270°.The paper frames this task as capturing information about pose, location, orientation, and object type.Source: Rotation · p. 6

DeepCluster:

  • DeepCluster trains the model to predict image-cluster assignments, recomputed with k-means each epoch, while Sobel filters remove color information.Its stated aim is to encourage features such as edges and shape.Source: DeepCluster · p. 6

3.4 Architecture Details

  • Experiments use an AlexNet-based backbone and flatten the last convolutional block into a 9,216-dimensional representation.The output tensor is 256 × 6 × 6; the task-specific heads vary by pretext task.Source: §3.4 · p. 6

Gabor Reconstruction:

  • The Gabor decoder mirrors the AlexNet encoder with transposed convolutions, while the downstream classifier maps the representation to the dataset's class count.The Gabor output layer maps to one channel; DeepCluster uses 10,000 clusters and reinitializes its final layer each epoch.Source: Gabor Reconstruction · pp. 6–7

4 Experiments

The experiments assess pretext-task performance and how task combinations affect it. Adding Gabor reconstruction to Rotation improves Rotation accuracy, while higher-level tasks also help Gabor reconstruction.

4 Experiments

  • The experiments ask how task combinations affect pretext performance, domain-shift generalization, cross-domain transfer, and performance when source-domain diversity is reduced.The questions cover both learned representations and the effects of changing the source data.Source: §4 · p. 7

4.1 Pretext Tasks

  • The study compares individual tasks and combinations trained by averaging normalized losses or sequentially fine-tuning tasks.In AVG, the feature extractor minimizes the average task loss; in FT, training shifts to a new task after the current task converges.Source: §4.1 · p. 7

Pretext Task Performance:

  • The supplementary table reports Rotation accuracy and Gabor reconstruction loss for individual and combined self-supervised tasks.Its caption identifies AVG as average-loss training and FT as sequential fine-tuning.Source: Table S1 · p. 8
  • Adding Gabor reconstruction increases Rotation accuracy by more than 7%, while Gabor reconstruction also benefits from adding Rotation and DeepCluster.Rotation accuracy decreases only slightly when Gabor reconstruction is added, indicating no strong conflict between those tasks.Source: Pretext Task Performance · p. 8, Table S1 · p. 8

4.2 Domain Generalization

The study evaluates self-supervised representations on VLCS and PACS using leave-one-domain-out fine-tuning, against random initialization and ImageNet-pretrained baselines. Comparisons with other domain-generalization methods are deferred because their evaluation setups differ.

4.2 Domain Generalization

  • Models are fine-tuned on three domains and evaluated on the held-out domain, with the best unseen-domain accuracy reported.Both benchmarks contain four datasets with the same classes.Source: §4.2 · p. 8
  • The baselines are a randomly initialized model and one pretrained on the full ILSVRC 2012 ImageNet training partition.ImageNet pretraining is described as a strong baseline because unseen-domain datasets often resemble ImageNet and share classes.Source: Baselines · p. 8

Baselines:

  • Comparisons with other domain-generalization methods are not direct because they use different stopping criteria, architectures, or loss combinations.Those comparisons are included in the supplement.Source: Baselines · p. 9

PACS:

On PACS, multi-task self-supervision improves domain-generalization performance, outperforming supervised pretraining on average and on three domains. The largest reported Sketch gain is associated with low- and mid-level pretext tasks.

PACS:

  • PACS tests generalization across Photo, Art painting, Cartoon, and Sketch, with seven classes per domain.The benchmark was proposed as a test bed with high overall domain shift from ImageNet.Source: PACS · p. 9
  • The three-task combination surpasses supervised pretraining by 2% on average and wins on Art painting, Cartoon, and Sketch.The PACS results table reports domain-generalization performance and indicates multi-task SSL outperforms supervised learning.Source: Table S2 · p. 9, PACS · p. 10
  • Combining Rotation with Gabor and DeepCluster boosts unseen-domain accuracy by 3.32% compared with Rotation alone.All single- and multi-task SSL approaches also significantly outperform random initialization.Source: PACS · pp. 9–10
  • 8.3% is the reported R + G + DC improvement over the supervised model on Sketch.The authors suggest Rotation and Gabor reconstruction may capture edges and shapes without texture; adding these low-level tasks also recovers performance lost by DeepCluster alone.Source: PACS · p. 10

VLCS:

On VLCS, the best multi-task SSL model nearly matches supervised pretraining on average, while performance varies across datasets. Qualitative comparisons report stronger object localization by SSL on PACS and VLCS examples.

VLCS:

  • VLCS combines natural images from four datasets, split into training and validation sets with 80% and 20% of examples, respectively.Its five classes are bird, car, chair, dog, and person.Source: VLCS · p. 10
  • R + DC averages 69.78% across VLCS datasets, above random initialization at 59.86% and near the fully supervised model at 70.69%.The authors identify R + DC as the best multi-task self-supervised model.Source: VLCS · p. 10
  • Five of six SSL strategies beat the fully supervised model on LabelMe, but supervised learning slightly leads on PASCAL VOC, Caltech101, and SUN09.R + DC performs best overall among SSL tasks, yet multi-task SSL does not uniformly outperform individual tasks on VLCS.Source: VLCS · p. 11

Qualitative Differences:

  • GradCAM comparisons show SSL focusing on object parts in PACS and localizing small class-relevant objects in VLCS, while supervised predictions attend more to background or distractors.For VLCS LabelMe examples, the authors report this pattern for bird, car, and person classes.Source: Qualitative Differences · p. 11

4.3 Cross-domain Transfer

The cross-domain transfer experiments fine-tune on one source domain and evaluate on another. When the source resembles ImageNet, self-supervised pretraining with unlabeled images matches or exceeds supervised pretraining, including for PACS Sketch.

4.3 Cross-domain Transfer

  • The experiments use one source domain for fine-tuning and evaluate on a different target domain.For VLCS, LabelMe is the target; for PACS, Sketch is the target, with each remaining domain used in turn as source.Source: §4.3 · p. 11
  • When source domains resemble ImageNet—Caltech101 for VLCS and Photo for PACS—self-supervised features generalize better than supervised features.The reported comparison uses roughly 1.2 million unlabeled pretraining images and 1,500 labeled fine-tuning images versus the same approximate counts labeled.Source: §4.3 · p. 12
  • With a single source domain, self-supervised pretraining outperforms the supervised baseline for PACS Sketch as target.The figure identifies this result as transfer using unlabeled images.Source: Figure 7 · p. 13
  • The authors conclude self-supervision can mitigate effects of limited visual diversity between pretraining and fine-tuning datasets.Source: §4.3 · p. 13

4.4 Combination with Other Domain Generalization Methods

The paper tests self-supervised pretraining as an initialization for Invariant Risk Minimization (IRM). Combining IRM with self-supervision improves average out-of-distribution performance over supervised learning on PACS and VLCS.

4.4 Combination with Other Domain Generalization Methods

  • The authors fine-tune pretrained models with IRM to seek a representation whose best predictor is shared across training domains.They use the same hyperparameters as in the colored MNIST experiments.Source: §4.4 · p. 13
  • For VLCS transfer to LabelMe, self-supervised learning with unlabeled images is comparable to or outperforms the supervised baseline.Source: Figure 8 · p. 14
  • IRM with self-supervised pretraining outperforms supervised learning on average across PACS and VLCS.Table S4 reports IRM domain-generalization performance; the text summarizes the average comparison across both benchmarks.Source: §4.4 · p. 14, Table S4 · p. 14
  • With IRM, self-supervision improves best target accuracy on 3 of 4 PACS domains and outperforms supervision on 2 VLCS targets, including Caltech101.In earlier ERM experiments, supervised learning was superior to self-supervision on Caltech101.Source: §4.4 · p. 14

5 Conclusion

The conclusion finds that self-supervised representations, particularly those learned from multiple pretext tasks, can match or exceed fully supervised feature extractors for domain generalization. It also reports improved localization and potential gains from combining self-supervision with other domain-generalization techniques.

5 Conclusion

  • Multi-task self-supervised representations can match or exceed fully supervised feature extractors on domain generalization and improve localization.Source: §5 · p. 15
  • Self-supervision can be combined with other techniques aimed at learning representations amenable to domain generalization.Source: §5 · p. 15

Supplementary Material

The supplementary material compares the paper's models with prior domain-generalization approaches and gives implementation details for the Gabor filter bank.

Comparing with domain generalization strategies:

  • The comparison with CIDDG and MLDG is not directly comparable because architectures and training budgets differ, though the authors report their best models as comparable to supervised methods with additional domain-generalization techniques.The supplementary tables cover VLCS and PACS; the comparison also includes JiGen and supervised DeepAll baselines.Source: Supplementary Material · p. 18, Table S5 · p. 18, Table S6 · p. 18

Supplementary Material

Details of Gabor filter bank hyperparameters:

Loading 2003.13525v1…