Self-Supervised Learning Across Domains

1 INTRODUCTION

• Existing strategies rely largely on supervised learning, while self-supervised approaches often require tailored architectures and have rarely considered paintings or sketches across domains.The introduction notes that self-supervised methods have mainly been applied to real-world photos.Source: §1 · p. 1
• The approach is motivated by human visual learning, in which categorization and learning regularities such as object-part relations occur together.The passage cites infants’ and toddlers’ simultaneous learning of categories and visual regularities.Source: §1 · p. 1
• The work jointly learns object recognition from supervised data and domain generalization from self-supervised information about spatial relations among image parts.The introduction presents this as a multi-task approach to object recognition across domains.Source: §1 · p. 1, Figure 1 · p. 1
• Jigsaw learning treats patch reordering as image-level classification, allowing object recognition and patch prediction to share a backbone without architectural changes.The jigsaw procedure splits images into patches, shuffles them, and classifies recomposed images; the figure describes a 3 × 3 grid and selected permutations.Source: §1 · p. 2, Figure 2 · p. 2
• The study evaluates rotation recognition and jigsaw puzzles alongside supervised learning, with ablations and experiments in single-source, multi-source, predictive, and partial adaptation.It also analyzes successful and failure cases and extends prior work on predictive and partial domain adaptation.Source: §1 · p. 2

2 RELATED WORK

• Self-supervised learning generates surrogate labels from inherent image attributes, then transfers learned initial layers to a supervised downstream task.Pretext tasks may use geometric transformations, patch order, external sensory cues, or video regularities.Source: §2 · p. 2
• Domain generalization trains without access to the target, whereas domain adaptation uses labeled source data and unlabeled target data.In multi-source adaptation, source-domain labels may be unknown.Source: §2 · p. 2
• Domain-shift methods include feature-level representation learning and adversarial alignment, model-level training changes, and data-level image synthesis.The reviewed approaches also include domain-alignment or aggregation layers and low-rank parameter decomposition.Source: §2 · pp. 2–3
• Predictive domain adaptation uses labeled source and auxiliary unlabeled domains plus metadata to compose a model without target data; partial adaptation handles targets containing only some source classes.Partial adaptation methods may reweight source samples or use classifier disagreement or feature-norm matching.Source: §2 · p. 3
• The authors identify self-supervision for domain adaptation and domain generalization as uninvestigated in the reviewed literature.Their work studies self-supervised learning across domains by combining supervised and self-supervised knowledge in a multi-task framework.Source: §2 · pp. 2–3

3 METHOD

• Rotation recognition predicts one of four image orientations, while jigsaw prediction reorders shuffled 3 × 3 patches; both tasks share a backbone with supervised classification through separate output branches.The transformed images receive self-supervised labels, and the auxiliary task is label-agnostic, so it can use supervised and unsupervised domains.Source: §3 · p. 3

3.1 Domain Generalization

• For domain generalization, the objective combines source-label classification with self-supervised prediction on transformed images, while target-time predictions use only the object classifier.The self-supervised loss also applies to original images; rotated or shuffled images do not contribute to supervised classification loss.Source: §3.1 · p. 3, §3 · p. 3

3.2 Domain Adaptation

• For domain adaptation, self-supervised tasks can use unlabeled target images, alongside entropy minimization on target predictions, whose success depends on moderate domain shift.The text describes entropy minimization as encouraging the decision boundary to pass through low-density target areas, while warning that larger shifts risk incorrect pseudo-labels.Source: §3.2 · p. 3

3.3 Partial Domain Adaptation

• For partial domain adaptation, where target classes are a subset of source classes, the method drops source self-supervision and weights source classes using target predictions, with an adversarial domain classifier.The authors state that dropping the source self-supervised term lets that module focus on the target without involving extra source classes; Fig. 3 diagrams the weighting and adversarial components.Source: §3.3 · pp. 3–4, Figure 3 · p. 4

3.4 Implementation details

• The implementation supports AlexNet- or ResNet-style backbones, task-specific fully connected heads, and end-to-end fine-tuning of pretrained feature layers.For domain generalization, α weights the self-supervised loss and β regulates the batch mix of original and transformed images; the jigsaw setup uses a 3 × 3 grid and P = 30 permutations.Source: §3.4 · p. 4

4 EXPERIMENTS

• The evaluation first tests rotation and jigsaw pretexts for domain generalization, then uses them as auxiliary tasks with supervised learning and turns to domain adaptation and partial domain adaptation.Source: §4 · pp. 4–5

4.1.1 Data and Setup

• PACS, VLCS, and Office-Home provide DG testbeds with different domain and category coverage, evaluated by fine-tuning on three source domains and testing on the remaining PACS domain.The study also includes single-source digit experiments and fine-grained-car predictive domain adaptation, described in later sections.Source: §4.1.1 · p. 5

4.1.2 Self-Supervised Pretraining

• Rotation pretraining performs best, slightly ahead of patch-based Jigsaw, while whole-image Jigsaw does not appear to be a good pretext-model choice.The section compares patch-based and whole-image Jigsaw models with Rotation on DG classification accuracy.Source: §4.1.2 · p. 5, Table 1 · p. 5

4.1.3 Supervised Pretraining and Multi-task Learning

• Joint supervised and self-supervised learning outperforms single-task supervised training across domains, with a single-branch architecture better suited to the task and Rotation slightly stronger than Jigsaw.The comparison holds across architectures; whole-image Jigsaw simplifies combining the auxiliary task with supervised learning.Source: §4.1.3 · p. 5, Table 1 · p. 5

4.1.4 Multi-Source Domain Generalization

• The comparison spans low-rank, domain-specific aggregation, meta-learning, and adversarial DG methods; a data-augmentation method using extra collections is excluded.The excluded approach uses MSCOCO and WikiArt, which the other considered references do not exploit.Source: §4.1.4 · p. 5
• DeepAll baseline differences likely reflect undocumented inconsistencies or library implementations, so the authors report each method's reference to show relative improvements.They frame this reporting choice as a way to reveal possible inconsistencies.Source: §4.1.4 · p. 6
• On PACS, the multi-task approach matches or beats competitors on average except DDAIG on ResNet-18; analogous observations hold on VLCS.DDAIG uses source-domain annotations, which the multi-task method does not require.Source: §4.1.4 · p. 6, Table 2 · p. 6, Table 3 · p. 6
• On Office-Home, Rotation gains more than three percentage points over DeepAll, and Jigsaw+Rotation performs even better; DDAIG improves slightly more than one point over its DeepAll reference.Rotation is more suitable than Jigsaw as the auxiliary task on this dataset.Source: §4.1.4 · p. 6, Table 4 · p. 6

4.1.5 Single-Source Domain Generalization

• The multi-task approach is usually more stable and performs better than Adv.DA across hyper-parameters, although Adv.DA can reach high peaks.Exceptions occur on SVHN for Jigsaw at data bias β = 0.5 and Rotation at self-supervised task weight α = 0.9.Source: §4.1.5 · p. 6, Figure 4 · p. 7
• The experiments train on 10,000 MNIST digit samples and test on MNIST-M and SVHN, resizing all images to 32 × 32 RGB.The model uses the same backbone and experimental setting as the Adv.DA comparison.Source: §4.1.5 · p. 6
• On MNIST-M, Jigsaw and Rotation perform similarly to Adv.DA, while both significantly outperform it on SVHN.The comparison is shown for Jigsaw in the top row and Rotation in the bottom row of Figure 4.Source: §4.1.5 · pp. 6–7, Figure 4 · p. 7

Jigsaw ablation

• In AlexNet-PACS DG ablation with a 3×3 grid and P = 30, at β = 0.6, Jigsaw with α = 0.1 or 1 performs statistically equal to or better than DeepAll.At α ∼ 1 and β = 1, the puzzle task is trivial and may increase the risk of data memorization and overfitting.Source: §4.1.6 · p. 7
• At β = 0, fewer than 20% accuracy results because the classifier cannot converge, regardless of whether Jigsaw is active.Only the correctly ordered version of each image reaches the object classifier, drastically reducing its effective batch size.Source: §4.1.6 · p. 8

Jigsaw hyper-parameter tuning

• Across Jigsaw settings, accuracy varies by 1.5 percentage points as P ranges from 5 to 1000, and results remain almost always above DeepAll.Changing the grid from 2 × 2 to 4 × 4 also yields limited variation, with all results above DeepAll.Source: §4.1.6 · p. 8

Rotation ablation

• Rotation results remain above DeepAll and vary by around 1 percentage point on average as either parameter changes.The reported setting uses α = 0.4 and β = 0.4; Rotation has four classes, or three if the original orientation is excluded.Source: §4.1.6 · p. 8

Self-supervised performance

• Object, Rotation, and Jigsaw accuracy grow simultaneously, and Jigsaw remains effective at reordering shuffled patches as P increases.Jigsaw accuracy decreases as the number of permutation classes makes the task more difficult.Source: §4.1.6 · p. 8

Visual explanation and failure cases

• CAM maps show better object-class localization with Jigsaw or Rotation than with DeepAll; Rotation is less precise than Jigsaw on sketches, cartoons, and paintings, but works reasonably well on photos.For both tasks, recognition mistakes are associated with flaws in data interpretation, while localization remains meaningful.Source: §4.1.7 · p. 8

Setting

• Predictive DA has one labeled and several unlabeled source domains, with source and target metadata available to relate the target to known sources.The target model is formed without target images, although prior knowledge of source-target similarity simplifies the task.Source: §4.1.8 · p. 8

Motivation

• The method needs no source or target domain labels, reducing annotation costs and reliance on metadata that may be missing or noisy.The experiments also test a fine-grained recognition problem distinct from standard domain-generalization datasets.Source: §4.1.8 · pp. 8–9

Baseline and dataset

• The CompCars evaluation uses four car categories across 30 domains defined by five viewpoints and manufacturing years, averaging results over 870 source-target pairs.The four categories are MPV, SUV, sedan, and hatchback; viewpoint and year serve as metadata.Source: §4.1.8 · p. 9, Figure 9 · p. 9

Results

• The multi-task method significantly improves over source-only, LP, and MCC; AdaGraph scores higher, but the gap to Jigsaw is limited.The authors describe their approach as a reliable, inexpensive fallback when metadata is noisy or missing.Source: §4.1.8 · p. 9, Table 5 · p. 9

4.2.1 Single- and Multi-Source Domain Adaptation

• Single-source experiments use Office-Home and multi-source experiments use PACS; self-supervision can involve source and target data or target data alone.The minimal variation when source-side self-supervision is disabled indicates that most of the adaptive effect originates from the target task.Source: §4.2.1 · p. 9
• On Office-Home, the method improves over its baseline, DAN, JAN, and DANN, but trails HAFN, SAFN, and SAFN+ENT.The authors note that HAFN/SAFN’s extra fully connected layer appears particularly helpful in this cross-domain setting; disabling source-side self-supervision changes results minimally.Source: §4.2.1 · p. 9, Table 6 · p. 10
• With style-rich sources, the method beats Dial, DDiscovery, HAFN, and SAFN; Rotation is stronger and more stable than Jigsaw, while combining both yields a small average gain.The ablation attributes the major adaptive effect to self-supervised tasks on the target rather than to entropy loss.Source: §4.2.1 · pp. 9–10, Table 7 · p. 10

4.2.2 Partial Domain Adaptation

• Both Jigsaw and Rotation beat the first group of adaptive references; on VisDA2017 they beat PDA-specific methods, while ETN leads on Office-31.HAFN/SAFN results are comparable to or worse than the proposed method; ETN uses domain-pair-specific parameter selection, unlike the shared settings used here.Source: Table 8 · p. 10, §4.2.2 · p. 11
• On Office-31, adding target-class statistics to Jigsaw improves average accuracy by two percentage points over plain Jigsaw, reaching a result comparable to ETN within standard deviation.Jigsaw-γ more precisely identifies missing target classes in the A→W shift; adding a domain classifier gives a further small average improvement.Source: Table 8 · p. 10, Figure 10 · p. 10, §4.2.2 · p. 11

• Jigsaw-puzzle solving and image-orientation recognition integrate readily with supervised learning in a multi-task approach.Source: §5 · p. 11
• Table 9 covers partial domain adaptation classification accuracy on VisDA2017, with 12 source classes and 6 target classes.Source: Table 9 · p. 11
• The method improves cross-domain robustness and generalization performance, with results competitive with more elaborate domain adaptation and generalization methods.Source: §5 · p. 11
• The study points toward adaptive methods beyond object classification, including semantic segmentation, detection, and 3D visual learning under domain shift.Source: §5 · p. 11