Source-linked AI summary
Domain Generalization by Solving Jigsaw Puzzles
Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, Tatiana Tommasi
TL;DR
Domain-generalization methods struggle to transfer visual knowledge across domains, while unsupervised signals offer potentially style-independent regularities but have often required specialized training procedures. JiGen jointly learns supervised object classification and self-supervised jigsaw permutation recognition in one end-to-end network, and experiments report improved domain generalization and competitive unsupervised adaptation. The paper concludes that jointly learning intrinsic image invariances and semantic labels is effective, while noting that broader tasks and regularities remain to be tested.
Problem
Supervised methods struggle to generalize across domains, while existing unsupervised approaches often require tailored architectures and have not been broadly tested across large gaps such as paintings or sketches.
Method
JiGen jointly trains a shared convolutional network for supervised object classification and self-supervised jigsaw permutation recognition on images from multiple domains.
Results
JiGen improves domain generalization across PACS, VLCS, and Office-Home, supports single-source regularization and target-feature adaptation, and its jigsaw classifier remains effective as permutation difficulty increases.
Takeaways & Limitations
Jointly learning semantic labels and intrinsic spatial image invariances provides an effective framework for domain adaptation and generalization across pretrained convolutional architectures.
Takeaways & Limitations
The paper focuses on one type of invariance and calls for testing other regularities and applications beyond object categorization.
Abstract
from arXiv · showhide
Human adaptability relies crucially on the ability to learn and merge knowledge both from supervised and unsupervised learning: the parents point out few important concepts, but then the children fill in the gaps on their own. This is particularly effective, because supervised learning can never be exhaustive and thus learning autonomously allows to discover invariances and regularities that help to generalize. In this paper we propose to apply a similar approach to the task of object recognition across domains: our model learns the semantic labels in a supervised fashion, and broadens its understanding of the data by learning from self-supervised signals how to solve a jigsaw puzzle on the same images. This secondary task helps the network to learn the concepts of spatial correlation while acting as a regularizer for the classification task. Multiple experiments on the PACS, VLCS, Office-Home and digits datasets confirm our intuition and show that this simple method outperforms previous domain generalization and adaptation solutions. An ablation study further illustrates the inner workings of our approach.
1. Introduction
JiGen addresses the difficulty of recognizing objects across domains by jointly learning supervised object labels and unsupervised spatial regularities from jigsaw puzzles. The authors report that this shared learning supports generalization to new domains.
- Supervised vision methods often struggle to transfer knowledge and generalize across domains despite strong performance in well-defined settings.
- Unlabeled data may provide visual information less tied to domain style, but prior unsupervised approaches often require specialized architectures and finetuning.
- Biological visual learning motivates combining object categorization with learning regularities about object parts and their spatial relations.
- JiGen jointly optimizes object classification and jigsaw solving in an end-to-end architecture, sharing a convolutional backbone across both tasks.
- The method uses jigsaw solving as a side objective to learn spatial co-location while classifying objects across multiple source domains.
- JiGen captures shared knowledge across multiple sources, regularizes single-source learning, and can use unlabeled target samples for feature adaptation.
2. Related Work
Related work spans jigsaw-puzzle reconstruction, domain generalization, and domain adaptation. JiGen takes an orthogonal approach by jointly exploiting supervised labels and unsupervised signals inherent in images.
- Solving Jigsaw Puzzles: Jigsaw-puzzle solvers recover an original image from shuffled patches, with methods addressing patch permutations, patch dimensions, completeness, and homogeneity.
- Solving Jigsaw Puzzles: Recent deep-learning jigsaw methods process separate tiles and then recombine their representations, requiring dedicated architectures and transfer procedures.
- Domain Generalization and Adaptation: Domain generalization seeks models that perform well across multiple distributions using transferable knowledge from limited population sources.
- JiGen: The proposed framework combines ordered-image object classification with shuffled-image permutation recognition.
- Domain Generalization and Adaptation: Domain adaptation uses labeled source data and unlabeled target examples, but training must be rerun when the application target changes.
- Domain Generalization and Adaptation: JiGen investigates jointly exploiting supervised and unsupervised inherent image signals as an orthogonal solution within the domain-generalization framework.
3. The JiGen Approach
JiGen trains a shared convolutional network on object classification and jigsaw permutation recognition using images from multiple source domains. At test time, only the object classifier predicts labels for new target images, while the jigsaw task supplies the auxiliary learning signal.
- Problem setup: The domain-generalization objective is to learn from multiple labeled source domains a model that performs well on new target populations sharing the same categories.
- Jigsaw task: JiGen decomposes source images into n × n patches, shuffles them, and assigns permutation indices to create a second classification task.
- Shared network: The object and jigsaw classifiers share feature parameters, while a separate fully connected layer recognizes permutation indices.
- Joint optimization: The network minimizes object-classification and jigsaw cross-entropy losses, with the jigsaw loss also applied to ordered images.
- Inference: At test time, JiGen uses only the object classifier to predict labels on new target images.
- Domain adaptation: For unsupervised domain adaptation, unlabeled target images participate in the jigsaw task, while ordered target images also receive an entropy-based uncertainty objective.
- Implementation: Experiments fix the jigsaw task to 3 × 3 patch grids and P = 30 permutations, with shuffled and ordered images mixed within each batch.
4. Experiments
JiGen is evaluated across domain generalization and adaptation settings, using PACS, VLCS, Office-Home, and digit datasets. Experiments examine backbone choices, ablations, jigsaw-task behavior, and single- versus multi-source training.
- Experimental settings: JiGen combines image classification with a jigsaw-puzzle task and is evaluated on PACS, VLCS, Office-Home, and digit datasets.The experiments cover multi-source domain generalization, unsupervised domain adaptation, and single-source generalization.
- Backbone comparison: Combining the jigsaw puzzle with classification improves average performance over the corresponding C-CFN-Deep All model, although C-CFN-Deep All remains below standard Alexnet.This comparison uses a context-free siamese-ennead backbone on PACS.
- Multi-source domain generalization: On PACS, VLCS, and Office-Home, JiGen generally matches or exceeds competing domain-generalization methods across the reported backbone and target-domain settings.With Alexnet, JiGen produces the best average PACS result; with Resnet-18 it is slightly below D-SAM on average but wins three of four PACS target cases.
- Jigsaw-task behavior: The jigsaw and object classifiers improve simultaneously, and the jigsaw model remains effective as permutation difficulty increases.Jigsaw recognition accuracy decreases as the number of permutation classes grows, but the classifier still effectively reorders shuffled patches.
- Single-source domain generalization: For single-source digit generalization, JiGen is more stable than Adv.DA, nearly matches it on MNIST-M, and significantly outperforms it on SVHN.JiGen is better than Adv.DA’s lower accuracy value in all but one reported SVHN setting.
- Unsupervised domain adaptation: In unsupervised domain adaptation, adding the jigsaw task to unlabeled target samples yields competitive results and retains its advantage across target jigsaw-loss weights.The reported PACS results outperform competing methods on average and particularly on the difficult sketches target domain.
5. Conclusions
JiGen combines object classification with learning intrinsic image invariances, supporting domain generalization and adaptation. The authors identify broader applications and other regularities as directions for future work.
- The framework focuses on spatial co-location of image parts and accommodates a wide range of pretrained convolutional architectures.
- Future work could examine other regularities and applications such as semantic segmentation and person re-identification.
A. Appendix
The appendix analyzes auxiliary self-supervision for domain generalization, emphasizing how jigsaw-based spatial regularities guide attention toward informative object regions. It also evaluates image-rotation prediction as an alternative auxiliary task.
- The appendix provides further experiments on jigsaw puzzles and other self-supervised tasks as auxiliary objectives for cross-domain generalization.
- Relative patch positions capture domain-shared and class-discriminative regularities, encouraging localization and recombination of relevant object subparts.
- The visual-analysis experiments use Class Activation Maps to examine which image regions the learned model emphasizes.
- Rotation recognition with four angle classes was tested as an alternative self-supervised task on PACS using the Alexnet-based protocol.
- Table 5 reports rotation-recognition results alongside cartoons and sketches containing objects with unusual orientations.