PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Domain Adaptation for Computer Vision Tasks

Jiaolong Xu, Liang Xiao, Antonio M. Lopez

arXiv:1907.10915v3cs.CVcs.LG

TL;DR

The paper asks whether self-supervised visual representation learning can support domain adaptation without target-domain main-task labels. It jointly trains pretext and main tasks using automatically labeled target images, and reports state-of-the-art or comparable adaptation performance across object recognition and urban-scene semantic segmentation, while identifying limits in pretext-task design.

  • Problem

    Whether self-supervised visual representation learning can support domain adaptation had not been explored, despite the practical cost of obtaining task ground truth.

  • Method

    The method jointly trains a source-supervised main task and a target-image pretext task through a shared encoder to learn domain-invariant representations.

  • Results

    The method achieves state-of-the-art domain adaptation performance in object recognition and urban-scene semantic segmentation, with rotation prediction comparable to state-of-the-art domain adaptation methods.

  • Takeaways & Limitations

    Self-supervision provides an alternative route to domain adaptation, and simple rotation prediction can be complementary to adversarial and batch-normalization-based methods.

  • Takeaways & Limitations

    The study investigates only several simple self-supervised strategies, and designing suitable pretext tasks for domain adaptation remains open.

Abstract

from arXiv · show

Recent progress of self-supervised visual representation learning has achieved remarkable success on many challenging computer vision benchmarks. However, whether these techniques can be used for domain adaptation has not been explored. In this work, we propose a generic method for self-supervised domain adaptation, using object recognition and semantic segmentation of urban scenes as use cases. Focusing on simple pretext/auxiliary tasks (e.g. image rotation prediction), we assess different learning strategies to improve domain adaptation effectiveness by self-supervision. Additionally, we propose two complementary strategies to further boost the domain adaptation accuracy on semantic segmentation within our method, consisting of prediction layer alignment and batch normalization calibration. The experimental results show adaptation levels comparable to most studied domain adaptation methods, thus, bringing self-supervision as a new alternative for reaching domain adaptation. The code is available at https://github.com/Jiaolong/self-supervised-da.

1 INTRODUCTION

The paper addresses whether self-supervised learning can support domain adaptation by jointly learning pretext and main tasks, using target images without main-task labels. It proposes a generic framework and complementary strategies for object recognition and urban-scene semantic segmentation.

  • Motivation: Human labeling for recognition, detection, and segmentation is costly and error-prone, motivating methods that reduce required supervision.The paper situates domain adaptation and self-supervision within broader efforts to obtain more from limited labeling budgets.
  • Research gap: Unlike prior self-supervised learning used mainly for transfer, this work investigates whether it can learn domain-invariant representations for domain adaptation.The pretext task provides automatically generated labels on target images while the main task uses source-domain labels.
  • Contributions: The proposed generic method jointly learns a pretext task and a main task through multi-task learning.The framework targets object recognition and semantic segmentation of urban scenes.
  • Contributions: Image rotation prediction is used as a central pretext task, with several variations assessed for domain adaptation performance.The study focuses on how pretext-task choices affect adaptation rather than proposing a new self-supervised learning family.
  • Contributions: Prediction layer alignment and batch normalization calibration are introduced as complementary strategies to further improve self-supervised domain adaptation.These strategies are presented as additional contributions alongside the generic framework and pretext-task study.

2 RELATED WORK

Related work spans self-supervised representation learning and unsupervised domain adaptation, including pretext tasks, input or feature alignment, and output-space adaptation. The paper positions its approach as a multi-task alternative that connects source and target domains through auxiliary self-supervision.

  • Self-supervised visual representation learning: Self-supervised representation learning creates labels from transformed or otherwise automatically constructed image samples for training generic visual features.Examples include patch-location prediction, image colorization, inpainting, clustering-derived classes, and rotation-angle prediction.
  • Self-supervised visual representation learning: Rotation prediction has shown promising results for high-level representation learning and is adopted in this work alongside patch-location prediction.The paper also notes an improvement that decouples rotation-related and unrelated features.
  • Unsupervised domain adaptation: Unsupervised domain adaptation methods address shifts through input-level, feature-level, or output-space adaptation.The reviewed input-level approaches include GAN-based methods and image stylization, while other methods align features or outputs.
  • Relation to prior adaptation methods: Prior auxiliary-task approaches for semantic segmentation are similar in spirit to the paper’s multi-task design, but the proposed auxiliary tasks and complementary strategies are not restricted to segmentation.The paper contrasts its scope with spatial-prior self-labeling and super-pixel distribution matching.
  • Relation to prior adaptation methods: The paper reports that its method outperforms a jigsaw-puzzle approach on object recognition and semantic segmentation, with target segmentation accuracy superior to most compared methods.The reported segmentation comparison places the method behind only one cited method in the discussed set.

3 METHOD

The method jointly trains a main task on labeled source images and a pretext task on automatically transformed target images through a shared encoder. It complements self-supervision with prediction-layer alignment and batch-normalization calibration strategies.

  • Framework: The framework shares encoder E between a source-supervised main-task network and a target-domain pretext network.For semantic segmentation, E is paired with decoder S for the main task and network P for the pretext task.
  • Framework: Joint training back-propagates main-task and pretext losses to the shared encoder using source and target samples, respectively.The encoder accumulates gradients from Lseg and Lp before its weights are updated.
  • Pretext tasks: Image rotation prediction uses four transformations—0, 90, 180, and 270 degrees—and trains P to output a distribution over them from encoder features.The pretext loss is optimized with respect to encoder and pretext-network parameters.
  • Pretext tasks: Spatial-aware rotation prediction splits each image into four regions, crops and rotates within regions, and expands labels from 4 to 16 combinations.The task encodes both geometric transformation and spatial layout information.
  • Objective and calibration: The combined objective balances segmentation and pretext losses with λp = 1 and adds adversarial and discriminator losses weighted by λadv = 0.01 and λd = 1.0.These weights are fixed for the reported experiments after validation tuning for the adversarial strategy.
  • Complementary strategies: Prediction-layer alignment places a domain discriminator after the encoder's prediction layer to distinguish source from target feature maps through adversarial training.The discriminator predicts domain labels, while the encoder is optimized to fool it.

4 EXPERIMENTS AND RESULTS

The experiments validate the proposed domain adaptation method on both object recognition and semantic segmentation.

  • Experimental scope: The experiments evaluate the proposed domain adaptation method on object recognition and semantic segmentation.Both tasks are used to validate the method.

4.1 Implementation Details

The implementation uses PyTorch on a single GTX 1080 Ti GPU, with specified architectures for object recognition and semantic segmentation.

  • Implementation: The method is implemented in PyTorch on a single GTX 1080 Ti GPU with 11 GB memory.Object recognition uses JiGen code with default hyperparameters and ResNet-18 and ResNet-50; segmentation uses DeepLab-v2 and DRN-26 architectures.

4.2 Domain adaptation for object recognition

On Office and PACS object-recognition adaptation, the rotation-based method performs competitively across domains and yields more discriminative, better-aligned features than non-adapted models.

  • Office dataset: Rot achieves the best average Office accuracy among the evaluated strategies, reaching accuracies comparable to state-of-the-art methods.MixRot and SPRot are close to Rot, while Rot+Adv+BN improves over Rot+Adv but remains below Rot.
  • Office dataset: Rot produces better category discrimination and stronger source-target domain alignment than non-adapted ResNet-50 features on A −→W.The comparison is based on t-SNE visualizations of category and domain information.
  • PACS dataset: Rot outperforms all baselines on multi-source PACS adaptation, although MixRot and SPRot exceed Rot on selected target domains.Rot nevertheless has the highest overall average accuracy, indicating the most robust performance among the compared methods.
  • PACS dataset: Rot shows better category discriminativity and domain alignment than the non-adapted SRC model on PACS.The t-SNE visualization is used to assess both class separation and cross-domain alignment.

4.3 Domain adaptation for semantic segmentation

For synthetic-to-real semantic segmentation, the study evaluates pretext-task settings and complementary alignment strategies, finding that task design and calibration materially affect adaptation performance.

  • Pretext task learning strategies: MixRot performs worse than Rot, likely because the mixed samples are dominated by source images and reduce domain-invariant representation power.The cited source-to-target sample counts are 24996 versus 2975.
  • Pretext task learning strategies: A more difficult spatial-aware rotation task, SPRot, produces worse adaptation than simple rotation prediction and is harder to converge.The paper identifies proper pretext-task design as an open area for further exploration.
  • Pretext task comparisons: Rot outperforms jigsaw-puzzle self-supervision on both SYN2CS and GTA2CS semantic-segmentation adaptation.For GTA2CS, jigsaw puzzle gains 1.2 percentage points, whereas Rot achieves a 6.2 percentage-point gain.
  • Pretext task learning strategies: Cropping size affects adaptation: 128 × 128 performs worst, while full-image inputs can yield a trivial 100% pretext accuracy and failed adaptation.The authors therefore emphasize selecting a crop that controls pretext-task difficulty.
  • Feature extraction layer: Middle-layer features perform slightly better than final-layer features, suggesting that pretext learning is not highly sensitive to the extraction layer.In this segmentation setup, the final layer corresponds to the prediction layer because the decoder is only an up-sampling layer.
  • Complementary strategies: Prediction-layer alignment improves Rot by 1.1 percentage points, while combining it with BN calibration improves Rot by 2.1 percentage points.BN calibration alone does not improve Rot in the reported complementary-strategy comparison, and the combined method can saturate on SYN2CS.
  • Complementary strategies: BN calibration alone reaches a best domain-adaptation gain of 6.8 percentage points across the evaluated datasets and networks.When combined with Rot or Rot+Adv, however, it adds only 1 or 2 percentage points.
  • Qualitative analysis: Rot improves qualitative segmentation over SRC, while Rot+Adv+BN yields the best class discrimination among the visualized feature representations.The paper reports that SRC produces noisy segmentation without domain adaptation.

4.4 Comparison to the state-of-the-art

The method is compared with recent domain adaptation approaches using similar architectures and spanning multiple alignment and training mechanisms. Adapted models achieve comparable accuracy to the state of the art, while ResNet-101 provides greater adaptation gain than DRN-26.

  • 4.4 Comparison to the state-of-the-art: Adapted models achieve comparable accuracies to recently published state-of-the-art domain adaptation methods.The comparison includes input-, feature-, and output-level alignment, curriculum, and self-labeling methods.
  • 4.4 Comparison to the state-of-the-art: Some compared methods perform worse than source-only training, giving them higher relative gains despite lower absolute results.
  • 4.4 Comparison to the state-of-the-art: ResNet-101 achieves better domain adaptation gain than the shallower DRN-26 architecture.

4.5 Discussion

Self-supervised representation learning reduces domain shift, with simple rotation prediction matching state-of-the-art performance and complementing existing adaptation methods. The discussion attributes this to joint use of source and target samples and multi-task learning, while experiments identify architecture depth and pretext-task design as important factors.

  • 4.5 Discussion: Image rotation prediction reduces domain shift and can achieve performance comparable to state-of-the-art domain adaptation methods.Rotation prediction is reported as more robust than MixRot, SPRot, and Jigsaw.
  • 4.5 Discussion: Rotation prediction is complementary to adversarial-based and batch-normalization-based domain adaptation methods.
  • 4.5 Discussion: Using source and target samples in a common supervised process helps learn cross-domain feature representations.The paper connects this explanation to feature visualizations of domain alignment.
  • 4.5 Discussion: Deeper architectures and stronger representation learning improve self-supervised domain adaptation, but more complex pretext tasks do not necessarily perform better.The experiments also report that adversarial and adversarial-plus-batch-normalization methods can further improve rotation prediction when adversarial adaptation is not worse.

5 CONCLUSION

The paper explores self-supervised learning for domain adaptation and shows that image-rotation prediction can achieve state-of-the-art domain adaptation performance. It evaluates pretext tasks and complementary strategies through object recognition and urban-scene semantic segmentation experiments, with further work proposed on additional tasks and vision applications.

  • 5 CONCLUSION: Image-rotation prediction self-supervision can achieve state-of-the-art domain adaptation performance.
  • 5 CONCLUSION: The study evaluates several pretext tasks and complementary domain adaptation strategies using object recognition and urban-scene semantic segmentation.
  • 5 CONCLUSION: The authors propose investigating more pretext tasks and applying the method to other relevant vision tasks.
Loading 1907.10915v3…