PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Scene Adaptation with Memory Regularization in vivo

Zhedong Zheng, Yi Yang

arXiv:1912.11164v3cs.CV

TL;DR

Unsupervised scene adaptation must transfer from labeled source data when target-scene annotations are difficult to obtain, and existing work largely focuses on inter-domain alignment. This paper uses the segmentation model’s own classifiers as memory to regularize their prediction discrepancy. It reports competitive results across synthetic-to-real and cross-city benchmarks, including a +11.3% improvement in mIoU over the baseline on SYNTHIA → Cityscapes.

  • Problem

    Scene adaptation must transfer from labeled source data because target-scene annotations are difficult to obtain, while inter-domain alignment leaves intra-domain knowledge and prediction uncertainty under-explored.

    Source: §1 · p. 1

  • Method

    The segmentation model itself serves as memory, and regularization reduces the prediction discrepancy between its primary and auxiliary classifiers.

    Source: §1 · p. 2

  • Results

    The method achieves competitive results on two synthetic-to-real and one cross-city benchmark; on SYNTHIA → Cityscapes, it improves mIoU by +11.3% over the baseline.

    Source: §4.3 · p. 6, §5 · p. 7

  • Takeaways & Limitations

    The method is complementary to previous work and is designed without extra parameters or external modules.

    Source: §1 · p. 2, §5 · p. 7

Abstract

from arXiv · show

We consider the unsupervised scene adaptation problem of learning from both labeled source data and unlabeled target data. Existing methods focus on minoring the inter-domain gap between the source and target domains. However, the intra-domain knowledge and inherent uncertainty learned by the network are under-explored. In this paper, we propose an orthogonal method, called memory regularization in vivo to exploit the intra-domain knowledge and regularize the model training. Specifically, we refer to the segmentation model itself as the memory module, and minor the discrepancy of the two classifiers, i.e., the primary classifier and the auxiliary classifier, to reduce the prediction inconsistency. Without extra parameters, the proposed method is complementary to the most existing domain adaptation methods and could generally improve the performance of existing methods. Albeit simple, we verify the effectiveness of memory regularization on two synthetic-to-real benchmarks: GTA5 -> Cityscapes and SYNTHIA -> Cityscapes, yielding +11.1% and +11.3% mIoU improvement over the baseline model, respectively. Besides, a similar +12.0% mIoU improvement is observed on the cross-city benchmark: Cityscapes -> Oxford RobotCar.

1 Introduction

Unsupervised scene adaptation addresses the lack of target-scene annotations by transferring knowledge from labeled source data, but domain alignment can leave target-specific learning underused. The paper proposes memory regularization in vivo to use the model’s own classifiers to reduce prediction inconsistency and exploit intra-domain knowledge.

1 Introduction

  • Brute-force source–target alignment promotes shared domain-agnostic features but can neglect target-specific features and compromise adaptation performance.The paper frames intra-domain learning as an orthogonal direction to inter-domain alignment.Source: §1 · p. 1
  • Target predictions can disagree between the primary and auxiliary classifiers, especially on unlabeled target data, while source predictions are relatively consistent.The authors use the auxiliary classifier as a probe of this intra-domain uncertainty.Source: §1 · p. 1, Figure 1 · p. 1
  • Memory regularization in vivo treats the running segmentation model as memory and aligns its primary and auxiliary predictions for the same input.The input acts as a key and the classifiers’ predictions as values; unlike external-memory approaches, it adds no parameters or modules.Source: §1 · p. 2
  • The method targets intra-domain knowledge and prediction uncertainty, complementing adaptation methods focused on inter-domain alignment.The authors describe the regularizer as compatible with most scene-segmentation networks.Source: §1 · p. 2

2 Related Works

Prior segmentation-adaptation work emphasizes aligning source and target domains, while memory-based learning commonly uses external memory or models. The paper positions its model-internal memory as a distinct approach to target-domain knowledge.

2.1 Domain Adaptation for Segmentation

  • Domain-adaptation methods use image translation, adversarial alignment, or confident pseudo-label self-training, and the proposed method is described as orthogonal to these approaches.The paper says memory regularization can be integrated with alignment and self-training methods.Source: §2.1 · p. 2

2.2 Memory-based Learning

  • Prior memory-based methods use external memory modules or models, whereas this method uses the running model itself without extra parameters or modules.The framework also applies memory regularization during both source-focused Stage I and target-focused, pseudo-label Stage II.Source: §2.2 · p. 3, Figure 3 · p. 3

3 Method

The method combines inter-domain alignment with an internal consistency objective between two classifiers, then fine-tunes on target-domain pseudo-labels. Memory regularization is applied to target predictions without adding model parameters.

3.1 Algorithm Overview

  • Stage-I combines source supervision and source–target adversarial alignment with memory regularization; Stage-II fine-tunes on target-domain pseudo-labels to learn target-specific knowledge.In Stage-II, memory regularization also helps prevent overfitting to noisy pseudo-labels.Source: §3.1 · p. 3, §3.2 · p. 4, §3.3 · p. 4

3.2 Domain-agnostic learning

  • Adversarial losses on both classifiers’ predictions encourage semantic-level alignment between source and target domains.Separate discriminators assess whether target predictions are close to source predictions in semantic space.Source: §3.2 · p. 4

3.3 Domain-specific learning

  • Memory regularization treats the model as a memory module and encourages its primary and auxiliary classifiers to agree on unlabeled target images.The same target image acts as the key, and each classifier's prediction is a value; the consistency objective uses KL divergence.Source: §3.3 · p. 4

3.4 Optimization

  • The memory-regularization weight is 0.1; segmentation and adversarial losses use separate classifier-specific weights.The auxiliary segmentation loss weight is 0.5; the primary and auxiliary adversarial loss weights are 0.001 and 0.0002, respectively.Source: §3.4 · pp. 4–5
  • Table 1 reports that memory regularization helps both classifiers, especially the auxiliary classifier, while combining their outputs improves the full model further.The caption describes a Stage-I ablation of memory regularization on the primary and auxiliary classifiers.Source: Table 1 · p. 5

3.5 Implementation Details

  • The implementation uses Deeplab-v2 with a ResNet-101 backbone, 0.1 dropout, and 1024 × 512 training crops from resized 1280 × 640 images.Training begins without memory regularization for 10k iterations, followed by two 25k-iteration stages.Source: §3.5 · p. 5

4 Experiment

The experiments evaluate memory regularization on synthetic-to-real and cross-city adaptation, with ablations examining its role in both training stages. The reported results show gains on the synthetic-to-real benchmarks and competitive cross-city performance.

4.2 Ablation Studies

  • In Stage I, adding adversarial and memory regularization losses improves the full model from 37.23% to 45.46% mIoU.In Stage II, the full model with memory regularization reaches 48.31% mIoU, higher than the model trained only on pseudo labels; the authors state that memory regularization helps prevent overfitting to noisy pseudo labels.Source: Table 2 · p. 5, Table 3 · p. 5, §4.2 · pp. 5–6

4.1 Dataset and Evaluation Metric

  • The evaluation uses synthetic GTA5 and SYNTHIA source data with unlabeled Cityscapes target data, plus a Cityscapes → Oxford RobotCar cross-city setting.The reported metric is mean Intersection over Union (mIoU), averaged over all classes.Source: §4.1 · p. 5

4.3 Comparisons with state-of-the-art methods

  • +11.1% mIoU improvement over the source-only model on GTA5 → Cityscapes and +11.3% mIoU accuracy improvement over the baseline on SYNTHIA → Cityscapes.The method reports 48.3% mIoU on GTA5 → Cityscapes and 53.8% mIoU* and 46.5% mIoU on SYNTHIA → Cityscapes.Source: Table 4 · p. 6, Table 5 · p. 6, §4.3 · p. 6
  • The method achieves 73.9% mIoU on Cityscapes → Oxford RobotCar.Cityscapes is described as sunny-weather data and Oxford RobotCar as rainy-weather data.Source: §4.3 · p. 6, Table 6 · p. 7

5 Conclusion

The conclusion presents memory regularization as a parameter-free way to use the model itself as memory for unsupervised scene adaptation. It reports competitive results across two synthetic-to-real benchmarks and one cross-city benchmark.

5 Conclusion

  • The model itself serves as the memory module, regularizing training without extra parameters or external modules.The method leverages intra-domain knowledge and reduces uncertainty in model learning.Source: §5 · p. 7
  • The method is complementary to previous work and achieves competitive results on two synthetic-to-real benchmarks and one cross-city benchmark.The benchmarks are GTA5 → Cityscapes, SYNTHIA → Cityscapes, and Cityscapes → Oxford RobotCar.Source: §5 · p. 7

Related papers

Loading 1912.11164v3…