PaperlayerView exact revision on arXiv

Source-linked AI summary

ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks

Onur Tasar, S L Happy, Yuliya Tarabalka, Pierre Alliez

arXiv:1907.12859v2cs.CV

TL;DR

Large spectral shifts between satellite training and test images can produce unsatisfactory segmentation maps, and translating images without preserving their semantics risks mismatched annotations. ColorMapGAN adapts training-image colors while retaining their semantics, and reported comparisons show higher IoU than CycleGAN in specified class-and-city cases.

  • Problem

    Large spectral differences between training and test images can yield poor segmentation, while image translation may alter semantic structures and mismatch original annotations.

    Source: §I · p. 2, B. Contributions · p. 3

  • Method

    ColorMapGAN generates target-like fake training images while preserving the semantic content of the originals, whose annotations are then used for fine-tuning.

    Source: B. Contributions · p. 3, 1) ColorMapGAN · p. 3

  • Results

    On Villach road IoU, CycleGAN is around 9% lower than ColorMapGAN, and on Bad Ischl building IoU, ColorMapGAN outperforms CycleGAN by 6%.

    Source: D. Results · p. 8

  • Takeaways & Limitations

    Preserving the original training-image semantics lets the original annotations remain usable for fine-tuning on generated images.

    Source: B. Contributions · p. 3

    ColorMapGAN slightly smooths tree patterns, which may explain CycleGAN's better tree result on pair 1.

    Source: D. Results · p. 11

Abstract

from arXiv · show

Due to the various reasons such as atmospheric effects and differences in acquisition, it is often the case that there exists a large difference between spectral bands of satellite images collected from different geographic locations. The large shift between spectral distributions of training and test data causes the current state of the art supervised learning approaches to output unsatisfactory maps. We present a novel semantic segmentation framework that is robust to such shift. The key component of the proposed framework is Color Mapping Generative Adversarial Networks (ColorMapGAN), which can generate fake training images that are semantically exactly the same as training images, but whose spectral distribution is similar to the distribution of the test images. We then use the fake images and the ground-truth for the training images to fine-tune the already trained classifier. Contrary to the existing Generative Adversarial Networks (GANs), the generator in ColorMapGAN does not have any convolutional or pooling layers. It learns to transform the colors of the training data to the colors of the test data by performing only one element-wise matrix multiplication and one matrix addition operations. Thanks to the architecturally simple but powerful design of ColorMapGAN, the proposed framework outperforms the existing approaches with a large margin in terms of both accuracy and computational complexity.

I. INTRODUCTION

Satellite semantic segmentation is important for map generation, but spectral differences between training and test images can sharply reduce CNN performance. The related work covers domain adaptation methods in computer vision and remote sensing, including image translation approaches that may disrupt the source-image semantics.

I. INTRODUCTION

  • CNN segmentation performs poorly when training and test images have large spectral differences, even when the images come from the same satellite.Illumination and atmospheric effects can produce substantial radiometric variation across images.Source: §I · p. 1
  • Unsupervised domain adaptation targets high-quality segmentation under domain shift without test annotations, while common augmentation may be insufficient for huge spectral differences.The cited examples include gamma correction and random contrast changes.Source: §I · p. 2

A. Related Work

  • Computer-vision domain adaptation includes adversarial distribution alignment, feature alignment, self-learning, and network regularization or normalization methods.The section also describes classifier-discrepancy and spatially aware approaches.Source: A. Related Work · p. 2
  • Image-to-image translation can create target-style training images, but artificial objects and artifacts can mismatch original annotations and lead to incorrect learning.CyCADA adds a segmentation-consistency step, while neural style transfer also cannot guarantee preservation of semantic structures.Source: A. Related Work · p. 2
  • Remote-sensing adaptation includes invariant-feature selection, classifier adaptation, active learning, and distribution alignment using methods such as histogram matching and optimal transport.The section notes that CycleGAN-generated remote-sensing images may have lower-than-desired quality because of heterogeneous, complex structures.Source: A. Related Work · pp. 2–3

B. Contributions

The paper addresses unpaired city-to-city adaptation where test annotations are unavailable and semantic structures must survive translation. ColorMapGAN changes training-image colors while preserving their object locations and shapes, allowing original annotations to remain usable for fine-tuning.

B. Contributions

  • The task uses geographically separate, unpaired training and test images with significant spectral differences and no test annotations.Paired images from the same geographic extent at different times would make the problem easier.Source: B. Contributions · p. 3
  • Preserving training-image semantics is essential because otherwise generated images and original annotations mismatch, leading the model to learn incorrect information.The preserved annotations can be used to fine-tune the classifier on generated images.Source: B. Contributions · p. 3

1) ColorMapGAN:

  • ColorMapGAN maps training-image colors to test-image colors while keeping object locations and shapes unchanged.The generated images are intended to have spectral distributions similar to test images while remaining semantically identical to the originals.Source: 1) ColorMapGAN · p. 3

2) Higher accuracy:

The experiments report that ColorMapGAN yields higher segmentation accuracy than the compared approaches, while its generator avoids convolutional and pooling operations. The paper also reviews the adversarial roles of the generator and discriminator.

2) Higher accuracy:

  • The framework with ColorMapGAN generates predicted maps with much higher accuracy than the compared image-generation, histogram-matching, gray-world, and direct-adaptation approaches.The experiments fine-tune models trained on original images using fake training images from the evaluated approaches.Source: 2) Higher accuracy · p. 3

3) Lower complexity:

  • ColorMapGAN's generator uses one element-wise matrix multiplication and one matrix addition, and its training time is significantly lower than other learning-based approaches.Unlike existing GAN generators, it performs no convolution or pooling operations.Source: 3) Lower complexity · pp. 3–4

II. GENERATIVE ADVERSARIAL NEURAL NETWORKS

  • In a GAN, the generator produces fake samples and the discriminator distinguishes real from fake data; training drives the generator toward samples indistinguishable from real data.The discriminator maximizes its objective while the generator minimizes its own objective.Source: §II · p. 4

III. METHODOLOGY

The framework trains an initial U-net, uses ColorMapGAN to create semantically preserved training images with test-like spectral distributions, fine-tunes the classifier, and segments test images. Its generator performs color mapping without convolutional or pooling layers.

A. The Overall Framework

  • The four-step framework trains an initial U-net, generates test-like fake training images, fine-tunes with those images and original ground truth, then classifies test images.The classifier is a modified U-net using Leaky-ReLU and omitting batch normalization.Source: A. The Overall Framework · p. 4

1) Generator:

  • ColorMapGAN seeks fake training patches with spectral distributions like test patches while preserving the training patches’ exact semantics.The generator omits convolutional and pooling layers to preserve semantics.Source: 1) Generator · p. 4
  • The generator transforms colors with learnable scale W and shift K matrices, applied by element-wise multiplication and addition.W starts at ones and K at zeros, so input and output are initially identical.Source: 1) Generator · p. 5
  • Because full color matrices exceed 50 million elements, the generator updates only W and K entries for colors present in each training patch.An index vector I identifies the colors found in a patch.Source: 1) Generator · p. 5
  • After transformation, values are clipped to [−1, 1], denormalized to [0, 255], and reshaped to the input patch’s dimensions.Only the indexed color entries are updated during each training iteration.Source: 1) Generator · p. 5

2) Discriminator:

The discriminator classifies local regions of an input patch as real or fake and averages its matrix of local outputs into a final value. Its architecture follows CycleGAN’s discriminator, and its objective uses LSGAN functions.

2) Discriminator:

  • The discriminator produces a matrix of local real/fake judgments and averages its elements to yield a final value.Its architecture is the same as CycleGAN’s discriminator.Source: 2) Discriminator · p. 5
  • In the architecture diagram, n denotes patch size and the number below each layer denotes its depth.Source: Figure 4 · p. 6
  • The discriminator objective uses the functions presented in LSGAN.The discriminator and generator are trained simultaneously by minimizing their objective functions.Source: 2) Discriminator · p. 6

A. Methods Used for Comparison

The comparison methods include an unadapted U-net, image-translation approaches, color-normalization methods, and direct classifier adaptation. The translation approaches vary in their mapping constraints and in how they represent image content and style.

A. Methods Used for Comparison

  • U-net is the no-adaptation baseline, trained on the training images and applied directly to test images.Source: A. Methods Used for Comparison · p. 6
  • CycleGAN uses two generators for opposite domain mappings and an L1 cycle-consistency constraint to reproduce the original images.Source: A. Methods Used for Comparison · p. 7
  • UNIT maps both domains through shared latent space, while MUNIT combines content from one domain with style from the other using AdaIN.Source: A. Methods Used for Comparison · p. 7
  • DRIT resembles MUNIT but combines content and style codes by concatenation rather than AdaIN.Source: A. Methods Used for Comparison · p. 7
  • Histogram matching aligns training and test histograms per color channel, whereas gray world standardizes images using an assumed natural-gray average color.Source: A. Methods Used for Comparison · p. 7
  • AdaptSegNet aligns predicted maps across domains; the comparison includes single- and multiple-classification-layer strategies, with ASPP removed from the single-layer network.The authors state that ASPP reduces satellite-image segmentation performance, especially for objects covering small areas.Source: A. Methods Used for Comparison · p. 7
  • For a fair comparison, the alternative methods replace ColorMapGAN’s image-generation step; gray world also changes the test image being segmented.That variant segments the test image after removing the illuminant effect.Source: A. Methods Used for Comparison · p. 7

B. Experimental Setup

Experiments use paired European cities with large spectral differences but similar class structures, assuming labels are available only for the training city. The training pipeline first trains U-net, then generates fake images and fine-tunes the classifier.

B. Experimental Setup

  • The two city pairs were chosen for large radiometric differences and similar object structures, with annotations available only for the training city.The pairs are Bad Ischl–Villach and Béziers–Roanne.Source: B. Experimental Setup · p. 7
  • Images were split into overlapping 256 × 256 patches, and Intersection over Union (IoU) was used for quantitative evaluation.Neighboring patches overlap by 32 pixels.Source: B. Experimental Setup · p. 8

C. Training Details

  • The pipeline trains U-net first, generates fake images with ColorMapGAN, then fine-tunes the trained network on those images.U-net training runs for 2,500 iterations; ColorMapGAN runs for 8,000 iterations, followed by 750 fine-tuning iterations.Source: C. Training Details · p. 8

D. Results

The results compare ColorMapGAN with image-translation and spectral-standardization approaches across two city pairs. ColorMapGAN preserves semantic structure and yields more favorable comparisons in several reported cases, while other methods retain advantages or limitations for particular classes and settings.

D. Results

  • MUNIT and DRIT can distort semantic content, while UNIT outputs are inconsistent across pairs; patchwise generation also causes tiling artifacts for these methods.The authors report that the framework using these methods performs poorly on test data; fake-data methods were generally averaged over 20 runs, versus three for AdaptSegNet.Source: D. Results · p. 8
  • Gray world reduces spectral differences but leaves residual shift, and its performance is mostly better than UNIT, MUNIT, and DRIT but remains limited.The fake images still show spectral differences within each city pair.Source: D. Results · p. 8
  • AdaptSegNet multi achieves 56.08% tree IoU in Villach, slightly below the proposed framework, but AdaptSegNet struggles with small buildings and thin roads.The authors report that multi-layer alignment can improve performance, especially for large-area objects such as forests.Source: D. Results · p. 8
  • CycleGAN’s road IoU is around 9% lower than ColorMapGAN’s for Villach, and its building IoU is 6% lower for Bad Ischl, while CycleGAN leads for trees.The authors note that the ColorMapGAN–CycleGAN comparisons need further analysis.Source: D. Results · p. 8
  • CycleGAN outputs can be lower-resolution, artifact-prone, and spectrally inconsistent across neighboring patches, whereas ColorMapGAN maps identical colors consistently across locations.ColorMapGAN’s tree smoothing may remove some patterns, which the authors suggest could explain CycleGAN’s advantage for trees in pair 1.Source: D. Results · p. 11
  • ColorMapGAN’s high-level feature matching preserves plausible rooftop and road colors better than histogram matching in the illustrated city pairs.Histogram matching produces cyan or dark-violet rooftops and reddish roads where ColorMapGAN outputs gray or black alternatives.Source: D. Results · p. 12
  • Building-color histograms for fake Roanne generated by ColorMapGAN resemble Béziers more closely than those produced by histogram matching.The authors expect resemblance, not identical histograms, across different cities.Source: D. Results · p. 12

E. Running Times

ColorMapGAN trains much faster than the other learning-based approaches, while non-learning methods are faster still but produce deficient results.

E. Running Times

  • Table IV reports training times for one iteration of ColorMapGAN and the other learning-based methods.Source: E. Running Times · p. 12
  • ColorMapGAN training is significantly shorter than that of the other learning-based approaches.The authors report about 6.5 minutes for 8,000 iterations; competing learning-based approaches take long hours with default parameters.Source: E. Running Times · p. 13
  • ColorMapGAN training takes about 6.5 minutes for 8,000 iterations.The comparison notes that the other learning-based approaches require long hours with their default parameters.Source: E. Running Times · p. 13
  • Gray-world runs almost instantly and histogram matching takes less than half a minute, but both non-learning methods have deficient output quality.Source: E. Running Times · p. 13

V. CONCLUDING REMARKS

The framework targets high-quality satellite-image maps under large spectral domain shifts. Across two city pairs, it outperformed existing approaches and generated fake images faster than some competitive unpaired translation methods.

V. CONCLUDING REMARKS

  • The framework outperformed existing approaches in city-to-city adaptation across two city pairs under large spectral shifts.It was presented to generate high-quality maps despite domain shift between training and test spectral bands.Source: §V · pp. 13–14
  • ColorMapGAN generated fake images significantly faster than some competitive unpaired image-to-image translation methods.Source: §V · p. 14
  • A proposed future direction is adaptation where object shapes, such as buildings, differ considerably in addition to the spectral difference.Source: §V · p. 14

S L Happy

The section gives brief biographies of S. L. Happy, Yuliya Tarabalka, and Pierre Alliez, covering education, research interests, and career roles.

S L Happy

  • S. L. Happy completed a joint MS–PhD degree at IIT Kharagpur in 2018 and works as a postdoctoral researcher at Inria Sophia Antipolis.His interests include machine learning, computer vision, hyperspectral image classification, medical image analysis, and facial expression analysis.Source: S L Happy · p. 16
  • Yuliya Tarabalka earned degrees in computer science and signal and image processing before receiving a joint PhD in signal and image processing and electrical engineering.Source: S L Happy · p. 16
  • Yuliya Tarabalka held research and postdoctoral roles from 2007 to 2019 and is currently research director at LuxCarta Technology.Her interests include image processing, pattern recognition, and efficient algorithms.Source: S L Happy · p. 16
  • Pierre Alliez is a senior researcher and team leader at Inria Sophia-Antipolis, specializing in geometry processing topics.His awards include a 2005 EUROGRAPHICS young researcher award and a 2011 European Research Council Starting Grant.Source: S L Happy · p. 16
Loading 1907.12859v2…