Source-linked AI summary
MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification
Daoyu Lin, Kun Fu, Yang Wang, Guangluan Xu, Xian Sun
TL;DR
Remote sensing classification often lacks the large labeled datasets required by supervised CNN methods. MARTA GANs learns representations from unlabeled data by using a generator and discriminator with multi-feature fusion, and it reports improved classification performance across remote sensing benchmarks compared with unsupervised alternatives.
Problem
Remote sensing CNN classification commonly requires large labeled datasets, while labeling the field’s extensive image repositories is prohibitively time-consuming and expensive.
Method
MARTA GANs uses a generator to create additional images and a discriminator as a feature extractor, combining mid- and high-level information through a multi-feature layer.
Results
MARTA GAN achieves 94.86±0.80% overall accuracy versus 87.76±0.64% for DCGAN on the UC-Merced dataset, while outperforming SCMF by 3.82% and LRFF by more than 4%.
Takeaways & Limitations
MARTA GANs provides a completely unsupervised representation-learning approach that learns interpretable representations from challenging remote sensing datasets.
Abstract
from arXiv · showhide
With the development of deep learning, supervised learning has frequently been adopted to classify remotely sensed images using convolutional networks (CNNs). However, due to the limited amount of labeled data available, supervised learning is often difficult to carry out. Therefore, we proposed an unsupervised model called multiple-layer feature-matching generative adversarial networks (MARTA GANs) to learn a representation using only unlabeled data. MARTA GANs consists of both a generative model $G$ and a discriminative model $D$. We treat $D$ as a feature extractor. To fit the complex properties of remote sensing data, we use a fusion layer to merge the mid-level and global features. $G$ can produce numerous images that are similar to the training data; therefore, $D$ can learn better representations of remotely sensed images using the training data provided by $G$. The classification results on two widely used remote sensing image databases show that the proposed method significantly improves the classification performance compared with other state-of-the-art methods.
I. INTRODUCTION
Remote sensing scene classification is difficult because images exhibit high intra-class and low inter-class variation, while deep models typically require extensive labeled data. MARTA GANs addresses this gap by learning representations from unlabeled data with adversarial training and multi-feature fusion.
- Remote sensing images vary substantially in object size, color, and angle, producing high intra-class variance and low inter-class variance.
- CNN-based scene-classification methods can succeed, but usually require enormous labeled datasets or fine-tuning from pretrained CNNs.
- GANs offer an unsupervised alternative for remote sensing because their generators can provide additional training data without labeling the full image repository.
- MARTA GANs learns remote sensing representations with unlabeled data by combining perceptual loss, feature-matching loss, and a multi-feature layer.
- The proposed method was evaluated on the UC-Merced Land-use and Brazilian Coffee Scenes datasets and reported higher overall classification accuracy than state-of-the-art unsupervised algorithms.
II. METHOD
MARTA GANs uses CNN-based generator and discriminator networks for adversarial learning on remote sensing images. The discriminator both distinguishes real from generated samples and serves as the feature extractor for classification.
- The generator produces samples from a prior noise distribution, while the discriminator distinguishes training images from generator-created samples.
- During classification, MARTA GANs treats the discriminator as a feature extractor rather than only as an adversarial classifier.
- Generated samples provide additional training data so the discriminator can learn a better representation of remote sensing images.
A. Training the discriminator
Discriminator training fixes the generator and optimizes the discriminator to assign high scores to real images and low scores to generated images.
- During discriminator training, the generator weights remain fixed.
- The discriminator maximizes D(x) for real training examples and minimizes D(x) for fake samples from the generator.
B. Training the generator
MARTA GANs trains the generator with perceptual and feature-matching objectives so its outputs resemble real training images and support discriminator learning.
- Generator objective: The generator is trained to fool the discriminator by maximizing D(G(z)), equivalently minimizing 1 − D(G(z)).Because D(G(z)) estimates the probability that a generated image is real, perceptual loss encourages generated images to resemble training samples.
- Adversarial training: The discriminator and generator alternately optimize opposing objectives, forming a minimax game.The discriminator adjusts its output on generated images while the generator seeks outputs that the discriminator classifies as real.
- Feature matching: Feature matching trains generated images to match expected activations in the discriminator’s multi-feature layer.The feature-matching loss compares expected feature values for real and generated samples.
- Final objective: The generator’s final training objective combines perceptual loss with feature-matching loss.The combined objective is Eqn. 5, used to train the generator.
C. Network architectures
The MARTA GANs architecture uses CNN-based generator and discriminator networks, with upsampling and multi-feature aggregation tailored to remote sensing images.
- Generator: The generator accepts 100 uniform random values and uses six deconvolutional layers to upsample 4 × 4 feature maps into 256 × 256 images.The input is reshaped into a four-dimensional tensor before spatial upsampling.
- Discriminator: The discriminator processes real and synthesized images with convolutions and concatenates 4 × 4 feature maps from its final layers.The maps are produced using 4 × 4 max pooling, 2 × 2 max pooling, and the identity function.
- Layer design: All convolutional and deconvolutional layers use 4 × 4 kernels with stride 2 to avoid uneven overlap in deconvolution.The generator uses ReLU except at its tanh output layer, while the discriminator uses LeakyReLU.
III. EXPERIMENTS
MARTA GANs was evaluated on the UC Merced Land Use and Brazilian Coffee Scenes datasets using cross-validation and a regularized linear classifier.
- Datasets: The experiments used the UC Merced Land Use and Brazilian Coffee Scenes datasets to evaluate MARTA GANs.These datasets were used to verify the effectiveness of the proposed method.
- Evaluation protocol: Both datasets were evaluated with a 5-fold cross-validation protocol and a regularized linear L2-SVM classifier.The implementation used TensorLayer 1, extended from Google TensorFlow.
A. UC Merced dataset
On the UC-Merced dataset, MARTA GANs learns representations from multi-feature discriminator layers and achieves stronger classification than DCGAN and other tested unsupervised methods. The experiments also identify f3 and the ℓfinal objective as the strongest evaluated choices, while visualizations show semantically meaningful global features.
- Feature and training choices: f3 achieved the highest accuracy among the evaluated multi-feature representations.f3 combines the last three convolutional layers, retaining high-level information while adding mid-level information.
- Feature and training choices: ℓfinal achieved the best performance among the evaluated perceptual and feature-matching loss choices.The comparison also evaluated data augmentation, which the authors describe as effective for reducing overfitting in the large deep network.
- Classification results: 94.86±0.80% overall accuracy was achieved by MARTA GAN, compared with 87.76±0.64% for DCGAN.The authors attribute the approximately 7% improvement to merging mid-level and global features through the multi-feature layer.
- Classification results: MARTA GAN achieved higher accuracy than DCGAN in several closely related scene classes, including dense residential, building, medium residential, and sparse residential.It also reached 100.00% accuracy in some categories, such as Beach and Airplane.
- Representation visualization: The 2-D visualization indicates that multi-feature representations contain abstract semantic information because closely related classes are also close in the embedded space.The visualization was produced by embedding UC-Merced global representations with t-SNE.
- Comparison with reference methods: MARTA GANs achieved the highest classification accuracy among the tested unsupervised methods, outperforming SCMF by 3.82% and LRFF by more than 4%.The comparison notes that supervised methods above 99% were fine-tuned from pretrained models using large labeled datasets, whereas MARTA GANs requires fewer parameters.
B. Brazilian Coffee Scenes dataset
The Brazilian Coffee Scenes dataset contains multispectral coffee and non-coffee scenes and tests MARTA GANs' generalization. Despite its challenging variability, MARTA GANs outperforms BIC, although results are worse than on UC-Merced.
- Dataset: Brazilian Coffee Scenes contains 2,876 SPOT-sensor scenes in green, red, and near-infrared bands, evenly split between coffee and non-coffee classes.Each scene has 64 × 64 pixels, with 1,438 tiles per class.
- Dataset: The dataset challenges classification through intra-class variability from crop management, plant age, spectral distortions, and shadows.
- Results: Despite reducing the task from 21 classes to 2, MARTA GANs performs significantly worse here than on UC-Merced.
- Results: MARTA GANs nevertheless achieves better results than BIC on Brazilian Coffee Scenes.
IV. CONCLUSION
The paper concludes that MARTA GANs learns interpretable, multi-scale representations from unlabeled remote sensing data. It also identifies higher-quality generation and semi-supervised classification as future extensions.
- Conclusion: MARTA GANs is a completely unsupervised representation-learning algorithm that can learn interpretable representations from challenging remote sensing datasets.
- Conclusion: Its multiple-feature-matching layer learns multi-scale spatial information for high-resolution remote sensing.
- Future work: Future work includes generating higher-quality remote sensing samples and using semi-supervised learning to improve classification accuracy.