PaperlayerView exact revision on arXiv

Source-linked AI summary

Land Use Classification in Remote Sensing Images by Convolutional Neural Networks

Marco Castelluccio, Giovanni Poggi, Carlo Sansone, Luisa Verdoliva

arXiv:1508.00092v1cs.CV

TL;DR

Remote-sensing scene classification must handle limited training data, which can cause large CNNs to overtrain. The paper compares CaffeNet and GoogLeNet with training from scratch, feature extraction, and fine-tuning, and reports strong performance across two different datasets, including gains over reference methods.

  • Problem

    Available remote-sensing datasets are too small to train large CNNs correctly without overtraining and poor test-set generalization.

  • Method

    The paper evaluates CaffeNet and GoogLeNet using training from scratch, pre-trained feature vectors, and fine-tuning on remote-sensing scene-classification data.

  • Results

    CNNs achieve almost 3% and almost 5% gains over the best reference on UC Merced Land Use and Brazilian Coffee Scenes, respectively.

  • Takeaways & Limitations

    Careful fine-tuning generally provides very good results, while shallow feature-vector adaptation gives up much of CNNs’ potential.

Abstract

from arXiv · show

We explore the use of convolutional neural networks for the semantic classification of remote sensing scenes. Two recently proposed architectures, CaffeNet and GoogLeNet, are adopted, with three different learning modalities. Besides conventional training from scratch, we resort to pre-trained networks that are only fine-tuned on the target data, so as to avoid overfitting problems and reduce design time. Experiments on two remote sensing datasets, with markedly different characteristics, testify on the effectiveness and wide applicability of the proposed solution, which guarantees a significant performance improvement over all state-of-the-art references.

I. INTRODUCTION

Remote-sensing scene classification is difficult because scenes combine variable land covers and objects across scales, while different categories can remain visually similar. The paper evaluates CNN architectures and training strategies to address limited remote-sensing data, reporting gains on two datasets.

  • Scene classification assigns images to semantic categories, but increasing abstraction from pixels to objects and scenes makes the task progressively harder.
  • High intra-class variability and low inter-class distance arise from changes in scale, orientation, and shared land covers or objects across categories.The UC-Merced example differs primarily in building density between dense and medium residential classes.
  • Pixel- and object-based spectral, textural, and geometrical features become less effective as classification targets scene semantics.The paper motivates more complex descriptors for capturing scene-level meaning.
  • Deep neural networks learn and combine internal descriptions at increasing abstraction levels, and performance figures favor them for these tasks.
  • CNN experiments compare CaffeNet and GoogLeNet with training from scratch and fine-tuning of pre-trained networks on two markedly different remote-sensing datasets.The design addresses the scarcity of remote-sensing training data.
  • Almost 3% and almost 5% gains over the best reference are obtained on UC Merced Land Use and Brazilian Coffee Scenes, respectively.

II. RELATED WORK

Related work develops increasingly sophisticated local, spatial, color, and fused descriptors for remote-sensing scene classification. Earlier ConvNet use was competitive but used pre-trained networks only as shallow feature generators rather than training on remote-sensing data.

  • BOVW quantizes extracted features using a learned visual-word dictionary and feeds their histogram representation to a classifier, typically an SVM.
  • Spatial pyramid and randomized partition methods augment histogram features by modeling spatial layout at multiple resolutions or through weighted partition patterns.
  • SCK, SPCK, and PSR extend spatial modeling to capture absolute and relative feature arrangements, with PSR targeting greater rotation and translation robustness.
  • Other approaches use dense SIFT representations, Earth mover’s distance approximations, feature augmentation, color descriptors, and combinations of structural, shape, textural, and color features.
  • A three-layer Perceptron can classify handwritten digits with accuracy exceeding 97% when configured with 784, 30, and 10 neurons.
  • The earlier ConvNet study used pre-trained CNN outputs as shallow feature vectors for SVM classification and performed no training on remote-sensing data.It remained competitive with the previous state of the art.

III. CONVOLUTIONAL NEURAL NETWORKS

CNNs address MLP scalability through local receptive fields, shared weights, and layered feature extraction. Their practical use became viable with improved computing, datasets, and learning methods, while convolutional and pooling layers build progressively more expressive representations.

  • A neuron computes an output by applying a nonlinearity to a weighted input sum compared with a threshold.
  • MLPs become difficult for image problems because densely connected layers require many weights, large training sets, and substantial computational power.
  • CNNs sharply reduce free parameters by restricting neurons to local receptive fields and sharing weights across a layer.
  • CNN feature banks convolve filters with the three input color bands to produce parallel feature maps for subsequent layers.The described CaffeNet first layer produces 96 feature maps.
  • Large CNN training became feasible through affordable GPU computing, large labeled image datasets, and improved learning solutions.
  • CNN layers combine low-level features from smaller image windows into more expressive features representing larger portions of the image.

1) Convolutional layers:

Convolutional layers learn filters over local image windows, while pooling reduces spatial input size and can provide translation invariance. Their behavior is controlled by window support, stride, filter count, and padding.

  • 1) Convolutional layers:: Convolutional layers learn filters over image windows, with filter count, spatial support, stride, and optional zero-padding controlling the output.
  • 1) Convolutional layers:: Pooling applies local nonlinear operations such as max() to reduce input size, lower parameter requirements, and provide some translation invariance.Pooling uses its own window support and stride hyper-parameters.

2) Pooling layers:

These layers use inhibition-inspired schemes to improve generalization, typically with sigmoid rather than ReLU neurons.

  • Inhibition-inspired schemes aim to improve generalization.
  • These schemes are typically used with sigmoid neurons.
  • They are typically not used with ReLU neurons.

3) Normalization layers:

Fully connected layers are typically placed near the network’s end to summarize lower-level information for the final decision.

  • Fully connected layers are typically used as the last few network layers.
  • Removing constraints lets them better summarize information from lower-level layers.
  • Earlier size-reducing layers keep their complexity affordable despite full connectivity.

IV. USING CNNS FOR REMOTE SENSING SCENE

The work applies CaffeNet and GoogLeNet to remote-sensing scene classification, comparing training from scratch with two ways of adapting pre-trained CNNs. Limited remote-sensing training data motivates transfer from natural-image networks, especially when low-level image characteristics are similar.

  • Architectures: The study uses CaffeNet and GoogLeNet for remote-sensing scene classification.
  • Architectures: CaffeNet contains five convolutional layers, pooling after each, and three fully connected layers.
  • Architectures: GoogLeNet uses inception modules that reduce depth before expensive filters, enabling multiple filters to process features at different scales.
  • Transfer learning: Limited remote-sensing training data creates a major problem when applying CNNs developed for natural-image datasets.
  • Transfer learning: Pre-trained CNNs can be repurposed because lower-layer features such as edges and color blobs may transfer across tasks.
  • Learning modalities: The comparison includes training from scratch, fine-tuning selected high-level layers, and using a pre-trained network’s penultimate output as features.

V. EXPERIMENTAL RESULTS

Experiments evaluate the CNN approach on two remote-sensing datasets with different characteristics using multiple training modalities and moderate data augmentation.

  • Datasets: The experiments use UC-Merced aerial optical images and Brazilian Coffee Scenes satellite images with an infrared band.
  • Datasets: UC-Merced supports extensive comparison with prior work because many researchers use the dataset.
  • Experimental setup: Training iterations were set through preliminary experiments for each modality and dataset.
  • Experimental setup: Fine-tuning uses a first-layer learning rate one tenth of the others, while feature-vector training updates only the last fully connected layer.
  • Experimental setup: All experiments use mirroring and random cropping to increase the effective training-set size.

A. UC-Merced

On UC-Merced, fine-tuned CNNs performed best, with GoogLeNet selected as the proposed solution. The method outperformed reference approaches across the dataset and most classes, while errors concentrated among visually similar categories.

  • Fine-tuning reached 95.48% overall accuracy with CaffeNet and 97.10% with GoogLeNet, outperforming training from scratch by about 10% and 5%, respectively.The feature vector approach remained 1–3% below fine-tuning.
  • Fine-tuning required far fewer training iterations than training from scratch and already produced 95.12% and 96.48% at 5,000 iterations.Training from scratch used 100,000 iterations, compared with 20,000 for fine-tuning.
  • GoogLeNet with fine-tuning was selected because it consistently provided the best performance while being slightly less computationally demanding than CaffeNet.The reported proposed approach uses GoogLeNet fine-tuned for 20,000 iterations.
  • The proposed method achieved a minimum gain of almost 3% over comparable reference methods on UC-Merced.The comparison used methods evaluated under the same five-fold cross-validation protocol.
  • The proposed method performed best almost uniformly across classes, although dense residential accuracy was lowest at 84.5% because of closely related neighboring classes.For dense residential, reference methods performed more than 10% worse.
  • Only 8 of 420 images in fold #1 were misclassified, and the errors involved classes with very close visual neighbors.The paper notes that some errors could also challenge a human photointerpreter.

B. Brazilian Coffee Scenes

The Brazilian Coffee Scenes dataset differs substantially from UC-Merced in spectral characteristics, sample availability, and classification difficulty. On this challenging dataset, GoogLeNet trained from scratch achieved the best result, outperforming prior references despite weaker feature-vector performance.

  • The dataset uses green, red, and near-infrared SPOT imagery and contains balanced coffee and non-coffee classes after mixed tiles were discarded.The retained data were organized into five almost equally sized folds.
  • The dataset differs from UC-Merced because its spectral bands are unlike the RGB ImageNet pre-training data and it provides many more samples per class.These differences make training from scratch a more interesting option for CNNs.
  • GoogLeNet trained from scratch provided the best overall result, at almost 92% accuracy.Training from scratch performed better here than feature-vector extraction, partly because the dataset has more samples per class.
  • Results were significantly worse than on UC-Merced despite the Brazilian dataset having only two classes, reflecting its challenging variability.Reported sources of variability include crop management, plant age, spectral distortions, and shadows.
  • Feature-vector performance was relatively poor, attributed to the marked difference between Brazilian imagery and the ImageNet data used for pre-training.Fine-tuning also achieved its best results at 10,000 iterations.
  • The best result was almost 5% higher than the top prior result from BIC and 7% higher than CNNs using the feature vector approach.The BIC comparison is reported in Table IV, whose reference data come from.

VI. CONCLUSIONS

The study evaluates CNNs for remote sensing scene classification using two architectures and three design modalities across two datasets with different properties. Careful fine-tuning performs well, while the next major challenge concerns data from imaging modalities beyond aerial images.

  • VI. CONCLUSIONS: CNNs were evaluated with two architectures and three design modalities on two remote sensing datasets with different properties.The architectures were CaffeNet and GoogLeNet; the modalities included training from scratch and adaptations of pre-trained networks.
  • VI. CONCLUSIONS: Almost 3% better performance than the best reference was obtained on the UC-Merced dataset.The conclusion attributes this result to the proposed CNN method.
  • VI. CONCLUSIONS: Near-perfect performance on aerial images shifts the next major challenge toward classifying data acquired with other imaging modalities, including SAR.The paper identifies Synthetic Aperture Radar as an important direction for future research.
Loading 1508.00092v1…