PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification

Keiller Nogueira, Otávio A. B. Penatti, Jefersson A. dos Santos

arXiv:1602.01517v1cs.CV

TL;DR

Remote sensing applications often lack the labeled data and computational resources needed to train new ConvNets, creating a need to understand how existing networks can be reused. The paper systematically evaluates full training, fine-tuning, and feature extraction across six ConvNets and three datasets. Fine-tuning tends to perform best, especially when its features are classified with a linear SVM, and the study reports state-of-the-art results on all three datasets.

  • Problem

    Training new ConvNets is difficult in many remote sensing applications because it requires substantial labeled data and computational power.

  • Method

    The paper compares full training, fine-tuning, and feature extraction using six popular ConvNets across three remote sensing datasets.

  • Results

    Fine-tuning tends to be the best strategy, with fine-tuned features combined with a linear SVM providing the best results.

  • Takeaways & Limitations

    The results support using fine-tuning and an external linear SVM when exploiting existing ConvNets for remote sensing classification.

  • Takeaways & Limitations

    The authors note that broader evaluation is needed to determine whether these conclusions apply to other domains.

Abstract

from arXiv · show

We present an analysis of three possible strategies for exploiting the power of existing convolutional neural networks (ConvNets) in different scenarios from the ones they were trained: full training, fine tuning, and using ConvNets as feature extractors. In many applications, especially including remote sensing, it is not feasible to fully design and train a new ConvNet, as this usually requires a considerable amount of labeled data and demands high computational costs. Therefore, it is important to understand how to obtain the best profit from existing ConvNets. We perform experiments with six popular ConvNets using three remote sensing datasets. We also compare ConvNets in each strategy with existing descriptors and with state-of-the-art baselines. Results point that fine tuning tends to be the best performing strategy. In fact, using the features from the fine-tuned ConvNet with linear SVM obtains the best results. We also achieved state-of-the-art results for the three datasets used.

1 Introduction

Remote sensing requires effective visual feature extraction, but conventional descriptors do not straightforwardly capture its spatial and spectral characteristics. This paper analyzes three ways to exploit existing ConvNets and compares them across datasets and descriptors.

  • Remote sensing feature extraction remains challenging because classical techniques do not straightforwardly encode spatial information in remotely sensed images.
  • Deep ConvNets learn features and classifiers jointly from data, but their use involves complex tuning, computational burden, overfitting risk, and limited interpretability.
  • The study evaluates full training, fine-tuning, and pre-trained ConvNets as feature extractors in remote sensing scenarios with limited labeled data.
  • The experiments compare six existing ConvNets across three remote sensing datasets, alongside low- and mid-level descriptors and state-of-the-art baselines.
  • Fine-tuning tends to perform best, and the study reports state-of-the-art classification results on all three remote sensing datasets used.

2 Related Work

Remote sensing has motivated specialized feature descriptors because aerial scenes and multiple spectral bands challenge conventional visual representations. Related work develops and compares full training, fine-tuning, and feature-extraction uses of ConvNets.

  • Remote sensing descriptors must address non-visible spectral information and aerial-scene characteristics that limit direct application of traditional techniques.
  • Full training provides dataset-specific features and architectural control but requires considerable amounts of annotated data.
  • Fine-tuning resumes training of a pre-trained ConvNet on new data while leveraging generic early-layer features such as edges and color blobs.
  • Feature extraction removes the final classification layer and uses outputs from preceding layers as deep features without parameter adjustment.
  • This paper differs from earlier studies by evaluating alternative practices for exploiting existing ConvNets rather than only studying architecture or training protocols.

3 Background Concepts

ConvNets process inputs through layered units that learn increasingly complex visual representations, optimize a loss during training, and produce class probabilities through a classifier. Their training combines feed-forward computation with backpropagation-based parameter updates.

  • Processing Units: An artificial neuron combines input variables with weights and a bias, then applies an activation function to produce an output.
  • Processing Units: ReLU activation outputs max(0, z), helping avoid saturation, induce sparsity, and reduce vanishing-gradient problems compared with sigmoid and tanh functions.
  • Network Components: Convolutional layers capture image features, with early layers learning edges and corners and later layers learning structures, objects, and shapes.
  • Network Components: Convolution uses a fixed-size window that moves across an image, while pooling selects useful features for subsequent layers and preserves spatial location.
  • Network Components: Fully connected layers connect every previous-layer neuron to each neuron in the next layer, but their many parameters make overfitting possible.
  • Network Components: Softmax converts classifier outputs into a K-dimensional categorical probability distribution for the input sample.
  • Training: Training minimizes a loss function by using optimization methods such as stochastic gradient descent to update weights and biases.
  • Training: Backpropagation computes error derivatives through the network, while feed-forward propagation carries information toward the classifier before the errors are propagated backward.

4 Strategies for Exploiting ConvNets

The paper compares full training, fine-tuning, and feature extraction for reusing ConvNets on new remote-sensing datasets. These strategies trade dataset-specific adaptation and control against data, computation, and overfitting constraints.

  • Full training: Full training requires substantial labeled data and computational resources, while convergence and overfitting remain difficult to control.The network’s highly non-convex optimization makes convergence uncertain and tuning nontrivial.
  • Full training: Full training randomly initializes and trains an existing ConvNet on the target dataset, providing dataset-specific features and architectural control.The paper evaluates existing architectures rather than designing an entirely new ConvNet.
  • Fine-tuning: Fine-tuning exploits generic early-layer features while adapting later layers that are more specific to the original training classes.Reduced learning rates constrain parameter updates to small variations from the pre-trained state.
  • Fine-tuning: Fine-tuning resumes training from pre-trained parameters and adapts them to the target dataset, either across all layers or only higher-level layers.The latter option keeps some earlier layers fixed to address overfitting concerns.
  • Feature extraction: Feature extraction ignores the final classification layer, selects a network layer, and combines its deep features with a machine-learning classifier.This paper uses features from the last layer before classification with a linear classifier, avoiding retraining or tuning.

5 Experimental Setup

The experimental setup evaluates strategies for exploiting deep features in remote-sensing scene classification. It includes a protocol and dataset examples, with Figure 4 illustrating the UCMerced Land Use Dataset.

  • Experimental setup: The experiments evaluate which strategies are most suitable for exploiting deep features in remote-sensing scene classification.The paper introduces the protocol, datasets, descriptors, and ConvNets used for evaluation.
  • Dataset examples: Figure 4 provides examples from the UCMerced Land Use Dataset.

5.1 Datasets

The study uses three remote-sensing datasets spanning visible-spectrum aerial scenes, globally sourced high-resolution scenes, and multispectral coffee-crop classification. Their differing visual properties support evaluation of strategy robustness and effectiveness.

  • Dataset selection: The three datasets cover visible-spectrum aerial land-use scenes, globally sourced high-resolution scenes, and multispectral coffee-versus-non-coffee scenes.They represent distinct visual properties and remote-sensing settings.
  • UCMerced Land Use Dataset: 2,100 UCMerced images are equally divided among 21 land-use classes, with 256 × 256-pixel aerial scenes from United States locations.
  • RS19 Dataset: 1,005 RS19 images are divided into 19 classes, with samples collected from regions worldwide and variations in resolution, scale, orientation, and illumination.
  • Brazilian Coffee Scenes: 2,876 Brazilian Coffee Scenes images are equally divided between coffee and non-coffee classes and generated from multispectral SPOT imagery over four counties in Minas Gerais.The images use green, red, and near-infrared bands.
  • Brazilian Coffee Scenes: The Brazilian Coffee Scenes dataset has high intraclass variance, varying plant ages, and shadow-related spectral distortions.These properties make the dataset challenging for classification.

5.2 Classical Feature Extraction Strategies

The study compares deep features against classical global and mid-level visual descriptors. The classical evaluation spans color, texture, gradient, and BoVW configurations with varied sampling, coding, and pooling choices.

  • Global low-level descriptors: Seven global low-level descriptors based on color, texture, and gradient properties are evaluated against deep features.The selection includes descriptors such as ACC, BIC, LCH, SASI, LAS, GIST, and HOG.
  • Global low-level descriptors: HOG is tested with cell sizes of 14 × 14, 20 × 20, 40 × 40, and 80 × 80 pixels while retaining nine orientation bins.
  • BoVW descriptors: BoVW represents images through codebooks of visual patches and statistics of visual-word occurrences.The paper treats BoVW as a mid-level representation with a history of strong computer-vision performance.
  • BoVW descriptors: BoVW configurations vary sampling, low-level descriptor, pooling, coding scheme, and codebook size.Sampling may be sparse or dense; descriptors include SIFT and OpponentSIFT; pooling includes average, max, and WSA.
  • BoVW descriptors: SIFT is used for UCMerced and RS19, whereas OpponentSIFT is used for Brazilian Coffee because color may provide greater discriminating power.

5.3 ConvNets

The study evaluates several established ConvNet architectures under feature extraction, fine-tuning, and full-training settings. The networks differ in architecture, feature-extraction layer, computational demands, and the training strategies feasible for each model.

  • Selected ConvNets: Seven established ConvNet families are evaluated, including PatreoNet, AlexNet, CaffeNet, GoogLeNet, VGG16, OverFeatS, and OverFeatL.Most networks were implemented in Caffe, whereas OverFeat models were provided separately.
  • PatreoNet: PatreoNet is used only for full training because it lacks pre-training on large datasets.Its architecture contains three convolutional, three pooling, and three fully connected layers, including softmax.
  • AlexNet and GoogLeNet: AlexNet and GoogLeNet are evaluated as feature extractors, fine-tuned, and fully trained from scratch across the datasets.AlexNet extraction uses its last fully connected layer, producing 4,096-dimensional vectors; GoogLeNet uses its last pooling layer, producing 1,024-dimensional vectors.
  • VGG16: VGG16 is used for feature extraction and fine-tuning on UCMerced and RS19, but not for Coffee fine-tuning or full training.The feature vector extracted from its last fully connected layer has 4,096 dimensions.
  • Training constraints: VGG16 training fails to converge in the excluded settings because its memory requirements restrict batch sizes, while larger batches and related parameters support ConvNet convergence.OverFeatS and OverFeatL are used only as feature extractors because no models were provided for fine-tuning or full training, yielding 4,096-dimensional vectors.

5.4 Experimental protocol

Experiments use five-fold cross-validation, with fold-specific training, validation, and testing assignments. Accuracy is averaged across classes and folds, while pre-trained feature extraction involves no additional training.

  • Cross-validation: All experiments use five-fold cross-validation with approximately equal-sized, non-overlapping folds.UCMerced and RS19 folds are class-unbalanced, whereas the Brazilian Coffee folds are balanced.
  • Data splits: Each training or fine-tuning run uses three folds for training, one for validation, and one for testing.When fold assignments change, the network is trained or fine-tuned from the beginning.
  • Parameter settings: Fine-tuning and full training preserve the original authors’ parameters except for two parameters varied according to Table 2.Feature extraction with a pre-trained network involves no training and therefore has no parameters to vary.
  • Evaluation: Performance is reported as mean accuracy and standard deviation across the five folds.For each fold, accuracy is first averaged across classes before the final fold average is computed.
  • Hardware: Experiments run on an Intel i7 system with 64 GB RAM and two GPUs: a GTX770 and a GTX Titan X.The GPUs provide 4 GB and 12 GB of internal memory, respectively.

6 Results and Discussion

The experiments compare ConvNet feature extraction, fine tuning, and full training across three remote sensing datasets and against descriptor and state-of-the-art baselines. Fine tuning generally performs best, especially when fine-tuned features are classified with linear SVM, although dataset characteristics affect the relative performance of strategies.

  • 6.1 Generalization Power Evaluation: ConvNet features achieved at least 90% average accuracy on UCMerced and RS19, outperforming the evaluated low- and mid-level descriptors.On UCMerced, CaffeNet, AlexNet, and VGG16 exceeded 93% average accuracy, while RS19 also showed ConvNets above 90%.
  • 6.1 Generalization Power Evaluation: 87.03% ± 1.17 was the highest Brazilian Coffee Scenes descriptor accuracy, achieved by BIC and exceeding the ConvNet descriptors in that dataset.VGG16 reached 85.36% ± 2.08%, while BIC benefited from encoding color and texture-related regions.
  • 6.2 Comparison of ConvNets Strategies: Fine tuning was usually the best strategy across all three datasets, with the largest advantage over alternatives on UCMerced and RS19.On Coffee Scenes, the difference was smaller, but fine-tuned descriptors remained slightly superior.
  • 6.2 Comparison of ConvNets Strategies: Fine-tuned initialization tended to outperform full training, while full training was weak on UCMerced and RS19 but improved performance on Coffee Scenes.The authors associate the fine-tuning advantage with better initialization in the search space and the Coffee Scenes difference with its multispectral domain.
  • 6.2 Comparison of ConvNets Strategies: Replacing the ConvNet softmax layer with linear SVM usually improved performance, particularly for fine-tuned and fully trained descriptors.The improvement was smaller on the Coffee Scenes dataset.
  • 6.3 Comparison with Baselines: 99.47±0.50 was the best UCMerced result, obtained by classifying fine-tuned GoogLeNet features with linear SVM.The same configuration also yielded the best result on RS19, while fine-tuned CaffeNet achieved 94.45 ± 1.20 on Coffee Scenes, reported as the current state of the art there.

7 Conclusions

The study evaluates three strategies for exploiting existing ConvNets in remote sensing and finds that fine tuning, especially with a linear SVM, tends to perform best. It also reports state-of-the-art results across the three datasets and identifies future work on spectral-spatial representation and broader applicability.

  • The experiments evaluate full training, fine tuning, and pre-trained ConvNets used as feature extractors across remote sensing datasets.
  • Fine-Tuned Descriptors from GoogLeNet achieved the highest accuracy rates on the RS19 Dataset.
  • Fine-Tuned Descriptors from CaffeNet achieved the highest accuracy rates on the Brazilian Coffee Scenes Dataset.
  • Fine tuning tends to be the best strategy, particularly when fine-tuned features are classified with a linear SVM.
  • The study compares multiple ConvNets and strategies with traditional descriptors and state-of-the-art baselines, obtaining state-of-the-art results on all three datasets.
  • Future work includes improving spectral-spatial representations and evaluating whether the conclusions apply beyond remote sensing.
Loading 1602.01517v1…