PaperlayerView exact revision on arXiv

Source-linked AI summary

Algorithms for Semantic Segmentation of Multispectral Remote Sensing Imagery using Deep Learning

Ronald Kemker, Carl Salvaggio, Christopher Kanan

arXiv:1703.06452v3cs.CVcs.AI

TL;DR

Non-RGB remote sensing lacks the large labeled datasets needed to pretrain DCNNs, motivating alternatives to real-image pretraining. The paper generates synthetic MSI to initialize DCNN segmentation frameworks and evaluates them on the new RIT-18 UAS dataset. Synthetic initialization transfers to real imagery and is reported to reduce overfitting while improving segmentation performance.

  • Problem

    Non-RGB remote sensing has insufficient large, manually annotated datasets for conventional DCNN pretraining, creating a label-scarcity gap.

  • Method

    The paper uses DIRSIG to generate synthetic MSI and label maps for DCNN initialization, then applies the initialized networks to real-MSI semantic segmentation.

  • Results

    Syntheticly initialized models effectively transfer features to real imagery, improve semantic-segmentation performance, and make RefineNet-Sim less prone to overfitting.

  • Takeaways & Limitations

    RIT-18 provides a benchmark for MSI semantic segmentation, while synthetic imagery offers a way to train end-to-end DCNN frameworks despite limited annotated data.

  • Takeaways & Limitations

    Synthetic-to-real transfer can be difficult because synthetic and real imagery have different feature-space distributions, known as the synthetic gap.

Abstract

from arXiv · show

Deep convolutional neural networks (DCNNs) have been used to achieve state-of-the-art performance on many computer vision tasks (e.g., object recognition, object detection, semantic segmentation) thanks to a large repository of annotated image data. Large labeled datasets for other sensor modalities, e.g., multispectral imagery (MSI), are not available due to the large cost and manpower required. In this paper, we adapt state-of-the-art DCNN frameworks in computer vision for semantic segmentation for MSI imagery. To overcome label scarcity for MSI data, we substitute real MSI for generated synthetic MSI in order to initialize a DCNN framework. We evaluate our network initialization scheme on the new RIT-18 dataset that we present in this paper. This dataset contains very-high resolution MSI collected by an unmanned aircraft system. The models initialized with synthetic imagery were less prone to over-fitting and provide a state-of-the-art baseline for future work.

1. Introduction

Semantic segmentation benefits from DCNNs, but non-RGB remote sensing lacks the large labeled datasets that support conventional pretraining. The paper therefore uses synthetic MSI to initialize DCNNs before adapting them to real MSI, evaluated on the six-band RIT-18 UAS dataset.

  • Motivation: DCNNs have improved RGB semantic segmentation through pretraining on datasets containing more than one million labeled images.These networks are typically adapted from image classification to semantic segmentation.
  • Motivation: Non-RGB remote sensing has substantially scarcer labels, making pretraining on hand-labeled real images currently infeasible.Existing hyperspectral semantic-segmentation benchmarks may consist of a single image mosaic.
  • Approach: The paper explores automatically labeled synthetic MSI as an alternative source for pretraining DCNN-based semantic-segmentation systems.This substitutes large quantities of generated data for unavailable manually annotated non-RGB imagery.
  • Approach: DIRSIG generates diverse synthetic MSI and label maps under varied weather and lighting conditions, which initialize DCNNs before real-MSI segmentation fine-tuning.The pretrained DCNN is combined with two fully convolutional semantic-segmentation models using real MSI.
  • Dataset: RIT-18 is a new six-band, high-resolution MSI dataset acquired by an unmanned aircraft system for evaluating non-RGB semantic-segmentation frameworks.Training, validation, and testing folds support standardized comparison and more realistic deployment evaluation.

2. Related Work

Related work spans end-to-end DCNNs for RGB segmentation, learned features for non-RGB imagery, GEOBIA, benchmark datasets, and synthetic-data pretraining. The paper addresses MSI label scarcity with RIT-18 and synthetic imagery while adapting fully convolutional models.

  • RGB semantic segmentation: RGB semantic segmentation commonly initializes a convolutional network on ImageNet and appends a segmentation network, reducing fine-tuning data requirements from millions to thousands of images.The segmentation network reconstructs feature responses to the input dimensions before assigning pixel labels.
  • RGB semantic segmentation: Earlier fully convolutional models improved multiscale recognition and boundary sharpness through deconvolution, spatial unpooling, or atrous convolutions, but some still produced coarse label maps.CRF post-processing was also used to sharpen boundaries.
  • RGB semantic segmentation: SharpMask and RefineNet instead use skip-connections to incorporate refinement into end-to-end semantic segmentation rather than relying on separate boundary-sharpening post-processing.The paper adapts these fully convolutional models for MSI.
  • Non-RGB sensors: Non-RGB segmentation research has used learned spatial-spectral features, stacked autoencoders, and self-taught learning because annotated hyperspectral and multispectral data are scarce.Self-taught approaches advanced the state of the art across three benchmark datasets.
  • Non-RGB sensors: Features learned from one hyperspectral image failed to transfer properly to another dataset, motivating unsupervised training across multiple images for better generalization.This limitation is distinct from self-taught methods trained on larger quantities of unlabeled data.
  • High-resolution remote sensing: GEOBIA handles high-resolution spatial variability with image objects or superpixels, whereas the paper’s DCNN approach is end-to-end and avoids time-consuming unsupervised segmentation.The paper also identifies faster processing and increased classification performance as advantages of its approach.
  • MSI datasets: RIT-18 contributes very-high-resolution UAS imagery with 18 object classes and six VNIR bands, including two additional NIR bands beyond ISPRS benchmarks.Its unbalanced class distribution makes strong performance a low-shot learning challenge.
  • Synthetic imagery: Synthetic imagery is cheaper and easier to obtain than manually annotated data, but the synthetic gap can hinder transfer from synthetic to real imagery.Domain adaptation techniques have been used to mitigate this distribution difference.

3. Methods

The methods generate synthetic multispectral imagery with DIRSIG for DCNN pretraining, then adapt SharpMask and RefineNet for MSI semantic segmentation. The paper also introduces MRS+SCAE and compares these approaches with established segmentation methods.

  • 3.1. Synthetic Image Generation using DIRSIG: DIRSIG generates large labeled synthetic MSI datasets by simulating scenes, sensors, platforms, weather, lighting, and flight conditions.The synthetic scene contains 109 labeled classes, and the resulting dataset includes 4.7 million training patches and 520 thousand additional patches.
  • 3.1. Synthetic Image Generation using DIRSIG: Because the DIRSIG scene is accurate only to 0.5-meter GSD, imagery is generated at 0.5, 0.75, and 1 meter to encourage scale-invariant features.RIT-18 has approximately ten times higher spatial resolution, making scale variation a mitigation for the synthetic-to-real resolution gap.
  • 3.2. Fully Convolutional Segmentation Models: SharpMask and RefineNet are adapted as fully convolutional MSI segmentation models that restore pixel-level outputs through learned refinement modules.SharpMask merges low-level spatial features with high-level semantic features, while RefineNet adds residual convolution, multi-resolution fusion, and chained residual pooling components.
  • 3.2.1. SharpMask: SharpMask uses the first four ResNet-50 macro-layers, retains batch normalization, and combines convolutional and segmentation features to recover the original image dimensions.Its bridge network introduces feature variability before the segmentation refinement modules.
  • 3.2.2. RefineNet: RefineNet uses all five ResNet-50 macro-layers and pools features across multiple window sizes to capture background context for spectrally and spatially similar classes.The model uses four pooling window sizes, while the architecture illustration shows two as an example.
  • 3.3. Comparison Semantic Segmentation Algorithms: The comparison includes pixel classifiers, mean-pooled pixels, MICA, SCAE, a GEOBIA-inspired method, and the introduced MRS+SCAE method.MRS+SCAE combines SCAE spatial-spectral features with mean-shift-derived object features across multiple segmentation scales.

4. RIT-18 Dataset

RIT-18 is a very-high-resolution, six-band MSI benchmark collected by a UAS and annotated for 18 semantic classes. Its geographically separated folds and severe class imbalance create a challenging evaluation setting.

  • 4. RIT-18 Dataset: RIT-18 is a 4.7-cm-resolution benchmark for UAS-collected multispectral semantic segmentation.The dataset targets applications including crop health sensing, nutrient prescription, irrigation engineering, and crop-field variability.
  • 4. RIT-18 Dataset: Training and validation data come from one park location, while test data come from a different location sharing many class types.The spatial separation provides distinct collection locations for evaluation within Hamlin Beach State Park.
  • 4. RIT-18 Dataset: The imagery was collected with a Tetracam Micro-MCA6 sensor containing six independent optical systems with VNIR bandpass filters.The sensor was mounted on a UAS platform for the data collection flight.
  • 4. RIT-18 Dataset: Each fold contains a six-band orthomosaic, a valid-data mask, and a corresponding hand-annotated classification map.The orthomosaics have dimensions of 9,393×5,642 for training, 8,833×6,918 for validation, and 12,446×7,654 for testing.
  • 4. RIT-18 Dataset: The dataset contains 18 class labels whose instance counts differ by orders of magnitude, leaving some classes underrepresented.This imbalance is identified as a source of increased segmentation difficulty.

5. FCN Training Procedures

The FCN models are trained either from random initialization or from ResNet-50 weights pretrained on synthetic DIRSIG imagery. Pretrained models use staged adaptation and were observed to reach peak accuracy in fewer epochs.

  • 5. FCN Training Procedures: Synthetic pretraining initializes ResNet-50 with approximately 4.7 million labeled 80×80 DIRSIG patches before MSI segmentation training.Each patch receives the most common pixel label for the image-classification pretraining task.
  • 5. FCN Training Procedures: Class imbalance is handled with class-weighted training, using weights derived from DIRSIG class frequencies and a class-weight parameter of µ = 0.25 for AA optimization.The training setup uses the Nadam optimizer, batch size 32, and weight decay 1e-4.
  • 5. FCN Training Procedures: Randomly initialized FCNs are trained end-to-end on 160 × 160 orthomosaic patches, with learning rates reduced when validation loss plateaus.The initial learning rate is 2e-3 and is reduced by a factor of 10 four times.
  • 5. FCN Training Procedures: Pretrained FCNs are trained in two stages: first freezing the pretrained portion, then jointly fine-tuning the convolutional and segmentation networks.This procedure first adapts the segmentation layers and then updates the full model together.
  • 5. FCN Training Procedures: Pretrained models, especially RefineNet, saturated at peak accuracy in fewer epochs than randomly initialized networks.The authors attribute this observation to most pretrained weights being close to their final solution.

6. Experimental Results

Experiments on RIT-18 compare randomly initialized and synthetic-initialized FCN models with baseline methods, showing benefits from synthetic pretraining while revealing class-imbalance and band-selection effects.

  • RIT-18 Results: Synthetic initialization made both SharpMask and RefineNet outperform all other algorithms in mean-class accuracy.RefineNet-Sim was the best-performing model, while randomly initialized RefineNet overfit classes with the most samples.
  • RIT-18 Results: RefineNet-Sim substantially outperformed RefineNet-Rdm, whereas SharpMask-Rdm was only slightly worse than SharpMask-Sim.The authors attribute the larger RefineNet gap partly to its 69 million trainable parameters, compared with SharpMask’s 11.9 million.
  • Baseline Comparisons: Unsupervised feature extraction improved baseline performance, with MICA increasing mean-class accuracy by 4.8 percent over the simpler MP experiment.The comparison suggests that larger-receptive-field low-level features can outperform smaller-extent higher-level features in this setting.
  • Baseline Comparisons: Unsupervised methods can miss small or uncommon classes because they learn dominant training-image spatial-spectral features without object annotations.Supervised DCNNs can learn more object-specific representations, but they require large annotated datasets to generalize well.
  • Error Analysis: The models misclassified black panels, low-level vegetation, ponds, and bright lake wave crests because of few samples or similar spatial-spectral characteristics.The paper suggests adding DIRSIG examples that replicate difficult conditions, such as vegetation in a body of water.
  • Band Analysis for RIT-18: 4-band systems overfit dominant RIT-18 classes, while the full 6-band Micro-MCA6 configuration improved classification performance over simpler 4-band MSI systems.The band analysis examined the contribution of additional near-infrared information using mean-class accuracy.

7. Discussion

The discussion concludes that FCN architectures and synthetic imagery initialization address MSI segmentation challenges, while RIT-18 provides a benchmark with practical scope boundaries.

  • Discussion: End-to-end FCN segmentation provided superior classification performance on fourteen of eighteen RIT-18 classes.The authors characterize supervised DCNN features as more discriminative than features from unsupervised learning methods.
  • Discussion: Generated synthetic imagery effectively initialized DCNN architectures and offset the absence of large quantities of annotated image data.Random initialization produced degraded mean-class accuracy, which is useful for unbalanced class distributions.
  • Future Applications: DIRSIG could generate large custom datasets for multispectral, hyperspectral, LIDAR, or combined sensor modalities.Moving to hyperspectral data will likely require model modifications for higher dimensionality.
  • Future Applications: DIRSIG initialization could improve networks for semantic segmentation, object detection, and target tracking in hyperspectral imagery.This extends the proposed initialization scheme beyond the segmentation task evaluated in the paper.
  • Dataset Contribution: RIT-18 was introduced as an MSI semantic-segmentation benchmark with high spatial resolution, many object classes, and broad spectral coverage.The dataset is intended to support comparisons and practical remote-sensing applications using UAS-derived imagery.
  • Dataset Contribution: RIT-18 orthomosaic absolute accuracy is limited to 10 feet because of onboard GPS accuracy.This prevents overlaying the dataset with other imagery, although the authors state it does not affect their registered semantic-segmentation results.

8. Conclusion

Synthetic imagery can initialize DCNN semantic-segmentation frameworks for real MSI when annotated data are scarce. The paper also introduces RIT-18 as a challenging MSI benchmark, while noting remaining model and data limitations.

  • Synthetic-data features transferred to real-world imagery and prevented the RefineNet-Sim model from overfitting during training.
  • The network initialization scheme increased semantic-segmentation performance compared with traditional classifiers and unsupervised feature extraction techniques.
  • RIT-18 provides an improved and more challenging benchmark for semantic segmentation of multispectral imagery.The dataset is intended to standardize evaluation through the IEEE GRSS evaluation server and is practical because UAS platforms are easier and cheaper to fly.
  • The explored DCNN models still produce classification maps with some salt-and-pepper label noise.The authors identify newer end-to-end segmentation frameworks, GEOBIA methods, or CRF-based algorithms as possible remedies.
  • Future improvements include deeper or newer network architectures, finer DIRSIG scene resolution, and more diverse synthetic classes.

Appendix A. RIT-18: Dataset Creation Details

RIT-18 orthomosaics are created by registering six spectral bands, normalizing exposure, and processing imagery through photogrammetric reconstruction before manual quality control.

  • The RIT-18 dataset was collected with multispectral and other remote-sensing payloads in a new UAS laboratory, using an orthomosaic generation pipeline.
  • Six independent spectral-band images are co-registered because the manufacturer’s affine transformation caused noticeable error at the collection height.
  • Per-image registration uses SIFT features, k-Nearest Neighbor matching, match filtering, RANSAC homography estimation, and transformation.Per-image processing is critical when wind causes platform motion or the UAV flies over trees.
  • Exposure is normalized across bands because each frame and spectral channel uses a different integration time, with longer exposures potentially causing motion blur.
  • PhotoScan aligns images, reconstructs a dense point cloud and 3D mesh, generates a WGS-84 orthomosaic, and supports manual cleanup.Manual removal addresses defocused images, motion blur, and moving objects that affect the reconstruction.

Appendix B. RIT-18: Class Descriptions

RIT-18 assigns semantic classes to water, beach, vegetation, asphalt, and road markings, with some labels generated automatically and corrected manually.

  • The dataset distinguishes lake water at Lake Ontario from pond water in the inland pond, and includes sand or beach, lifeguard chair, and buoy classes.
  • Vegetation is divided into grass, trees, and low-level vegetation classes.Grass labels include mixtures and use neighboring pixel information; missed NDVI-generated grass spots were manually added.
  • Asphalt covers parking lots, roads, and walkways, while painted asphalt surfaces form the road-marking class.Cement and stone paths remain background, and some shaded parking lines required manual labeling.

Appendix B.4. Underrepresented Classes

Underrepresented and visually similar classes make RIT-18 difficult, especially when objects occupy few pixels or appear only a handful of times.

  • Mean-class accuracy is especially important because massive classes can overshadow underrepresented classes in classification experiments.
  • Small objects such as people and picnic tables are difficult to classify because they occupy a minute image fraction and are surrounded by larger classes.
  • Rare classes include wood targets, an orange UAS landing pad, lifeguard chairs, and buildings, with some appearing only a few times.The orange landing pad has a single instance in every fold.
  • Similarity between the white wooden target and lifeguard chair makes semantic scene information important for classification accuracy.

Appendix C. SCAE Architecture

SCAE is a three-stage unsupervised spatial-spectral feature extractor built from independently trained convolutional autoencoders. Their outputs are concatenated, pooled, reduced with WPCA, standardized, and classified with an MLP.

  • SCAE Architecture: SCAE uses three individual convolutional autoencoders trained independently to extract progressively higher-level spatial-spectral features.Each subsequent autoencoder receives features from the previous autoencoder’s last hidden layer.
  • SCAE Architecture: The first CAE processes random training-data image patches, while subsequent CAEs process the preceding CAE’s hidden-layer features.
  • Feature Aggregation: Outputs from all three CAEs are concatenated, mean-pooled, and reduced to 99% of the original variance using WPCA.
  • Feature Aggregation: The resulting feature response is scaled to zero-mean/unit-variance and passed to an MLP with one hidden layer.
  • CAE Architecture: Each CAE combines convolution and max-pooling operations with symmetric convolution and upsampling for reconstruction, using skip connections to reduce reconstruction error.
  • Feature Aggregation: After whole-image processing, the three N × 32 feature responses are concatenated, mean-pooled with a 5 × 5 filter, reduced with WPCA, and classified by an MLP.

Appendix D. Additional Results

Additional results examine confusion patterns and training behavior for models with and without DIRSIG pre-training. DIRSIG pre-training is associated with better test-set classification and reduced overfitting, while the generated dataset remains class-imbalanced.

  • Confusion Matrices: Confusion-matrix rows are normalized by class, with bright diagonal cells indicating high classification accuracy and bright off-diagonal cells indicating common misclassifications.
  • Confusion Matrices: DIRSIG-pre-trained RefineNet classified the RIT-18 test set better than the randomly initialized RefineNet, which overfit classes with more training samples.
  • Training Curves: SharpMask-Sim and RefineNet-Sim had closely tracking training and validation loss curves, indicating that the models were not overfitting the training data.
  • Dataset Distribution: The generated DIRSIG training dataset is class-imbalanced, motivating use of a weighted loss function when pre-training ResNet-50.
  • Training Curves: Training and validation loss and accuracy plots compare SharpMask and RefineNet models initialized with DIRSIG weights.
Loading 1703.06452v3…