PaperlayerView exact revision on arXiv

Source-linked AI summary

Land Cover Classification from Remote Sensing Images Based on Multi-Scale Fully Convolutional Network

Rui Li, Shunyi Zheng, Chenxi Duan, Ce Zhang

arXiv:2008.00168v2cs.CVcs.AIeess.IV

TL;DR

Land-cover classification supports resource management, but field surveys are labor-intensive and 2D convolution can collapse temporal information. The paper proposes a multi-scale FCN, extended to 3D for spatio-temporal imagery, and reports gains over FGC on spatial datasets alongside state-of-the-art mIoU on two spatio-temporal datasets.

  • Problem

    Field surveys are time-consuming and labor-intensive, geographical distribution information is often missing, and 2D CNN feature extraction can collapse multi-temporal information.

    Source: §I · pp. 1–2

  • Method

    The paper proposes an encoder-decoder MSFCN with multi-scale convolutional blocks, channel attention, and global pooling, and extends it to 3D spatio-temporal imagery.

    Source: §I · p. 2, §IV · p. 9

  • Results

    MSFCN brings near 3% improvements in both mIoU and F1-score versus FGC on WHDLD; on GID, gains exceed 3% in mIoU and 2% in F1-score, and it obtains state-of-the-art mIoU on two spatio-temporal datasets.

    Source: C. Results on WHDLD and GID · p. 7, D. Results on 2015 and 2017 datasets · p. 8

  • Takeaways & Limitations

    Experiments on two spatio-temporal datasets demonstrate that 3D CNN is a valid method for exploiting information from spatio-temporal images.

    Source: §IV · p. 9

Abstract

from arXiv · show

In this paper, a Multi-Scale Fully Convolutional Network (MSFCN) with multi-scale convolutional kernel is proposed to exploit discriminative representations from two-dimensional (2D) satellite images.

I. INTRODUCTION

The introduction motivates automated land-cover classification and proposes a multi-scale FCN that captures local and global features, with a 3D extension for spatio-temporal imagery. It also situates this approach among prior methods and describes its evaluation across spatial and spatio-temporal datasets.

I. INTRODUCTION

  • Field surveys produce high-quality land-cover information but are time-consuming and labor-intensive, and geographical distribution information is often missing.Land-cover classification is framed as foundational to land-resource management, cultivated-area evaluation, and economic assessment.Source: §I · p. 1
  • Manual descriptors restrict the flexibility and adaptability of pixel-based classifiers, while deep learning can automatically capture nonlinear and hierarchical features.The introduction says deep learning can use periods, spectra, and interactions between land-cover types, unlike vegetation indices that consider finite bands.Source: §I · pp. 1–2
  • Prior work adapted temporal CNNs, hierarchical segmentation, and encoder-decoder networks to land-cover classification and satellite imagery.Examples include temporal CNNs and fine-grained U-Net for sparse Sentinel-2 annotations.Source: §I · p. 2
  • MSFCN uses encoder-decoder FCN branches with different kernel sizes to capture local and global features, plus channel attention and global pooling.The paper extends the network to 3D for spatio-temporal satellite images.Source: §I · p. 2
  • 2D CNN processing can collapse temporal information, whereas 3D CNNs preserve temporal dimensions but require substantial computation for pixel-by-pixel segmentation.The introduction cites a 3D encoder-decoder FCN with global pooling and attention as a method able to capture features from the whole input while improving accuracy and efficiency.Source: §I · p. 2
  • The paper compares 2D MSFCN with five segmentation models and 3D MSFCN with five temporal or 3D models, and reports quantitative experiments on two spatial and two spatio-temporal datasets.It also compares a 3D Attention U-Net extension with MSFCN.Source: §I · p. 2

II. METHODOLOGY

The feature-extraction discussion describes 3D convolution over spatio-temporal feature maps and explains why kernel size matters. It also presents batch normalization as a step applied to generated 3D feature maps.

A. Feature Extraction using 3D CNN

  • 3D convolution slides a kernel across time, height, and width to produce output 3D feature maps.The input is described by its time, height, width, and channel dimensions, while the kernel has temporal and spatial dimensions.Source: A. Feature Extraction using 3D CNN · p. 3
  • Generated 3D feature maps are normalized by batch normalization, then scaled and shifted by trainable parameters before activation.The activation is ReLU in this model.Source: A. Feature Extraction using 3D CNN · p. 3
  • Kernel size is critical because it determines the receptive field, and extracted-feature quality limits model performance.Source: A. Feature Extraction using 3D CNN · p. 3

B. Multi-Scale Convolutional Block

The multi-scale convolutional block addresses the fixed receptive field of conventional FCN blocks by combining local and global feature extraction. Its parallel branches are merged and passed through a final convolution and batch normalization.

B. Multi-Scale Convolutional Block

  • MSCB combines local and global visual patterns to expand the receptive field beyond the constant scale of conventional FCN blocks.The conventional block uses two stacked 3D convolutions with the same kernel size.Source: B. Multi-Scale Convolutional Block · p. 3
  • Two stacked (3 × 3 × 3) convolutions form the global branch, while a single (3×3×3) convolution captures local patterns.The stacked branch has a receptive field equivalent to a (5 × 5 × 5) convolution.Source: B. Multi-Scale Convolutional Block · p. 4
  • The branch outputs are added, then processed by a (1 × 1 × 1) convolution with batch normalization.Source: B. Multi-Scale Convolutional Block · p. 4

C. Channel Attention Block and Global Pooling Module

The channel attention block (CAB) reweights concatenated encoder and decoder features, while the global pooling module (GPM) brings global spatio-temporal context from the encoder’s highest level into the decoder. Both modules reweight feature maps, but CAB also fuses encoder-decoder features.

C. Channel Attention Block and Global Pooling Module

  • CAB addresses the equal channel weights implied by the score-map calculation because features from different stages have different discrimination and prediction consistency.Source: C. Channel Attention Block and Global… · p. 4
  • CAB takes concatenated encoder and decoder feature maps and adaptively realigns channel-wise dependencies using a generated weight vector.A final convolution generates globally consistent spatio-temporal feature maps.Source: C. Channel Attention Block and Global… · p. 4
  • GPM uses global spatio-temporal consistency to transform the encoder’s highest-level feature maps into corresponding decoder features.The module reweights feature maps and can be viewed as an attention mechanism.Source: C. Channel Attention Block and Global… · p. 4

D. Network Architecture

MSFCN combines multi-scale convolutional blocks with GPM and CAB in an encoder-decoder architecture, preserving temporal information during spatial pooling and producing 2D segmentation maps. Cross-entropy measures the disparity between those maps and ground truth for evaluation and backpropagation.

D. Network Architecture

  • The encoder uses four multi-scale blocks with 32, 64, 128, and 256 output channels, followed by spatial max-pooling that preserves temporal information.GPM enhances global spatio-temporal consistency at the highest encoder layer, and CAB fuses encoder and decoder features.Source: D. Network Architecture · p. 5, Figure 6 · p. 5
  • Decoder transposed convolutions restore feature maps to input size, with intervening (3 × 3 × 3) convolutions used to avoid checkerboard patterns.Source: D. Network Architecture · p. 5
  • A (t × 3 × 3) convolution and a (1 × 1 × 1) convolution coalesce the final 3D features into 2D segmentation maps, which cross-entropy compares with ground truth.The loss is used for evaluation and backpropagation.Source: D. Network Architecture · p. 5

III. EXPERIMENTAL RESULTS

The experiments are set up to test 2D MSFCN on WHDLD and GID and 3D MSFCN on two GF2 spatio-temporal datasets. The datasets span labeled urban and broader land-cover imagery, with the GID experiments using a selected subset of images.

III. EXPERIMENTAL RESULTS

  • The experimental section introduces datasets and settings for testing MSFCN, then compares the performance of different frameworks.Source: §III · p. 5

A. Datasets

  • 2D MSFCN is evaluated on WHDLD and GID, while 3D MSFCN is evaluated on two GF2 spatio-temporal satellite datasets.Source: A. Datasets · p. 5
  • WHDLD contains 4,940 256 × 256 RGB images at 2 m/pixel over Wuhan, labeled as bare soil, building, pavement, vegetation, road, or water.Source: A. Datasets · p. 5
  • The GID comprises 150 large RGB images across 60 Chinese cities, but the experiments select 15 images to cover its six land-cover classes.Source: A. Datasets · p. 5
  • The two GF2 datasets use four bands at 4 m resolution: four monthly images from June–September 2015 and seven from June–December 2017.Both datasets were atmospherically corrected and geometrically rectified.Source: A. Datasets · p. 5

B. Experimental Setting

The experiments compare 2D MSFCN with established segmentation models on WHDLD and GID, and 3D MSFCN with 1D, 2D, and 3D alternatives on two GF2 time-series datasets. The evaluation uses held-out data and several segmentation metrics, with mIoU designated as primary.

B. Experimental Setting

  • 2D MSFCN is compared with SegNet, FC-DenseNet, U-Net, Attention U-Net, and FGC; 3D MSFCN is compared with 1D U-Net, 2D U-Net, 3D U-Net, Conv-LSTM, 3D FGC, and 3D Attention U-Net.The 2D comparisons use WHDLD and GID; the 3D comparisons use the two GF2 spatio-temporal datasets.Source: B. Experimental Setting · p. 5
  • WHDLD images are split 60%/20%/20% for training, validation, and test; GID is divided into 256 × 256 patches, with 60%/20%/20% assigned likewise.Training data for both datasets are augmented, and training stops when validation accuracy no longer increases for 10 epochs.Source: B. Experimental Setting · pp. 5–6
  • For the 2015 and 2017 GF2 datasets, red-rectangle regions containing all classes were selected for training, while the remainder was used for prediction; selected training portions were zeroed to avoid data leakage.Images were zero-padded to 2816 × 1536 (2015) and 2304 × 1280 (2017), then segmented into non-overlapping 256 × 256 patches.Source: B. Experimental Setting · p. 6, Figure 9 · p. 6
  • For the GF2 datasets, images are zero-padded and divided into non-overlapping 256 × 256 patches; each model trains for 100 epochs before test-set evaluation.The 2015 images are enlarged to 2816 × 1536 and the 2017 images to 2304 × 1280.Source: B. Experimental Setting · p. 6
  • The evaluation reports OA, AA, Kappa, mIoU, FWIoU, and F1-score, with mIoU selected as the primary indicator.The WHDLD and GID figures show example images with ground truth; the GF2 figure shows the two datasets and their training regions.Source: B. Experimental Setting · p. 6, Figure 7 · p. 6, Figure 8 · p. 6

C. Results on WHDLD and GID

On WHDLD and GID, MSFCN outperforms the compared algorithms on the reported evaluation indexes, including mIoU and F1-score. The section also reports per-class performance, confusion matrices, visual results, and model complexity.

C. Results on WHDLD and GID

  • Nearly 3% gains on WHDLD mIoU and F1-score, and gains exceeding 3% on GID mIoU and 2% on GID F1-score, are reported for MSFCN versus FGC.Table III presents the experimental results for WHDLD and GID.Source: C. Results on WHDLD and GID · p. 7
  • MSFCN achieves the best per-class F1-score in most WHDLD classes and all GID classes.Table IV reports the per-class F1-score results.Source: C. Results on WHDLD and GID · p. 7
  • MSFCN's confusion matrix has a more distinct diagonal structure than the other methods, which the paper interprets as stronger class discrimination.The heat maps use dark blue diagonal blocks to indicate class-wise discrimination.Source: C. Results on WHDLD and GID · p. 7
  • The authors report that MSFCN does not add redundant parameters or high computational complexity compared with other algorithms.Table V compares parameter counts and computational complexity.Source: C. Results on WHDLD and GID · p. 7

D. Results on 2015 and 2017 datasets

On the two GF2 spatio-temporal datasets, 3D MSFCN achieves state-of-the-art mIoU, while the reported comparisons favor 3D-based models over 2D-based models. The paper also compares per-class scores and computational complexity.

D. Results on 2015 and 2017 datasets

  • Models based on 3D CNN dramatically outperform models based on 2D CNN, while Conv-LSTM also exceeds the 2D-based models.The paper attributes the 2D limitation to loss of the temporal dimension during spatio-temporal feature extraction; the 1D U-Net performs worst.Source: D. Results on 2015 and 2017 datasets · pp. 7–8
  • MSFCN obtains state-of-the-art mIoU on two datasets, with the reported comparison covering the 2015 and 2017 datasets and its per-class F1-score performance.It obtains the best performance in all classes on the 2015 dataset and most classes on the 2017 dataset.Source: D. Results on 2015 and 2017 datasets · p. 8
  • In visual comparisons across the two datasets, MSFCN differentiates corn (green) and grass (yellow) better than other models in the fourth-column example.The first three segmentation-map rows are from the 2015 dataset, and the remainder are from the 2017 dataset.Source: D. Results on 2015 and 2017 datasets · p. 8
  • The authors describe MSFCN's computational complexity as not unacceptable.Table VIII compares parameter counts and computational complexity.Source: D. Results on 2015 and 2017 datasets · p. 8

E. Effectiveness of the Multi-Scale Convolutional Block and Attention Mechanisms

Ablations report improved mIoU when the multi-scale block, channel attention, and global pooling are introduced. The depth and channel investigations identify performance–complexity trade-offs for the selected MSFCN design.

E. Effectiveness of the Multi-Scale Convolutional Block and Attention Mechanisms

  • Introducing all blocks raises mIoU to 0.60366 on WHDLD and 0.75127 on GID, above the 3D U-Net baselines of 0.55706 and 0.69417.The multi-scale blocks alone yield 0.57098 and 0.71992; adding channel attention and global pooling brings further reported mIoU improvements.Source: E. Effectiveness of the Multi-Scale… · p. 8

F. Investigation about the Number of Layers and Channels

  • The four-layer MSFCN exceeds the three-layer variant by more than 5% on mIoU without enormous increases in parameters or computational complexity, while the five-layer variant uses four times the parameters.The authors describe the five-layer choice as inefficient despite its performance improvements.Source: F. Investigation about the Number of… · p. 8
  • The proposed channel configuration [32, 64, 128, 256] exceeds the narrow variant by nearly 5% on mIoU, while the wide variant gains nearly 1% at five times the parameters and computational complexity.The authors conclude that the proposed MSFCN balances performance and complexity.Source: F. Investigation about the Number of… · pp. 8–9

IV. CONCLUSION

Experiments support MSFCN on two spatial datasets and the 3D CNN on two spatiotemporal datasets. The paper also reports that its findings are included in the article.

IV. CONCLUSION

  • MSFCN combines multi-scale spatial feature extraction with channel and global-context consistency, and extends to 3D CNN for temporal interactions among land-cover classes.The multi-scale blocks capture local and global information; channel attention and global pooling enhance channel and global contextual consistency.Source: §IV · p. 9
  • Experiments on two spatial datasets support MSFCN’s effectiveness, while two spatiotemporal datasets demonstrate that 3D CNN is a valid method for exploiting spatiotemporal images.The authors also explore layer and channel counts as references for designing FCN-based land-cover classification networks.Source: §IV · p. 9

V. DATA AVAILABILITY STATEMENT

  • The data supporting the study’s findings are included in the article.Source: §V · p. 9

Related papers

Loading 2008.00168v2…