PaperlayerView exact revision on arXiv

Source-linked AI summary

ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data

Foivos I. Diakogiannis, François Waldner, Peter Caccetta, Chen Wu

arXiv:1904.00592v3cs.CV

TL;DR

Semantic segmentation of very high resolution aerial imagery is challenging because urban classes have high within-class variation, low between-class variation, and occlusion or shadow interactions. The paper proposes ResUNet-a with conditioned multitask inference and a Dice-based Tanimoto loss, achieving top-rank performance on ISPRS Potsdam, with the best model reporting a 92.9% average F1 score over all classes.

  • Problem

    Very high resolution urban imagery is difficult to segment because classes vary internally, overlap in spectral appearance, and interact through occlusions and cast shadows.

  • Method

    The paper combines a UNet encoder/decoder, residual and atrous convolutions, pyramid scene parsing pooling, conditioned multitask inference, and the Tanimoto Dice-loss variant.

  • Results

    92.9% average F1 score over all classes was achieved by the best model on the ISPRS 2D Potsdam dataset, which attained top-rank performance against published results.

  • Takeaways & Limitations

    The conditioned multitasking framework combined with Tanimoto loss is reported as a reliable solution for performant semantic segmentation under heavily imbalanced data.

  • Takeaways & Limitations

    Erroneous inference can appear as square blocks because of the PSPPooling layer, which is particularly problematic for regression outputs.

Abstract

from arXiv · show

Scene understanding of high resolution aerial images is of great importance for the task of automated monitoring in various remote sensing applications. Due to the large within-class and small between-class variance in pixel values of objects of interest, this remains a challenging task. In recent years, deep convolutional neural networks have started being used in remote sensing applications and demonstrate state of the art performance for pixel level classification of objects. \textcolor{black}{Here we propose a reliable framework for performant results for the task of semantic segmentation of monotemporal very high resolution aerial images. Our framework consists of a novel deep learning architecture, ResUNet-a, and a novel loss function based on the Dice loss. ResUNet-a uses a UNet encoder/decoder backbone, in combination with residual connections, atrous convolutions, pyramid scene parsing pooling and multi-tasking inference. ResUNet-a infers sequentially the boundary of the objects, the distance transform of the segmentation mask, the segmentation mask and a colored reconstruction of the input. Each of the tasks is conditioned on the inference of the previous ones, thus establishing a conditioned relationship between the various tasks, as this is described through the architecture's computation graph. We analyse the performance of several flavours of the Generalized Dice loss for semantic segmentation, and we introduce a novel variant loss function for semantic segmentation of objects that has excellent convergence properties and behaves well even under the presence of highly imbalanced classes.} The performance of our modeling framework is evaluated on the ISPRS 2D Potsdam dataset. Results show state-of-the-art performance with an average F1 score of 92.9\% over all classes for our best model.

1. Introduction

Semantic segmentation of very high resolution urban imagery is difficult because classes vary internally, overlap spectrally, and interact through occlusions and shadows. The paper addresses these challenges with ResUNet-a, a CNN framework combining object-aware learning, multitasking, a new Dice-based loss, and multiscale augmentation.

  • Challenges: Urban objects show high within-class variability and low between-class variability, making semantically different objects difficult to distinguish from imagery.Differences in materials and structures create variation in color, size, shape, and texture, while objects such as rooftops, sidewalks, and roads can share spectral signatures.
  • Challenges: Pixel-based classifications can produce salt-and-pepper artifacts and fail to exploit the rich information content of very high resolution data.Traditional object-based alternatives also depend on segmentation parameters and preselected features that may be suboptimal.
  • Approach: CNN-based methods provide end-to-end, object-based classification with minimal feature engineering and features that characterize whole or partial objects.The proposed approach assigns class-probability vectors to pixels using features derived from objects such as cars, trees, and building corners.
  • Approach: ResUNet-a combines a UNet encoder/decoder with residual connections, atrous convolutions, pyramid scene parsing pooling, and multitask inference.The paper presents single-task and multitask variants and analyzes multiple Dice-loss formulations.
  • Approach: The proposed Dice-loss variant is designed to speed convergence, improve performance, and remain effective under large class imbalance.The framework also uses multiscale input augmentation during training to improve performance and avoid overfitting.

2. Related Work

Related work established encoder-decoder, atrous-convolution, multicontext, boundary-aware, and data-fusion strategies for semantic segmentation. ResUNet-a builds on these directions while retaining a UNet-style multiscale framework and conditioning segmentation on auxiliary predictions.

  • Computer vision foundations: Fully Convolutional Networks replaced final fully connected layers and substantially improved semantic segmentation performance in computer vision.Later Deeplab models highlighted atrous convolutions and used conditional random fields to refine segmentation.
  • Encoder-decoder models: U-Net introduced an encoder-decoder paradigm that progressively upsamples lower-resolution features toward the original image resolution.CNN multicontext feature extraction therefore requires upsampling that preserves information across spatial resolutions and supports fine boundaries.
  • Remote-sensing segmentation: Remote-sensing approaches explored atrous convolutions, pretrained encoders, asymmetric hourglass networks, inception-like modules, and multicontext aggregation.These methods differed in downsampling, decoder symmetry, feature contexts, and loss choices.
  • Boundary-aware methods: Boundary-aware segmentation methods predicted object boundaries before concatenating them with image features for a subsequent pixel-classification network.ResUNet-a differs by placing boundary prediction at the end of its architecture so boundary prediction affects all features.
  • Data fusion: Dense Pyramid Network used group convolutions to process digital surface models and orthophotos separately, then combined their information with channel shuffling and pyramid pooling.Its feature extractor was based on DenseNet.

3. The ResUNet-a framework

The ResUNet-a framework combines a UNet encoder-decoder with residual blocks, atrous convolutions, pyramid pooling, and multitask outputs. It defines d6 and d7 variants and supports either independent or conditioned multitask inference.

  • Architecture: The framework combines a UNet encoder-decoder backbone, modified residual blocks, atrous convolutions, and pyramid scene parsing pooling.The architecture is designed to preserve information across spatial contexts while extracting features at multiple receptive-field scales.
  • Multitasking: ResUNet-a learns four complementary outputs: segmentation mask, class boundaries, distance transform, and HSV image reconstruction.The distance transform represents each pixel's distance to the closest boundary on a [0,1] gray-level scale.
  • Model variants: The d6 and d7 architectures differ in depth, with six versus seven encoder ResBlock-a units; d6 includes a PSPPooling layer after its encoder blocks.The d7 model adds one residual building block in both encoder and decoder relative to d6.
  • Architecture: The d6 implementation begins with a 1 × 1 convolution and uses up to three parallel atrous convolutions within each residual block.The initial 1 × 1 convolution increases feature count while avoiding cross-pixel summarization with a larger kernel.
  • Model variants: ResUNet-a d6 has approximately 52M trainable parameters, whereas d7 has approximately 160M for the same initial filter size of 32.The parameter counts remain almost identical for the corresponding multitask models.
  • Multitasking: The conditioned multitask variant predicts the distance map, then boundaries, and finally segmentation using progressively concatenated intermediate outputs.The simple variant produces the four tasks independently without directly feeding complementary-task predictions into segmentation.

3.2. Loss function

The section compares Dice-family loss formulations for segmentation, focusing on how their gradient fields affect optimization. It motivates a complement-based Tanimoto loss whose gradients point toward the ground truth from arbitrary initializations.

  • Loss-function background: Dice-family formulations can be numerically equivalent while producing different gradients for gradient-based optimization.The compared variants map inputs to [0, 1], but their gradient behavior differs in deep-learning applications.
  • Optimization behavior: The authors report that squared-denominator losses point toward the ground truth more reliably than alternatives across random weight initializations.They also report faster convergence when the loss is complemented with a dual form measuring complement overlap.
  • Complement formulation: The proposed complement formulation averages T(pi, li) with T(1 − pi, 1 − li) to incorporate overlap information from complementary regions.The complement inner product captures probability components corresponding to zero-valued ground-truth entries.
  • Optimization behavior: For D1, gradients from random points do not necessarily direct optimization toward the ground truth point (1, 0).This behavior is illustrated through gradient fields for multiple Dice-family variants.
  • Optimization behavior: The complement Tanimoto loss has straight gradient-flow lines pointing to the ground truth from any random initialization, with a second-order-derivative extremum at the ground truth.The section presents this as evidence for its superiority among Dice-coefficient-based losses for deep-learning model training.
  • Continuous variables: The Tanimoto loss can also recover continuous variables in [0, 1] when network outputs are restricted to that range.The example uses a ground-truth vector l = (0.25, 0.85) and compares the Tanimoto function with its complement.

3.3. Data augmentation

The training pipeline uses geometric augmentation so the model sees transformed versions of image–mask pairs rather than the exact same images in each iteration. Random rotations, centers, and zoom factors are completed with reflect padding.

  • Augmentation strategy: Geometric augmentation ensures that each training iteration uses a different transformed batch of image and ground-truth mask pairs.The stated purpose is to avoid overfitting by preventing repeated exposure to exactly the same image set.
  • Augmentation strategy: Each image–mask pair is rotated around a random center and zoomed in or out using a random scale factor.The transformations vary the perspective and scale of the same object scenery.
  • Boundary handling: Reflect padding fills image regions left outside the frame after geometric transformations.

3.4. Inference methodology

Inference over large orthophotos uses overlapping padded tiles to reduce uncertainty near tile edges. The method steps through 256×256 windows with a 64-pixel stride and 128-pixel reflect padding.

  • Tile-based inference: A 256×256 inference window provides less confident classifications near its edges because contextual information is missing outside the tile.
  • Tile-based inference: The method steps across the image with a stride of 256/4 = 64 pixels to obtain multiple inference probabilities for each pixel.
  • Boundary handling: Each inference tile receives reflect padding of 256/2 = 128 pixels to compensate for missing information beyond tile boundaries.

4. Data and preprocessing

The study uses the ISPRS Potsdam dataset, combining VISNIR imagery with normalized DSM data for six-class semantic segmentation. Two 256×256-patch datasets vary spatial resolution and field of view to examine context and detail.

  • Dataset: The Potsdam data combine four spectral bands with a normalized digital surface model for training.The spectral bands are red, green, blue, and near infrared, with 5 cm ground sampling distance.
  • Dataset: The segmentation labels cover impervious surfaces, buildings, cars, low vegetation, trees, and background.
  • Preprocessing: The FoV×4 dataset uses half-resolution 256×256 patches, providing four times the field-of-view area of full-resolution patches.This design increases contextual coverage within the same patch size.
  • Preprocessing: FoV×4 contains approximately 10k image-mask pairs and ∼50GB of data, whereas FoV×1 contains ∼40k pairs and ∼250GB.FoV×4 supports faster experimentation, while FoV×1 retains finer spatial detail.

5. Architecture and Tanimoto loss experimental analysis

The experiments evaluate convergence, architecture modules, multitask conditioning, and loss functions under controlled training settings. Results favor conditioned multitasking, middle PSPPooling, and the Tanimoto with complement loss, while exposing instability from final-layer pooling.

  • Architecture ablation: The ablation study adds atrous convolutions, PSPPooling, and conditioned multitasking to a baseline while controlling training hyperparameters.The first experiment compares model complexity and convergence rather than fully fine-tuned final performance.
  • Architecture ablation: ResUNet-a-cmtsk reaches the baseline ResUNet’s performance level at approximately epoch 40 instead of approximately 120 epochs.Atrous convolutions nearly double convergence rate; PSPPooling adds capacity but instability, which conditioned multitasking fixes.
  • PSPPooling: A middle PSPPooling layer makes d7v2 converge much faster than d7v1 despite d7v2 having greater parameter complexity.The comparison uses identical fixed hyperparameters without learning-rate reduction.
  • PSPPooling: Final-layer PSPPooling can produce square-block inference errors and is especially problematic for regression outputs.The pooling area can dominate predictions in square subregions, motivating its exclusion from color and distance-transform evaluations.
  • Conditioned multitasking: Conditioned multitasking shows lower training variance than standard multitasking and higher median performance despite random MCC fluctuations.Identity reconstruction in HSV further reduces performance-metric variance, stabilizing gradient updates.
  • Conditioned multitasking: The conditioned multitask model provides faster, more consistent convergence and better boundary segmentation than single-output models.
  • Loss analysis: The Tanimoto with complement loss converges faster to an optimal value than Dice loss under identical hyperparameters and training epochs.Across the experiments, it gives the best performance; the related complement variant converges more slowly and has greater variance.

6. Results and discussion

ResUNet-a performs competitively on Potsdam, with conditioned multitasking and higher-resolution training improving results for classes requiring fine detail. The d7v2 conditioned model ranks first among the compared frameworks across classes.

  • Evaluation setup: The evaluation compares ResUNet-a models trained on both FoV×4 and FoV×1 versions of the ISPRS Potsdam dataset.The reported evaluation uses publicly available test labels and compares performance with published models.
  • Model results: ResUNet-a d6 cmtsk significantly improves over single-task d6 for Building, LowVeg, and Tree classes.The gain is especially relevant where the single-task model under-segments tree boundaries after image reduction.
  • Model results: The d7v1 cmtsk model shows slightly higher performance across tested models, although FoV×4 differences are marginal and class-dependent.Dataset annotation errors are reported as an eventual upper bound on performance.
  • Resolution and field of view: Higher-resolution FoV×1 training improves classes requiring finer details, while Building scores better with the wider FoV×4 d6 cmtsk dataset.
  • Comparison with other frameworks: ResUNet-a d6 ranks third by average F1 among the comparison models, while d7v2 cmtsk ranks first in every class.The paper notes that many competing performance differences are marginal.
  • Comparison with other frameworks: The comparison includes models whose competitors may use external pretraining and transfer learning, affecting the evaluation context.ResUNet-a results were generated using eroded-boundary ground-truth masks as recommended for Potsdam evaluation.

7. Conclusions

The paper presents ResUNet-a, a multitasking architecture paired with the Tanimoto loss for semantic segmentation of high-resolution aerial images. Evaluated on the ISPRS Potsdam dataset, the framework achieves top-rank performance, while the Tanimoto loss supports convergence and imbalanced-data training.

  • Conclusions: ResUNet-a combines a multitasking encoder/decoder architecture with the Tanimoto loss for semantic segmentation of high-resolution aerial images.The framework is designed for the ISPRS 2D semantic segmentation Potsdam dataset.
  • Conclusions: The architecture conditions final segmentation on predicted object boundaries and distance transforms, while also inferring input-image reconstruction.Its encoder/decoder design uses residual units, parallel atrous convolutions, and pyramid scene parsing pooling.
  • Conclusions: The Tanimoto loss speeds training convergence, supports regression tasks, and behaves well with heavily imbalanced datasets.Its balanced gradients make it suitable for multitasking training.
  • Conclusions: ResUNet-a d7v2 achieves top-rank performance and a clear improvement over published state-of-the-art results on the ISPRS Potsdam dataset.The authors characterize the combination of conditioned multitasking and the proposed loss as a reliable solution for performant semantic segmentation.

Appendix A. Software implementation and training characteristics

The implementation uses MXNet GLUON, Adam optimization, multi-GPU training, and manual gradient aggregation. Training uses an initial learning rate of 0.001 with scheduled reductions, while smaller batches can become unstable.

  • Software implementation and training characteristics: ResUNet-a was implemented with the MXNet deep learning library under the GLUON API.Models trained on FoV×4 used four NVIDIA Tesla P100 GPUs on a single node.
  • Software implementation and training characteristics: Manual gradient aggregation enabled a batch size of 256 when each GPU iteration could process only approximately 10 samples.This addressed the network’s computational complexity during FoV×4 training.
  • Software implementation and training characteristics: Training used Adam with an initial learning rate of 0.001 and momentum parameters (β1, β2) = (0.9, 0.999).The learning rate was reduced by an order of magnitude whenever validation loss stopped decreasing.
  • Software implementation and training characteristics: A batch size of 32 produced unstable training, whereas the learning rate was reduced three times during the overall schedule.
  • Software implementation and training characteristics: A software implementation of the ResUNet-a models is available on GitHub.

Appendix B. Boundary and distance transform from segmentation mask

The appendix provides routines for deriving object boundaries and distance transforms directly from segmentation masks. These targets support the auxiliary predictions used by the multitasking architecture.

  • Boundary and distance transform from segmentation mask: Boundary and distance-transform targets can be computed efficiently from segmentation ground-truth masks using Python computer-vision routines.Multiclass masks are supplied in one-hot encoding and processed iteratively per channel.
  • Boundary and distance transform from segmentation mask: Binary labels use 1 for on-class pixels and 0 for off-class pixels in a two-dimensional single-channel image.
  • Boundary and distance transform from segmentation mask: The boundary routine converts the label mask to uint8, applies Canny edge detection, dilates the result, and normalizes it to floating-point values.
  • Boundary and distance transform from segmentation mask: The distance routine converts the mask to uint8, applies a distance transform, and normalizes the output with NORM_MINMAX.

Appendix C. Inference results

The appendix presents classification results and error maps for all test TOP tiles in the ISPRS Potsdam dataset. The displayed figures compare input imagery, prediction errors, and inference maps across multiple tile groups.

  • Inference results: The inference-results section reports classification results and error maps for all test TOP tiles in the Potsdam ISPRS dataset.
  • Inference results: Figures C.17–C.19 show input images, differences between ground truth and predictions, and inference maps for tiles 4-13 through 7-13.The figures cover tile groups 4-13, 4-14, 4-15, 5-13, 5-14, 5-15, 6-13, 6-14, 6-15, and 7-13.
Loading 1904.00592v3…