PaperlayerView exact revision on arXiv

Source-linked AI summary

Classification With an Edge: Improving Semantic Image Segmentation with Boundary Detection

Dimitrios Marmanis, Konrad Schindler, Jan Dirk Wegner, Silvano Galliani, Mihai Datcu, Uwe Stilla

arXiv:1612.01337v2cs.CV

TL;DR

DCNNs provide broad context for semantic segmentation but lose spatial detail, blurring boundaries in high-resolution remote-sensing imagery. The paper integrates semantically informed boundary detection with segmentation in end-to-end networks, including a lightweight SegNet variant and high-end ensembles. Boundary detection significantly improves segmentation, while the high-end ensemble exceeds 90% overall accuracy on ISPRS Vaihingen.

  • Problem

    DCNN downsampling provides broad context but loses high-frequency detail, making accurate boundaries difficult in remote-sensing scenes containing many small objects.

  • Method

    The paper adds pixel-wise class-boundary likelihoods to end-to-end semantic segmentation networks, including a memory-efficient SegNet model and ensemble configurations.

  • Results

    Boundary detection significantly improves segmentation, and the high-end ensemble achieves >90% overall accuracy on the ISPRS Vaihingen benchmark.

  • Takeaways & Limitations

    Explicit class-boundary modeling is beneficial across the evaluated DCNN architectures, with the strongest gains reported for man-made classes with well-defined boundaries.

  • Takeaways & Limitations

    Further segmentation improvements may be small, tedious, and increasingly problem-specific, while the best-performing DCNN models are excessively large and complex.

Abstract

from arXiv · show

We present an end-to-end trainable deep convolutional neural network (DCNN) for semantic segmentation with built-in awareness of semantically meaningful boundaries. Semantic segmentation is a fundamental remote sensing task, and most state-of-the-art methods rely on DCNNs as their workhorse. A major reason for their success is that deep networks learn to accumulate contextual information over very large windows (receptive fields). However, this success comes at a cost, since the associated loss of effecive spatial resolution washes out high-frequency details and leads to blurry object boundaries. Here, we propose to counter this effect by combining semantic segmentation with semantically informed edge detection, thus making class-boundaries explicit in the model, First, we construct a comparatively simple, memory-efficient model by adding boundary detection to the Segnet encoder-decoder architecture. Second, we also include boundary detection in FCN-type models and set up a high-end classifier ensemble. We show that boundary detection significantly improves semantic segmentation with CNNs. Our high-end ensemble achieves > 90% overall accuracy on the ISPRS Vaihingen benchmark.

1. Introduction

DCNNs capture broad context for semantic segmentation but lose spatial detail, producing blurred boundaries in high-resolution remote-sensing scenes. The paper makes semantic class-boundaries explicit through joint boundary detection and segmentation, reporting improved accuracy and strong benchmark performance.

  • Motivation: DCNN pooling enlarges contextual coverage but creates a trade-off with accurate localization by losing high-frequency spatial detail.This trade-off is especially consequential for remote sensing, where scenes contain tens to hundreds of small objects.
  • Motivation: High-resolution remote-sensing images contain many small segments, including individual cars and trees, making precise boundaries important.The target imagery has ground sampling distances of 5–30 cm and exposes intricate spatial details.
  • Contribution: 6%: Explicitly modeling class boundaries improves labeling accuracy across the evaluated datasets.The paper reports that boundary-aware modeling significantly improves segmentation with CNNs.
  • Contribution: Boundary detection improves man-made object classes with well-defined boundaries but not vegetation classes with intrinsically fuzzy boundaries at the target resolution.The reported class-dependent pattern limits the improvement to settings where boundaries are sufficiently defined.
  • Results: The boundary-aware ensemble reaches state-of-the-art accuracy on the ISPRS semantic labeling benchmark.The paper also reports F1-scores above 80% for all object classes, with frequent well-defined man-made classes well above 90%.
  • Results: Ensembling reduces the bias of individual models when combining different initializations or architectures.The reported benefit applies both to repeated copies of one architecture and to heterogeneous model ensembles.

2. Related Work

Related work addresses semantic segmentation, boundary refinement, and neural edge detection through encoder-decoder models, skip connections, dilated convolutions, CRFs, and multi-task designs. The paper situates its approach within remote-sensing adaptations while noting limits of object-centric boundary methods for “stuff” scenes.

  • Semantic segmentation: Earlier CNN segmentation methods predicted pixel labels from local neighborhoods, while later FCN and encoder-decoder designs restored output resolution through upsampling.Comparisons reported FCNs as advantageous over per-pixel CNNs, though some implementations remained below state of the art.
  • Edge detection: HED formulates edge detection as a fully convolutional task, and its variants use multi-scale predictions with deep supervision.The paper uses this edge-detection line of work as the basis for integrating boundary information into segmentation.
  • Boundary refinement: Dilated convolutions and skip connections were proposed to mitigate boundary blurring by preserving or recovering high-frequency detail.These strategies target spatial-resolution losses introduced by downsampling and upsampling.
  • Edge detection: Semantically informed edge detection emphasizes edges lying on object contours, using either dedicated classifiers or jointly learned segmentation and edge-prediction networks.Related approaches include feature classification on candidate edge pixels and simultaneous edge detection with semantic segmentation.
  • Remote sensing: Object-centric class-boundary methods are efficient for localized compact objects but less applicable to remote-sensing scenes dominated by objects without well-defined bounding boxes.The distinction is between “things” with regions of interest and “stuff” that lacks clear object boxes.
  • Remote sensing: Remote-sensing FCN studies support two-stream processing for spectral and elevation information, pretrained networks, model averaging, and dilated convolutions.One cited result reports that dilated convolutions slightly mitigate pooling information loss at 40× computation time.

3. The Model

The model explores both a lightweight boundary-aware encoder-decoder and higher-performing multi-scale or ensemble configurations. Its core SegNet variant processes color and elevation streams separately, while boundary-aware SegNet matches more expensive competitors.

  • Design goals: The basic model is designed as an end-to-end, comparatively simple, memory-efficient architecture, while the study also explores maximum-performance configurations.The paper explicitly frames the design space as a trade-off between usability and task-specific performance.
  • Model variants: >90% overall accuracy: The high-end multi-scale ensemble achieves this result on the ISPRS Vaihingen benchmark.The paper notes that diminishing returns require a more complex model for this performance level.
  • SegNet foundation: SegNet combines fully convolutional processing with an encoder-decoder structure and tracks pooling locations for decoding.Its bottleneck representation has reduced spatial resolution before the decoding stage.
  • SegNet foundation: SegNet is lighter and easier to train because it omits fully connected layers, which consume more than 90% of parameters in a typical image-processing CNN.The paper nevertheless reports that standalone SegNet can underperform heavier architectures.
  • Boundary-aware SegNet: Learned class boundaries allow SegNet to match more expensive competitors despite its lighter architecture.This result is reported for the boundary-aware combination rather than SegNet alone.
  • Seg·h encoder-decoder network: Seg·h uses parallel color and elevation branches, concatenates their features, and converts the combined responses into class probabilities.The color branch uses pretrained SegNet weights, while the DSM branch is initialized with Xavier initialization.
  • Multi-scale processing: A three-scale configuration uses separate copies of Seg·h, upsamples their predictions, and fuses them before final class prediction.The scale-specific networks have individual per-scale weights.

hed·h boundary-detection network

The hed·h boundary detector predicts multi-scale class-boundary likelihoods from separate color and elevation streams. It uses deep supervision and a continuous boundary-score regression objective before supplying boundary likelihoods to semantic segmentation.

  • Architecture: HED is a multi-scale encoder-decoder CNN that outputs an image of edge likelihoods.Its side outputs are read before pooling layers and supervised at their respective scales.
  • Architecture: Each pre-pooling convolution produces a scale-specific boundary prediction, and the predictions are fused into a final multi-scale boundary map.Figure 2 depicts the scale-specific outputs and their fusion; feature-map concatenations combine streams.
  • Training objective: Deep supervision trains the multi-scale outputs with additional Euclidean-distance losses.This supervision is applied to the rectified output before each pooling layer.
  • Training objective: hed·h predicts continuous boundary likelihoods with regression rather than hard boundary classification to represent annotation uncertainty.The paper treats boundary location as inherently uncertain and models the likelihood of a boundary at each pixel.
  • Training objective: Distance-transform weighting emphasizes class-boundary pixels and compensates for the frequency imbalance between boundary and background pixels.The weights are normalized to [0...1] to remain consistent with the original HED model.
  • Two-stream processing: The hed·h network processes color and elevation in separate streams, concatenates their outputs, and uses 1×1 and fractional convolutions to reach output resolution.The color stream starts from original HED weights, while the DEM stream is trained from scratch.

fcn·h semantic segmentation network

The fcn·h family combines semantic segmentation with explicit class-boundary detection and uses complementary models for high-accuracy ensemble prediction, at substantial computational cost.

  • fcn·h architecture: fcn·h is an ensemble of two identical FCN architectures initialized with VGG and Pascal weights.The model uses separate image and DEM streams, additional skip connections, and fully connected layers.
  • Integrated boundary detection: Boundary-aware segmentation appends hed·h before fcn·h or seg·h, concatenating predicted boundary likelihoods with the corresponding inputs.The boundary detector receives colour imagery and DSM-derived inputs, producing an additional channel for segmentation.
  • Ensemble learning: Three boundary-aware networks—fcn·h-v, fcn·h-p, and seg·h—are combined by averaging their predicted class probabilities.The paper reports that their predictions are complementary and that the ensemble provides a significant performance boost.
  • Computational trade-offs: The ensemble is memory-hungry and expensive to train, so the authors recommend it mainly when aiming for highest accuracy.The two fcn·h models each have more than 100 million parameters, and trained fusion layers were not tested because the ensemble did not fit in GPU memory.
  • Training: Training the boundary and labeling components separately before assembly and fine-tuning gave the best results under their different outputs and loss functions.The overall boundary-plus-segmentation network required staged training even when pretrained components were used.

4. Experiment Results

Experiments evaluate the models on the ISPRS Vaihingen and Potsdam semantic-labeling datasets, using withheld test data and validation subsets with staged model training.

  • Datasets: The Vaihingen dataset contains 33 tiles at approximately 9 cm GSD, with 16 tiles for training and validation and 17 withheld for testing.Inputs include colour infrared imagery, a DSM, and manually annotated labels; an nDSM was also released.
  • Datasets: The Potsdam dataset contains 38 tiles at approximately 5 cm GSD, with 24 densely annotated training tiles and 14 withheld test tiles.Four training tiles were removed to form the validation set, and all reported Potsdam statistics refer to that set.
  • Dataset caveat: Potsdam reference labels contain imagined solid tree crowns in leaf-off imagery, producing semantic boundaries misaligned with actual image discontinuities.The annotation issue directly conflicts with the model’s assumption that class boundaries correspond to visible image evidence.
  • Training protocol: Each labeling and boundary-detection network was trained individually to convergence before the pretrained pieces were assembled and fine-tuned.The reported optimization used staged training with specified learning-rate schedules, batch sizes, momentum, and weight decay.

4.2. Results

Boundary-aware models are evaluated incrementally from a basic segmentation network through multi-scale processing and finally an ensemble, with performance varying across datasets and architectures.

  • Evaluation strategy: The evaluation adds components progressively: basic seg·h, class boundaries, multi-scale processing, and finally a DCNN ensemble.Post-processing with explicit smoothness or context models is not separately analyzed because fully connected CRFs did not improve results.
  • Vaihingen: 84.8% overall accuracy is achieved by single-scale seg·h on the Vaihingen validation set.The result is comparable to straightforward adaptations of state-of-the-art computer-vision DCNNs, which typically reach around 85%.
  • Vaihingen: 85.5% overall accuracy is achieved by the earlier fcn·h ensemble without explicit class-boundary detection.It performs better on low vegetation, impervious surfaces, and trees, but produces clearly worse building segmentations than seg·h.
  • Potsdam: 84.9%, 80.9% and 81.4% overall accuracy are reported on Potsdam for seg·h-sc1, fcn·h-p and fcn·h-v, respectively.The seg·h-sc1 network performs significantly better than the two standard fcn·h variants, while boundary information later closes the performance gap.

Effect of Class Boundaries

Explicit class-boundary detection improves semantic segmentation across CNN architectures and datasets, with the largest gains attributed to boundary information rather than multi-scale processing or ensembling.

  • 89.8% overall accuracy follows adding the hed·h boundary detector to seg·h, improving the single-scale model by more than 5 percentage points on Vaihingen.
  • Class boundaries significantly boost seg·h performance for all target classes, including vegetation classes with imprecise boundaries.The authors suggest imprecise vegetation boundaries can still help delimit high from low vegetation where DSMs are inaccurate.
  • 88.8% overall accuracy results when boundary detection is added to the fcn·h ensemble, a 3-percentage-point increase driven by buildings and trees.Car and low-vegetation classes lose accuracy in this comparison.
  • Up to 4.5 percentage points of overall-accuracy improvement appears on Potsdam when class boundaries are included.The tree class in seg·h-sc1 is an exception, with correctness falling from 74.4% to 68.6%.
  • Tables 2–4 report boundary-aware improvements for both tested CNN architectures and both datasets, while multi-scale and ensemble gains are smaller.
  • 0.2 percentage points is gained from explicit multi-scale processing, indicating smaller additional benefits than boundary detection on Vaihingen.The single-scale network apparently already captures relevant information in nadir-looking remote-sensing imagery.

Effect of the Ensemble

Ensembling complementary boundary-aware networks improves performance, especially over fcn·h, but benchmark results are sensitive to systematic nDSM errors that occur only in testing data.

  • The three-network boundary-aware ensemble beats seg·h and the two-model fcn·h ensemble, with a clear improvement over fcn·h.Its advantage over seg·h is marginal, while boundary augmentation allows seg·h to outperform its fcn·h counterparts.
  • 0.21 percentage points is the average ensemble gain across three Vaihingen validation tiles, while one tile shows a different outcome.Across the four tiles, boundary-aware models consistently beat their non-boundary counterparts.
  • One percentage point is added on Potsdam by averaging three networks whose individual overall accuracies are each approximately 85%.The boundary-supported ensemble performs best on the Potsdam validation set.
  • 3.1%, 9.3%, and 10.0% of three Vaihingen test images are covered by affected buildings missing from the nDSM.The missing-building errors result from incorrect filtering in the released elevation data.
  • Test-only nDSM errors cannot be handled correctly by a model because comparable situations are absent from the training set.
  • The paper reports both 89.4% and 90.3%, because competing methods used the biased test set and the correction was not repeated for Potsdam.The authors regard 90.3% as the true performance under training-test data consistency.

4.3. Comparison to state of the art

The class-contour ensemble ranks among the strongest published Vaihingen benchmark systems, exceeding 90% with corrected nDSM data while remaining near competing methods on the official test set.

  • 89.4% overall accuracy is achieved on the official Vaihingen test set, rising to 90.3% with the correct nDSM.
  • The model is among the top performers, compared with INRIA at 89.5% and ONERA at 89.8% on the benchmark.
  • Benchmark rankings are time-sensitive because several groups reached similar or slightly higher accuracy during the paper’s review.
  • Accuracy exceeds 90% over man-made classes, where segmentation boundaries are most well-defined and consequential.
  • Top-performing models have very similar overall performance and are all variants of fcn or segnet.
  • The corrected-nDSM result is better than most competitors on impervious surfaces and buildings, while vegetation scores remain comparable.

4.4. A word on data quality

The paper finds that annotation uncertainty and benchmark-data errors materially affect reported segmentation performance, especially for small or ambiguous classes.

  • Ground-truth inaccuracies occur in remote-sensing labels, including missing or incorrectly delineated objects illustrated in Figure 9.
  • Figures 7 and 8 compare predictions across boundary-aware model variants using class-specific colors for Vaihingen and Potsdam.
  • Tables 5–7 report confusion-matrix, model-identifier, and benchmark-comparison information for the reported Vaihingen results.
  • Cars show significant differences when new annotations, benchmark ground truth, and the best model output are compared.Cars are small and have a large fraction of pixels near object boundaries.
  • The dataset may be reaching its limits because annotation errors can influence conclusions when recent benchmark differences are generally below 2%.The authors argue that benchmarks should be extended or renewed every few years.

5. Conclusion

The paper develops DCNN models that explicitly represent semantic class boundaries and finds that this substantially improves segmentation across architectures and benchmarks. It also identifies benchmark annotation differences and class-specific boundaries as important limitations, while noting model size as a future concern.

  • Explicitly representing boundaries between semantic classes significantly improves different DCNN architectures and provides the largest performance boost in the final model.The final model achieves excellent performance on the ISPRS Vaihingen and Potsdam benchmarks.
  • The study evaluates semantic segmentation architectures across fully connected layers, class boundaries, multi-scale processing, and multi-network ensembles.
  • The benchmark may require re-annotation or replacement because significant differences occur between ISPRS Vaihingen ground truth, the authors’ annotation, and their best models.
  • The current boundaries are class-agnostic, leaving open whether class-specific boundaries could preserve which classes they separate.The paper suggests that detecting only boundaries might suffice if they form closed regions.
  • Model size is identified as an issue because the best-performing DCNN models are excessively large and complex.
Loading 1612.01337v2…