PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Segmentation of Remote Sensing Images with Sparse Annotations

Yuansheng Hua, Diego Marcos, Lichao Mou, Xiao Xiang Zhu, Devis Tuia

arXiv:2101.03492v1cs.CV

TL;DR

Dense pixel-level annotations for very-high-resolution aerial images are costly to produce. This paper trains segmentation CNNs from sparse labels using FESTA, which regularizes feature and spatial relations, and reports improved results with sparse annotations, with line-level labels performing best on both datasets despite fewer labeled pixels than polygons.

  • Problem

    Fully supervised segmentation of very-high-resolution images requires dense pixel-level labels that are labor- and time-consuming to produce and may require expert annotators for ambiguous boundaries.

    Source: §I · p. 2

  • Method

    FESTA complements sparse-label supervision by regularizing feature and spatial relations, using unlabeled pixels to improve features for semantic segmentation.

    Source: B. Feature and Spatial Relational… · p. 3

  • Results

    Line-level annotations led to the best performance on both datasets despite using an order of magnitude fewer labeled pixels than polygon annotations, and FESTA improved segmentation results with sparse labels.

    Source: E. Discussion on annotation type · p. 6, §IV · p. 6

  • Takeaways & Limitations

    The semi-supervised approach helps close the performance gap to dense-label models while using sparse annotations that can significantly reduce label-generation costs.

    Source: §IV · p. 6

    Mean F1 performance remained 13% below dense supervision on Vaihingen and 8% below it on Zurich.

    Source: E. Discussion on annotation type · p. 6

Abstract

from arXiv · show

Training Convolutional Neural Networks (CNNs) for very high resolution images requires a large quantity of high-quality pixel-level annotations, which is extremely labor- and time-consuming to produce. Moreover, professional photo interpreters might have to be involved for guaranteeing the correctness of annotations. To alleviate such a burden, we propose a framework for semantic segmentation of aerial images based on incomplete annotations, where annotators are asked to label a few pixels with easy-to-draw scribbles. To exploit these sparse scribbled annotations, we propose the FEature and Spatial relaTional regulArization (FESTA) method to complement the supervised task with an unsupervised learning signal that accounts for neighbourhood structures both in spatial and feature terms.

I. INTRODUCTION

Dense pixel-level ground truth makes fully supervised aerial-image segmentation costly and may require expert interpretation of boundaries and ambiguous regions. The paper proposes learning from sparse annotations with feature- and spatial-relation regularization, evaluated on two very-high-resolution datasets.

I. INTRODUCTION

  • Dense pixel-level ground truths are labor- and time-consuming to produce, and boundary or ambiguous pixels may require expert annotators.Examples of ambiguity include shadows.Source: §I · p. 2
  • Point- and scribble-supervised segmentation remain under-explored in remote sensing, despite scribbles providing stronger supervision than points and being easier to generate than dense labels.The introduction also discusses polygon-level annotations as a low-cost option covering more pixels than other sparse types.Source: §I · p. 2
  • The proposed framework learns from sparse annotations by regularizing feature and spatial relations, and is evaluated on the Vaihingen and Zurich Summer datasets.The paper considers point-, scribble-, and polygon-level annotations.Source: §I · p. 2

II. METHODOLOGY

Sparse annotations label only a small fraction of pixels and need not cover whole objects, reducing annotation effort by allowing complex boundaries and ambiguous pixels to be avoided. The methodology considers point-, scribble-, and polygon-level labels.

A. Supervision with Sparse Annotations

B. Feature and Spatial Relational Regularization

FESTA uses unlabeled image pixels by regularizing relations in both feature and spatial domains. For each feature vector, it draws on similar and dissimilar samples and a feature-similar spatial neighbor, alongside cross-entropy on annotated pixels.

B. Feature and Spatial Relational Regularization

  • FESTA supplements sparse supervision by encoding and regularizing pixel relations in feature and spatial domains, leveraging the many unlabeled pixels.The method aims to make learned features more useful for semantic segmentation.Source: B. Feature and Spatial Relational… · p. 3
  • For each CNN feature vector, FESTA identifies the most feature-similar sample as a neighbor and the least similar sample as a far-away sample.The proximity assumption motivates bringing the feature neighbor closer while increasing dissimilarity from the far-away sample.Source: B. Feature and Spatial Relational… · p. 3
  • FESTA selects the most feature-similar of the sample’s eight spatial neighbors to avoid pairing it with a neighbor at an object boundary.Source: B. Feature and Spatial Relational… · p. 3
  • The training objective combines FESTA regularization with categorical cross-entropy computed on annotated pixels.The FESTA loss uses trade-off parameters for its terms and the number of image pixels.Source: B. Feature and Spatial Relational… · p. 3

C. CRF for Boundary Refinement

A fully connected CRF is integrated to refine predictions from networks trained on scribbles. Its pairwise terms use appearance and smoothness kernels, with smoothing expected within homogeneous areas.

C. CRF for Boundary Refinement

  • A fully connected CRF is added to refine predictions from networks trained on scribbled annotations.Source: C. CRF for Boundary Refinement · p. 3
  • The CRF energy uses unary potentials from pixel label probabilities and pairwise potentials between pixels.The unary potential is the negative log probability of a pixel label.Source: C. CRF for Boundary Refinement · p. 3
  • The appearance kernel favors assigning similar-color pixels to the same class, while the smoothness kernel penalizes nearby pixels with different labels.This is expected to smooth class maps within homogeneous areas.Source: C. CRF for Boundary Refinement · p. 3

A. Dataset Description

The experiments use the Vaihingen and Zurich Summer aerial-image datasets, with dense annotations reserved for testing and evaluation. The datasets differ in image resolution, spectral bands, class coverage, and train-test split.

A. Dataset Description

  • Table I reports the numbers of pixels labeled by sparse point-, line-, and polygon-level annotations alongside dense annotations for both datasets.Background/Clutter is not considered.Source: Table I · p. 4, A. Dataset Description · p. 4
  • Vaihingen uses five test images (IDs 11, 15, 28, 30, 34) from a 16-image fully annotated subset; six land-cover classes are considered, with Background/Clutter excluded.The imagery has 9 cm spatial resolution and NIR, red, and green bands; each image covers an average 1.38 km².Source: A. Dataset Description · p. 4
  • Zurich Summer contains 20 images at 0.62 m resolution; 15 train the model and five (IDs 16–20) are held out for testing.Experiments use NIR, red, and green from the four available channels and consider eight urban classes, with uncategorized pixels labeled background.Source: A. Dataset Description · p. 4
  • Although both datasets provide dense pixel-wise annotations, the models are trained with scribbled annotations and dense labels are used only for test-phase metric calculation.The Vaihingen benchmark comprises 33 aerial images; 16 are fully annotated.Source: A. Dataset Description · pp. 3–4

B. Scribbled Annotation Generation

Annotators create point, line, or polygon labels within objects rather than tracing complete boundaries. The protocol seeks diverse class appearances and comparable time across annotation levels.

B. Scribbled Annotation Generation

  • Four annotators, including two non-experts, label seven, five, and three objects per class per image for point-, line-, and polygon-level annotations, respectively.Annotations should cover diverse appearances and different image positions; precise polygon and line boundaries are not required.Source: B. Scribbled Annotation Generation · p. 4, Figure 3 · p. 4
  • The Vaihingen polygon-annotation example highlights varied visual appearances among cars, with colors encoding impervious surfaces, buildings, low vegetation, trees, and cars.The legend assigns white, blue, cyan, green, and yellow to those classes, respectively.Source: Figure 3 · p. 4
  • Point- and line-level labels are dilated with a radius-3 disk, while pixels inside each polygon receive that polygon’s class label.The dilation assigns nearby pixels the class of the central point or line.Source: B. Scribbled Annotation Generation · p. 4
  • Sparse labels comprise several orders of magnitude fewer labeled pixels than dense annotations, while per-image annotation times range from 126 to 238 seconds across datasets and levels.Vaihingen times are 133, 126, and 161 seconds for point, line, and polygon labels; Zurich Summer times are 177, 162, and 238 seconds, respectively.Source: B. Scribbled Annotation Generation · p. 4

C. Training Details

Training uses an FCN-16s with a VGG-16 backbone and a loss combining annotation-based cross-entropy with FESTA regularization. Evaluation reports per-class and mean F1 scores and overall accuracy on both datasets.

C. Training Details

  • The FCN-16s uses VGG-16, fuses upsampled outputs from the last two convolutional blocks, and trains all weights with Nesterov Adam on 256×256 patches.Training runs for 100,000 iterations with batch size 5; the initial learning rate is 2e-04, and it decays by a factor of 10 when validation loss saturates.Source: C. Training Details · p. 4
  • The loss is defined as Eq. 2, with λ set experimentally to 0.1 for Vaihingen and 0.01 for Zurich Summer; trade-off parameters α, β, and γ are 0.5, 1.5, and 1.Source: C. Training Details · p. 4
  • Tables II and III report per-class F1, mean F1, and overall accuracy on the respective test sets, with sparse-label results averaged across four annotators as mean ± standard deviation.Dense-annotation results are provided as reference.Source: Table II · p. 5, Table III · p. 5
  • Dense CRF refines test predictions, with parameters θ1, θ2, and θ3 tuned on validation images and set to 30, 10, and 10.The same settings are reported as satisfactory for FCN and FCN-FESTA.Source: C. Training Details · p. 5

D. Comparing with Existing Methods

Across Vaihingen and Zurich Summer, FESTA improves segmentation results under sparse supervision, with line-level annotations producing the highest reported mean F1 scores. Comparisons include weighted-loss and dense-CRF baselines.

D. Comparing with Existing Methods

  • The comparison evaluates FCN-FESTA against FCN-WL on sparse labels, includes dense-label FCN results as a baseline, and tests dense-CRF refinement.Each model is trained and validated independently on sparse annotations; test metrics are calculated against dense annotations and averaged across four annotators.Source: D. Comparing with Existing Methods · p. 5
  • 70.58% mean F1 on Vaihingen and 69.43% on Zurich Summer are the reported line-level maxima for FESTA with dense CRF.The Vaihingen prose names the model FCN-FAST+dCRF; the Zurich Summer prose names FCN-FESTA+dCRF.Source: D. Comparing with Existing Methods · p. 5
  • On Vaihingen, FCN-FESTA improves mean F1 over FCN-WL by 3.95% with point-level and 3.33% with polygon-level supervision.With dense CRF, FCN-FESTA+dCRF improves over FCN+dCRF by 2.38% and 4.03%, respectively.Source: D. Comparing with Existing Methods · p. 5
  • On Zurich Summer, FESTA increases mean F1 by 3.92%, 1.37%, and 1.31% with point-, line-, and polygon-level annotations, respectively.The passage also reports that dense CRF significantly improves results for networks trained on point-level scribbles.Source: D. Comparing with Existing Methods · p. 5

E. Discussion on annotation type

Line-level annotations achieve the best performance on both datasets despite using an order of magnitude fewer labeled pixels than polygon annotations. Sparse-label results remain below dense-supervision results, with the gap varying by class.

E. Discussion on annotation type

  • Figures 4 and 5 show example segmentation results from line-annotated models on Vaihingen and Zurich Summer, respectively.Figure 5’s legend identifies road, soil, grass, tree, building, and background classes.Source: Figure 4 · p. 6, Figure 5 · p. 6
  • Line-level annotations lead to the best performance on both datasets despite using an order of magnitude fewer labeled pixels than polygon annotations.The authors suggest more balanced weighting of differently sized objects and the possibility of more scribbles per time budget as explanations.Source: E. Discussion on annotation type · p. 6
  • The mean F1 gap from dense supervision is 13% on Vaihingen and 8% on Zurich; Zurich railway is systematically missed even with dense supervision.The gap is smaller or absent for homogeneous classes and larger for diverse-material classes such as building and car in Vaihingen.Source: E. Discussion on annotation type · p. 6

IV. CONCLUSION

The paper proposes a semi-supervised framework for aerial-image segmentation with sparse annotations and evaluates it on Vaihingen and Zurich Summer. Results suggest improved segmentation while reducing annotation costs, though sparse-label models remain less accurate than dense-label models.

IV. CONCLUSION

  • The proposed framework combines sparse annotations with a semi-supervised learning objective for semantic aerial-image segmentation.Source: §IV · p. 6
  • Experiments on Vaihingen and Zurich Summer suggest the method improves segmentation results across several kinds of sparse annotations.Source: §IV · p. 6
  • The authors identify graph-based models and prior knowledge learned from label semantics as future directions for improving the framework.Source: §IV · p. 6
Loading 2101.03492v1…