Source-linked AI summary
Semantic Labeling in Very High Resolution Images via a Self-Cascaded Convolutional Neural Network
Yongcheng Liu, Bin Fan, Lingfeng Wang, Jun Bai, Shiming Xiang, Chunhong Pan
TL;DR
Urban VHR semantic labeling is challenging because confusing manmade objects and fine-structured objects make coherent and accurate predictions difficult. The paper proposes ScasNet, an end-to-end self-cascaded CNN that aggregates global-to-local contexts, refines objects coarse-to-fine, and corrects fusion residuals. Extensive experiments report state-of-the-art performance on three public datasets, including the Vaihingen and Potsdam benchmarks.
Problem
Urban VHR images contain confusing manmade objects and intricate fine structures, making coherent and accurate semantic labeling difficult.
Method
ScasNet sequentially aggregates multi-scale contexts globally to locally, progressively refines objects with shallow-layer features, and applies residual correction during multi-feature fusion.
Results
ScasNet achieves state-of-the-art performance on three public datasets, including the Vaihingen and Potsdam benchmarks, using image data without elevation data, ensembles, or postprocessing.
Takeaways & Limitations
The proposed end-to-end framework produces more coherent, complete, and accurate labeling with better robustness to occlusions and cast shadows than compared advanced deep models.
Takeaways & Limitations
ResNet ScasNet has difficulty converging without batch normalization, whereas VGG ScasNet can converge without it.
Abstract
from arXiv · showhide
Semantic labeling for very high resolution (VHR) images in urban areas, is of significant importance in a wide range of remote sensing applications. However, many confusing manmade objects and intricate fine-structured objects make it very difficult to obtain both coherent and accurate labeling results. For this challenging task, we propose a novel deep model with convolutional neural networks (CNNs), i.e., an end-to-end self-cascaded network (ScasNet). Specifically, for confusing manmade objects, ScasNet improves the labeling coherence with sequential global-to-local contexts aggregation. Technically, multi-scale contexts are captured on the output of a CNN encoder, and then they are successively aggregated in a self-cascaded manner. Meanwhile, for fine-structured objects, ScasNet boosts the labeling accuracy with a coarse-to-fine refinement strategy. It progressively refines the target objects using the low-level features learned by CNN's shallow layers. In addition, to correct the latent fitting residual caused by multi-feature fusion inside ScasNet, a dedicated residual correction scheme is proposed. It greatly improves the effectiveness of ScasNet. Extensive experimental results on three public datasets, including two challenging benchmarks, show that ScasNet achieves the state-of-the-art performance.
1. Introduction
Semantic labeling of urban VHR images remains difficult because confusing manmade objects and fine structures require coherent, accurate pixel-level predictions. ScasNet addresses these challenges through self-cascaded context aggregation, coarse-to-fine refinement, and residual correction in an end-to-end framework.
- Challenges: Urban VHR semantic labeling assigns every pixel to object categories but is difficult because objects vary structurally, overlap visually, and interact through occlusions and shadows.The task simultaneously covers multiple categories rather than only single-class extraction.
- Limitations of prior methods: Existing methods struggle to acquire multi-scale contexts efficiently for confusing objects and to exploit low-level features effectively for fine-structured objects.Directly combining both solutions in one network is further complicated by fitting residuals caused by semantic gaps across feature levels.
- Proposed method: ScasNet sequentially aggregates multi-scale contexts from global to local scales, retaining hierarchical dependencies for more coherent recognition of confusing manmade objects.The architecture captures contexts on CNN encoder outputs and successively aggregates them in a self-cascaded manner.
- Proposed method: ScasNet progressively refines fine-structured objects using low-level features learned by shallow CNN layers, supporting accurate labeling of detailed structures.The strategy follows a coarse-to-fine refinement process with long-span connections.
- Proposed method: A residual correction scheme addresses latent fitting residuals from semantic gaps during multi-feature fusion and improves the effectiveness of the two complementary solutions.These components form an end-to-end deep learning framework for semantic labeling.
- Evaluation: ScasNet outperforms state-of-the-art methods on the Vaihingen and Potsdam benchmarks using only image data, without elevation data, ensembles, or postprocessing.The current work also adds comparative experiments on two further challenging datasets and more extensive analyses.
2. Preliminaries
CNNs hierarchically learn feature maps through stacked trainable layers. ScasNet uses standard CNN operations, including convolution, pooling, dropout, interpolation, elementwise fusion, and softmax classification.
- CNN basics: CNNs hierarchically extract low-level and high-level features, with each layer receiving and producing sets of arrays called feature maps.Standard CNNs contain convolutional, nonlinear, and pooling layers.
- CNN basics: Convolutional layers apply shared-kernel dot products over local receptive fields to generate feature maps while reducing parameter count.A typical example is a 3×3 kernel with shared weights and bias across a feature map.
- Basic layers: Pooling performs spatial subsampling; ScasNet uses max-pooling, which selects the maximum value within each pooling region.Pooling reduces spatial resolution while retaining selected responses.
- Basic layers: Dropout randomly removes units during training to reduce overfitting, while bilinear interpolation resizes feature maps spatially.Elementwise sum fusion combines same-sized, same-channel feature maps, and softmax outputs posterior probabilities over categories.
3. Self-cascaded Convolutional Neural Network (ScasNet)
ScasNet addresses VHR urban-image labeling by combining sequential global-to-local context aggregation, coarse-to-fine refinement, and residual correction in one CNN. These components target confusing manmade objects, fine-structured objects, and fitting residuals from multi-feature fusion.
- Network Overview: ScasNet performs pixel-level semantic labeling using a self-cascaded CNN with five described aspects, including context aggregation, object refinement, residual correction, configuration, and learning.The network uses a conventional CNN encoder and is presented as an end-to-end model.
- Multi-scale contexts Aggregation: Dilated convolutions capture multi-scale contexts from the encoder’s final layer, expanding receptive fields and supplying wider high-level visual cues.The paper notes that dilation also alleviates the practically smaller receptive fields of deep features.
- Multi-scale contexts Aggregation: ScasNet sequentially aggregates contexts from global to local, preserving hierarchical dependencies that parallel stacking may lose.The experiment uses dilation rates 24, 18, 12, and 6; larger rates provide higher-level context and smaller rates lower-level context.
- Multi-scale contexts Aggregation: The architecture’s self-cascaded design acquires deep multi-scale contexts more efficiently than using multiple input images while retaining hierarchical cues and abstract semantics.The cited advantages are specifically framed for recognizing confusing objects in VHR imagery.
- Fine-structured Objects Refinement: The refinement strategy progressively reintroduces shallow-layer features through long-span connections, refining coarse maps and recovering low-level object details.This targets boundaries and localization degraded by repeated subsampling in FCN-based methods.
- Residual Correction: Residual correction compensates for latent fitting residuals caused by semantic gaps when multi-level features and contexts are fused.Embedded residual-correction modules are intended to prevent residual accumulation and support collaborative global-to-local and coarse-to-fine labeling.
4. Experiments and Evaluations
Experiments evaluate ScasNet on three public datasets, including Massachusetts building and ISPRS Vaihingen and Potsdam benchmarks, using qualitative, quantitative, ablation, and transfer-learning analyses. Across comparisons, ScasNet produces coherent, accurate labeling for confusing and fine-structured objects, while ablations and feature visualizations support its design choices.
- Dataset Description: ScasNet is evaluated on three public datasets, including the Massachusetts Building, ISPRS Vaihingen, and ISPRS Potsdam datasets.The experiments include benchmark datasets and Massachusetts building imagery with varied urban objects.
- Comparison with Deep Models: On Massachusetts imagery, Ours-ResNet generates more coherent labeling for confusing and fine-structured buildings than the compared deep models.The comparisons report difficulties with size-varied buildings and incomplete or inaccurate fine-structured labeling among several competitors.
- Comparison on Benchmark Test: On Vaihingen and Potsdam benchmark tests, ScasNet achieves coherent, accurate, smooth, and precisely localized results, with quantitative performance outperforming competing methods.The reported advantages include robustness to cast shadows and strong performance for fine-structured objects such as cars; Potsdam improvements extend across all categories.
- Model Analysis: Nearly 3% improvement in mean IoU results from self-cascaded context fusion over the baseline parallel multi-scale context stack.Residual correction further improves performance, while refinement improves category performance when combined with residual correction.
- Model Analysis: Transfer learning improves ScasNet, with a particularly significant benefit for the 101-layer ResNet model on the small remote-sensing datasets.The authors report only slight improvement for VGG ScasNet but significant improvement for ResNet ScasNet.
- Model Analysis: Feature visualizations show that multi-scale context aggregation produces coherent semantic responses, refinement recovers low-level object details, and residual correction mitigates information loss during fusion.Recovered boundary responses are visible for cars and trees, while post-correction fused feature maps show reduced deficiencies.
5. Conclusion
ScasNet combines self-cascaded global-to-local context aggregation, coarse-to-fine refinement, and residual correction for VHR semantic labeling. Experiments report stronger labeling quality, benchmark performance, and complexity advantages.
- 5. Conclusion: ScasNet sequentially aggregates multiscale global-to-local contexts to recognize confusing manmade objects.The network captures contexts from a CNN encoder and aggregates them successively in a self-cascaded architecture.
- 5. Conclusion: A coarse-to-fine refinement strategy progressively improves labeling of intricate fine-structured objects using shallow-layer low-level features.Residual correction also addresses fitting residuals caused by semantic gaps during multi-feature fusion.
- 5. Conclusion: ScasNet produces more coherent, complete, accurate, and robust labeling results than the compared advanced deep models.The reported robustness includes handling occlusions and cast shadows in both quantitative and visual evaluations.
- 5. Conclusion: ScasNet outperforms state-of-the-art methods on the Vaihingen and Potsdam ISPRS semantic-labeling benchmarks without elevation data, ensembles, or postprocessing.These benchmark comparisons were reported as of the paper’s submission date.
- 5. Conclusion: ScasNet offers space- and time-complexity advantages over some complex deep models.The paper also provides VGG and ResNet ScasNet variants for different accuracy–complexity trade-offs.