Source-linked AI summary
Foreground-Aware Relation Network for Geospatial Object Segmentation in High Spatial Resolution Remote Sensing Imagery
Zhuo Zheng, Yanfei Zhong, Junjue Wang, Ailong Ma
TL;DR
HSR remote-sensing object segmentation must handle scale variation, complex backgrounds, and foreground-background imbalance, whereas general methods mainly target scale variation. FarSeg introduces relation-based and optimization-based foreground modeling; experiments report effectiveness and a better speed–accuracy trade-off.
Problem
HSR remote-sensing segmentation involves scale variation, complex-background false alarms, and foreground-background imbalance, while general methods mainly address scale variation.
Method
FarSeg combines foreground-scene relation modeling, which enhances foreground features with correlated contexts, and foreground-aware optimization, which focuses training on foreground and hard background examples.
Results
FarSeg shows effectiveness and a better trade-off between speed and accuracy in comprehensive experiments.
Takeaways & Limitations
Explicit foreground modeling provides FarSeg’s supported strategy for reducing false alarms and alleviating foreground-background imbalance in HSR object segmentation.
Abstract
from arXiv · showhide
Geospatial object segmentation, as a particular semantic segmentation task, always faces with larger-scale variation, larger intra-class variance of background, and foreground-background imbalance in the high spatial resolution (HSR) remote sensing imagery. However, general semantic segmentation methods mainly focus on scale variation in the natural scene, with inadequate consideration of the other two problems that usually happen in the large area earth observation scene. In this paper, we argue that the problems lie on the lack of foreground modeling and propose a foreground-aware relation network (FarSeg) from the perspectives of relation-based and optimization-based foreground modeling, to alleviate the above two problems. From perspective of relation, FarSeg enhances the discrimination of foreground features via foreground-correlated contexts associated by learning foreground-scene relation. Meanwhile, from perspective of optimization, a foreground-aware optimization is proposed to focus on foreground examples and hard examples of background during training for a balanced optimization. The experimental results obtained using a large scale dataset suggest that the proposed method is superior to the state-of-the-art general semantic segmentation methods and achieves a better trade-off between speed and accuracy. Code has been made available at: \url{https://github.com/Z-Zheng/FarSeg}.
1. Introduction
HSR remote-sensing object segmentation is challenged by scale variation, complex backgrounds, and severe foreground-background imbalance. FarSeg addresses the latter two through explicit foreground modeling in its relation and optimization components.
- HSR imagery contains objects with large-scale variation, making object localization and recognition difficult.
- Complex backgrounds create false alarms because of their larger intra-class variance.
- A low foreground ratio produces a foreground-background imbalance problem in HSR imagery.
- FarSeg learns foreground-scene relations to associate foreground-correlated contexts and enhance foreground-feature discrimination, reducing false alarms.
- Foreground-aware optimization emphasizes foreground and hard background examples while down-weighting numerous easy background examples during training.
- FarSeg combines a multi-branch encoder based on FPN with foreground-scene relation and foreground-aware optimization components.
2. Related Work
General semantic segmentation research emphasizes spatial context and multi-scale representation, while remote-sensing applications often specialize methods for individual scenarios. The cited work argues that common HSR problems remain insufficiently modeled and motivates FarSeg.
- General Semantic Segmentation: Deep semantic segmentation methods learn feature representations from data and use convolutional networks to model image context.
- General Semantic Segmentation: Atrous convolution, ASPP, and PPM expand receptive fields or construct multi-scale features for semantic segmentation.
- General Semantic Segmentation: Encoder-decoder architectures reuse high-resolution shallow features and lightweight decoding to recover spatial detail with limited overhead.
- General Semantic Segmentation: General semantic segmentation methods mainly model multi-scale context, overlooking false alarms and foreground-background imbalance in HSR imagery.
- Semantic Segmentation in Remote Sensing Community: Remote-sensing applications include land-cover classification, building and road extraction, and vehicle detection, with many methods tailored to specific scenarios.
- Semantic Segmentation in Remote Sensing Community: The paper proposes FarSeg to address common false-alarm and foreground-background-imbalance problems across large-scale HSR imagery.
3. Foreground-Aware Relation Network
FarSeg combines multi-scale foreground representation, foreground-scene relation modeling, lightweight decoding, and foreground-aware optimization for HSR geospatial object segmentation.
- Multi-branch encoder: FarSeg uses a multi-branch encoder with an FPN-based foreground branch and a scene embedding branch sharing one backbone.The FPN produces pyramidal features for multi-scale object segmentation, while the scene branch learns geospatial scene representation.
- Foreground-scene relation: The F-S relation module aligns foreground and scene representations in a shared manifold and uses their similarity to re-weight foreground feature maps.Scale-aware and scene projection functions produce compatible representations; the resulting relation map enhances foreground discrimination.
- Foreground-scene relation: The scene embedding is computed from C6 and shared across pyramid levels because latent geospatial scene semantics are treated as scale-invariant.The scene projection uses a learnable 1×1 convolution with output dimension d_u, and pointwise inner products estimate similarity.
- Light-weight decoder: The lightweight decoder stacks channel transformations with optional 2× upsampling, aggregates pyramid outputs by mean, and produces an input-sized class probability map.Each pyramid level is decoded according to its scale, followed by 4× bilinear upsampling after aggregation.
- Foreground-aware optimization: Foreground-aware optimization addresses gradient domination by background examples through hard-example estimation, dynamic weighting, and back-propagation.Normalized focal-style weights preserve the loss sum, while annealing progressively shifts emphasis toward hard examples as confidence improves.
4. Experiments
Experiments on iSAID evaluate FarSeg against general segmentation methods, analyze its modules, and examine speed–accuracy and optimization behavior. Results show improved trade-offs, effective foreground-scene modeling, and gains from foreground-aware optimization.
- Experimental setting: iSAID contains 2,806 HSR images across 15 object categories, with training and validation sets used because the test set is unavailable.The dataset includes 655,451 instance annotations; experiments use semantic masks, training on 1,411 images and evaluating on 458 validation images.
- Comparison to general methods: FarSeg outperforms the compared CNN-based segmentation methods on the iSAID validation set.Comparisons include U-Net, FCN-8s, DenseASPP, Deeplab variants, Semantic FPN, RefineNet, and PSPNet, using mIoU as the main metric.
- Comparison to general methods: FarSeg achieves a better speed–accuracy trade-off on iSAID, supported by its lightweight and effective module design.Figure 6 plots FPS against mIoU, with circle radius representing parameter count.
- Foreground-Scene Relation Module: Adding F-S relation without and with scale-aware projection improves mIoU by 1.11% and 1.18%, respectively.The corresponding modules add 2.89 M and 1.12 M parameters, or approximately 10% and 4% relative increases.
- Foreground-Scene Relation Module: Scale-aware scene projection performs better, indicating that geospatial scene representation relates to scale and foregrounds.Visualizations show scale-specific foreground emphasis alongside scale-agnostic contextual emphasis across pyramid levels.
- Foreground-Aware Optimization: F-A optimization improves mIoU by 2.2% and 3.24% without extra computation or memory, while γ = 2 yields a 3.29% baseline gain and 63.71% mIoU.Naive softmax focal loss drops mIoU by 4.05%, normalization gains 6.54% over it, and cosine annealing provides a 0.63% gain; γ = 5 causes a decline.
5. Conclusion
FarSeg addresses false alarms and foreground-background imbalance through foreground-scene relation learning and foreground-aware optimization. Comprehensive experiments report its effectiveness and a better speed–accuracy trade-off.
- FarSeg targets false alarms and foreground-background imbalance as bottlenecks in HSR remote sensing object segmentation.
- The network learns foreground-scene relations to enhance foreground features and uses foreground-aware optimization for balanced training.
- Comprehensive experimental results show FarSeg’s effectiveness and a better trade-off between speed and accuracy.