Source-linked AI summary
Effective Use of Dilated Convolutions for Segmenting Small Object Instances in Remote Sensing Imagery
Ryuhei Hamaguchi, Aito Fujita, Keisuke Nemoto, Tomoyuki Imaizumi, Shuhei Hikosaka
TL;DR
Remote-sensing imagery contains small, crowded objects, creating a need for segmentation methods that retain resolution while capturing context. The paper attaches a decreasing-dilation Local Feature Extraction module to a dilated front-end to aggregate local features. Across three remote-sensing datasets, the method performs especially well for small objects; its scope is qualified by the differing severity of occlusion in remote-sensing imagery.
Problem
Small, crowded objects in high-resolution remote-sensing imagery are difficult to segment because aggressive dilation can fail to aggregate their local features.
Method
The proposed LFE module uses decreasing dilation after an increasingly dilated front-end to aggregate local features while preserving resolution.
Results
Across three remote-sensing datasets, the proposed model outperforms state-of-the-art methods such as U-Net and Deeplab, especially for small objects.
Takeaways & Limitations
The architecture is effective for segmenting crowded small object instances and may apply to other settings where crowded small instances matter.
Takeaways & Limitations
The method is presented for remote-sensing imagery where occlusion is not as serious as in ground-based images; severe occlusion may require integration with instance-aware pipelines.
Abstract
from arXiv · showhide
Thanks to recent advances in CNNs, solid improvements have been made in semantic segmentation of high resolution remote sensing imagery. However, most of the previous works have not fully taken into account the specific difficulties that exist in remote sensing tasks. One of such difficulties is that objects are small and crowded in remote sensing imagery. To tackle with this challenging task we have proposed a novel architecture called local feature extraction (LFE) module attached on top of dilated front-end module. The LFE module is based on our findings that aggressively increasing dilation factors fails to aggregate local features due to sparsity of the kernel, and detrimental to small objects. The proposed LFE module solves this problem by aggregating local features with decreasing dilation factor. We tested our network on three remote sensing datasets and acquired remarkably good results for all datasets especially for small objects.
1. Introduction
Remote-sensing imagery contains smaller, denser object instances than ground-based imagery, making it important to preserve resolution while capturing broad context. The paper proposes an LFE module with decreasing dilation to recover local features after increasingly dilated convolutions and reports strong results, especially for small objects.
- Motivation: Higher satellite-image resolution makes individual small objects distinguishable, and object counts can provide economically valuable information.The paper gives newly constructed buildings as an example of an informative object count.
- Motivation: Remote-sensing imagery differs from ground-based imagery because its objects are significantly smaller and more densely arranged.The comparison is illustrated using satellite and ground-based images with the same pixel dimensions.
- Challenge: Segmenting crowded small objects requires both broad context and high resolution, but subsampling expands receptive fields while progressively losing feature detail.Coarse features may miss small-object details that skip connections or hypercolumns cannot fully recover.
- Challenge: Increasing dilation expands receptive field without reducing resolution, but aggressive dilation sparsifies kernels and fails to aggregate local features of small objects.The paper identifies this sparsity as a side effect that can be detrimental to small-object segmentation.
- Proposed solution: The proposed LFE module attaches decreasingly dilated convolutions above increasingly dilated convolutions so kernel weights become denser and local features are aggregated.The module is presented as a rescue mechanism for increasingly dilated convolutions.
- Results: Across three remote-sensing datasets, the proposed model outperforms U-Net and Deeplab, especially for small objects, while LFE smooths grid-like effective receptive fields.The effective receptive field analysis is used to examine the LFE module's effect.
2. Related work
Related work applies established semantic-segmentation architectures to remote sensing, but resolution and localization remain central concerns. The paper distinguishes its approach by retaining resolution with dilated convolutions rather than relying on downsampling or stride-one pooling that reduces effective resolution.
- Semantic segmentation: Semantic segmentation assigns every input pixel a semantic category, while precise object localization remains a major challenge.Coarse spatial features from classification networks can produce unclear boundaries.
- Semantic segmentation: Prior semantic-segmentation work addresses localization using multi-resolution prediction maps, encoder-decoder architectures, skip connections, and related designs.These approaches combine or recover information across feature resolutions.
- Remote-sensing segmentation: Remote-sensing segmentation commonly adapts FCNs, skip connections, encoder-decoder architectures, or dilated convolutions from computer vision.The paper situates its method within these established architectural directions.
- Comparison with prior work: A closely related remote-sensing method uses stride-one max-pooling after dilated convolutions, which decreases the actual resolution of extracted feature maps.The paper contrasts this with its own resolution-preserving design.
- Proposed architecture: The proposed architecture uses dilated convolutions throughout its modules to preserve resolution while extracting large context and aggregating local features.Its front-end expands context, whereas the LFE module aggregates local features scattered by the front-end.
3. Proposed Method
The proposed model preserves resolution while combining large-context extraction with local-feature aggregation. Its LFE module addresses spatial inconsistency and lost local structure caused by aggressively dilated front-end convolutions.
- Post-processing: The model outputs a full-resolution probability map, which is thresholded to obtain individual object mask proposals.Each mask receives an object score computed as the mean probability within the mask.
- Front-end module: Dilated convolutions enlarge receptive fields while maintaining feature resolution, avoiding the resolution loss of subsampling layers.Increasing dilation expands kernel size and receptive field through increasingly sparse weight alignment.
- Front-end module: Aggressively increasing dilation weakens spatial consistency between neighboring units and prevents higher layers from extracting local structure.Adjacent information pyramids can become non-overlapping, with the non-overlap region growing as dilation increases.
- Local feature extraction module: The LFE module uses decreasing dilation factors after the increasingly dilated front-end to reconnect neighboring information pyramids and recover local structure.Its convolutional layers aggregate local features as kernel weights become denser.
4. Experiments
Experiments across three remote sensing datasets compare pooling and dilated front-ends, with and without LFE, using instance-aware and pixel-level metrics. LFE improves performance for small objects, smooths receptive-field irregularities, and remains effective as input resolution decreases.
- Experimental setup: The method is evaluated on Toyota City, Massachusetts Buildings, and Vaihingen datasets using APr, APr vol, and AR.The datasets support method validation and benchmarking against previously proposed segmentation methods.
- Model comparisons: LFE performs best on the instance metric, while dilated front-ends outperform pooling-based front-ends.Small front-end fields of view also outperform large ones in the reported Toyota City experiments.
- Effect of LFE module: +2.4% is the performance gain from LFE for the large-FOV comparison, versus +0.7% for its small-FOV counterpart.The reported gains are for Front-L+D+Large versus Front-L+D+LFE, and Front-S+D+Large versus Front-S+D+LFE, respectively.
- Metric analysis: Pixel-level and instance-level metrics can diverge: Front-S+D+Large improves APr by +7.0% over Front-S despite only a +0.4% pixel-level F1 improvement.Small objects have little impact on pixel-level metrics in these experiments.
- ERF and output analysis: LFE smooths grid-like effective receptive fields and output probability-map jaggedness in trained dilated models.The analysis attributes the smoothed receptive fields to recovering local information missed in lower layers.
- Resolution sensitivity: At lower input resolutions, LFE produces larger performance improvements and detects small objects more effectively than pooling-based front-ends.The sensitivity analysis uses downsampled Massachusetts Buildings datasets at 2 m and 3 m resolution.
5. Conclusion
The paper presents a dilated-convolution architecture for segmenting crowded small objects in remote sensing imagery. Its LFE module addresses a limitation of conventional dilated convolutions, and results across three datasets show particular effectiveness for small instances.
- Contribution: The proposed architecture uses dilated convolution to precisely segment crowded small object instances in remote sensing imagery.The conclusion identifies the architecture as the paper’s central contribution.
- Contribution: The paper identifies a problem in conventional dilated convolution and proposes an architecture to address it.The conclusion summarizes the proposed solution at the architectural level.
- Results: The method shows effectiveness for small object instances across three remote sensing datasets.The authors suggest application to various remote sensing tasks.
- Scope: The approach is expected to extend beyond remote sensing to crowded small-instance tasks such as cell segmentation, crowd counting, and pedestrian detection.These domains are presented as possible areas of applicability.