PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Aggregate Multi-Scale Context for Instance Segmentation in Remote Sensing Images

Ye Liu, Huifang Li, Chao Hu, Shuang Luo, Yan Luo, Chang Wen Chen

arXiv:2111.11057v4cs.CV

TL;DR

Remote-sensing instance segmentation is difficult because natural-image methods can degrade on top-view imagery with weakly discriminative object features. CATNet aggregates global context across feature, spatial, and instance domains. It reports state-of-the-art performance on four datasets at similar computational costs, while noting that its modules model each domain separately.

  • Problem

    Methods designed for natural images can degrade on top-view remote-sensing images, where scale variation, low contrast, and clustered distributions contribute to insufficiently discriminative object features.

    Source: §I · p. 2

  • Method

    CATNet combines DenseFPN, SCP, and HRoIE to aggregate global visual context in feature, spatial, and instance domains.

    Source: §I · p. 2

  • Results

    CATNet achieves new state-of-the-art performance on iSAID, DIOR, NWPU VHR-10, and HRSID at similar computational costs.

    Source: §I · pp. 2–3

  • Takeaways & Limitations

    The collaboration among the modules enhances discriminative object features for promoting object detection and instance segmentation performance.

    Source: §V · p. 13

    Each module aggregates context in only one domain, which may be sub-optimal when cross-domain context is correlated.

    Source: §V · p. 14

Abstract

from arXiv · show

The task of instance segmentation in remote sensing images, aiming at performing per-pixel labeling of objects at instance level, is of great importance for various civil applications. Despite previous successes, most existing instance segmentation methods designed for natural images encounter sharp performance degradations when they are directly applied to top-view remote sensing images. Through careful analysis, we observe that the challenges mainly come from the lack of discriminative object features due to severe scale variations, low contrasts, and clustered distributions. In order to address these problems, a novel context aggregation network (CATNet) is proposed to improve the feature extraction process. The proposed model exploits three lightweight plug-and-play modules, namely dense feature pyramid network (DenseFPN), spatial context pyramid (SCP), and hierarchical region of interest extractor (HRoIE), to aggregate global visual context at feature, spatial, and instance domains, respectively. DenseFPN is a multi-scale feature propagation module that establishes more flexible information flows by adopting inter-level residual connections, cross-level dense connections, and feature re-weighting strategy. Leveraging the attention mechanism, SCP further augments the features by aggregating global spatial context into local regions. For each instance, HRoIE adaptively generates RoI features for different downstream tasks. Extensive evaluations of the proposed scheme on iSAID, DIOR, NWPU VHR-10, and HRSID datasets demonstrate that the proposed approach outperforms state-of-the-arts under similar computational costs. Source code and pre-trained models are available at https://github.com/yeliudev/CATNet.

I. INTRODUCTION

Remote-sensing instance segmentation is challenged by weakly discriminative object features, while methods developed for natural images can degrade on top-view imagery. CATNet addresses this with global context aggregation across feature, spatial, and instance domains.

I. INTRODUCTION

  • Remote-sensing imagery presents scale variation, arbitrary orientation, clustered distributions, low contrast, and cluttered backgrounds, which contribute to complicated object patterns or interfering background information.The paper attributes these challenges to insufficiently discriminative object features.Source: §I · p. 2, Figure 2 · p. 2
  • DenseFPN, SCP, and HRoIE propagate multi-scale features, aggregate global spatial context, and adaptively generate instance- and task-specific RoI features, respectively.DenseFPN uses inter-level residual and cross-level dense connections with feature re-weighting; SCP applies adaptive weights to global information; HRoIE fuses features hierarchically.Source: §I · p. 2
  • CATNet treats context in feature, spatial, and instance domains, extending prior context modeling beyond spatial dependencies.The paper describes this as the first work to consider global visual context beyond spatial dependencies.Source: §I · p. 2
  • On iSAID, DIOR, NWPU VHR-10, and HRSID, CATNet achieves new state-of-the-art performance at similar computational costs.The proposed scheme is tested on all four datasets.Source: §I · pp. 2–3

II. RELATED WORK

The related work surveys instance-segmentation approaches for remote sensing and multi-scale feature pyramids. It motivates DenseFPN as a more flexible way to aggregate information across feature levels.

A. Instance Segmentation in Remote Sensing Images

  • One-stage methods segment at image level and separate objects afterward, while two-stage methods generate proposals before task-specific processing; the latter usually achieve better localization and mask prediction.The paper notes that CATNet's context strategy can integrate with both schemes, while HRoIE is unnecessary for one-stage methods.Source: A. Instance Segmentation in Remote… · p. 3

B. Multi-Scale Feature Propagation

  • Feature-pyramid designs progressed from FPN's top-down propagation to added bottom-up, multi-path, and other cross-level fusion strategies.The paper argues that learned, flexible information flows may reduce feature confounding and handle multi-scale features more effectively, motivating DenseFPN.Source: B. Multi-Scale Feature Propagation · pp. 3–4

C. Global Context Modeling

The paper distinguishes global context modeling from spatial dependencies alone, then presents CATNet as a three-module pipeline for remote-sensing detection and instance segmentation.

C. Global Context Modeling

  • Prior global-context methods model long-range spatial correlations but leave feature- and instance-domain dependencies unmodeled.The related-work discussion notes that NLNet computes pixel-level pairwise correlations at high computational cost, motivating simplified alternatives.Source: C. Global Context Modeling · p. 4

III. THE PROPOSED APPROACH; A. Overview

  • CATNet uses DenseFPN, SCP, and HRoIE to aggregate context in feature, spatial, and instance domains, respectively.The framework is built on Mask R-CNN: DenseFPN and SCP enhance the feature pyramid, then HRoIE generates task-specific RoI features for each proposal.Source: §III · p. 4, A. Overview · p. 4

A. Overview

  • Instance segmentation outputs object bounding boxes, binary masks, and category labels for objects in an image.The paper represents each output as a tuple containing a box, mask, and class vector.Source: A. Overview · p. 4

B. Dense Feature Pyramid Network

DenseFPN propagates features across pyramid levels to form a more informative feature pyramid. Its blocks combine top-down and bottom-up paths with residual and dense connections and learned feature reweighting.

B. Dense Feature Pyramid Network

  • Multi-scale feature propagation aims to aggregate visual features from different backbone stages and propagate them among pyramid levels, producing an enhanced feature pyramid with more informative features for downstream tasks; each feature map has 1/2^i of the input image resolution.Source: B. Dense Feature Pyramid Network · p. 4
  • The module projects C2–C5 to 256 channels, adds C6 by strided convolution, and processes the resulting pyramid through stacked aggregation blocks.Source: B. Dense Feature Pyramid Network · p. 5
  • Each block combines top-down and bottom-up aggregation with inter-level residual connections, cross-level dense connections, and feature reweighting.Source: B. Dense Feature Pyramid Network · p. 5
  • Residual connections preserve original features and help prevent gradient vanishing, while the flexible architecture and reweighting optimize feature-domain context flow during training.Source: B. Dense Feature Pyramid Network · p. 5

C. Spatial Context Pyramid

SCP augments pyramid features with global spatial context at each level, using pixel-wise reweighting to limit unnecessary context aggregation. The paper reports semantic alignment in its aggregation weights and improved performance with a smaller model when SCP follows DenseFPN.

C. Spatial Context Pyramid

  • SCP adds global spatial context within each feature-pyramid level because multi-level aggregation alone leaves features spatially local.Source: C. Spatial Context Pyramid · p. 5
  • SCP adds a pixel-informativeness path to context aggregation, using soft reweighting to fuse global features while reducing information confounding.Source: C. Spatial Context Pyramid · pp. 5–6
  • SCP’s aggregation weights tend to connect semantically similar categories, such as small vehicles with large vehicles.Source: C. Spatial Context Pyramid · p. 7
  • Placing SCP after DenseFPN performs better and yields a smaller model than incorporating CABlock in the backbone, according to the authors.Source: C. Spatial Context Pyramid · p. 7

D. Hierarchical Region of Interest Extractor

HRoIE replaces hard single-scale RoI assignment with hierarchical, task-specific feature fusion for each instance. It adaptively combines features from multiple pyramid levels, using different fusion directions for detection and mask prediction.

D. Hierarchical Region of Interest Extractor

IV. EXPERIMENTS

The experiments evaluate CATNet on four remote-sensing datasets spanning optical and SAR imagery. DIOR supports object-detection evaluation only, and the iSAID validation table reports class-wise segmentation results using ResNet-50 backbones.

IV. EXPERIMENTS

  • Evaluation covers iSAID, DIOR, NWPU VHR-10, and HRSID; the first three are optical datasets and HRSID is SAR.Source: §IV · p. 8
  • DIOR has bounding-box annotations only, so the paper evaluates object detection rather than instance segmentation on that dataset.Source: §IV · p. 8
  • The iSAID validation results are class-wise instance-segmentation results, and all compared models use ResNet-50 backbones.Source: Table VI · p. 9

A. Datasets and Evaluation Metrics •

The evaluation spans four remote-sensing datasets with varied imagery and annotations. Performance is measured using mAP, with IoU thresholds differing for DIOR.

A. Datasets and Evaluation Metrics •

  • iSAID contains 15 classes and 655,451 instances across 2,806 images, which are divided into patches for benchmarking.The standard patches are 800 × 800 with stride 200; smaller 512 × 512 patches with stride 128 are used for detailed comparisons and ablations.Source: A. Datasets and Evaluation Metrics • · p. 9
  • DIOR contains 23,463 aerial images with 190,288 instances across 20 classes; NWPU VHR-10 has 800 aerial images in 10 categories and uses a random 70/30 train-test split.HRSID contains 5,604 high-resolution SAR images with 16,951 ship instances, all annotated with pixel-level masks.Source: A. Datasets and Evaluation Metrics • · p. 9
  • mAP measures detection and segmentation, averaging IoU thresholds from 0.05 to 0.95 except on DIOR, where only IoU 0.5 is used.Model efficiency is assessed with parameter count and FLOPs.Source: A. Datasets and Evaluation Metrics • · p. 9

B. Implementation Details

CATNet is evaluated against established baselines on iSAID, where it achieves state-of-the-art detection and instance-segmentation performance, including with a lighter backbone. The section also describes the implementation and test setup.

B. Implementation Details

  • Mask R-CNN, Faster R-CNN, and RetinaNet with ResNet-50 are the baselines, using ImageNet-pretrained backbones fine-tuned with detectors.Unless specified otherwise, DenseFPN modules contain five basic blocks.Source: B. Implementation Details · p. 9
  • For iSAID test benchmarking, training uses both training and validation sets under a 3× schedule; the validation tuning setup uses a 1× schedule.The 1× schedule stops at epoch 12, while the test benchmark follows the common 3× setting.Source: B. Implementation Details · p. 10

C. Instance Segmentation on iSAID Dataset

  • CATNet achieves state-of-the-art detection and instance-segmentation performance on the iSAID test set, even with a 40% lighter backbone.Multi-scale training further boosts performance; the results use either 1400 × 800 inputs or five short-side scales.Source: C. Instance Segmentation on iSAID… · p. 11
  • CATNet performs better on most iSAID categories, including storage tank, baseball diamond, and tennis court, and remains comparable on others.The comparison reports class-wise instance-segmentation results.Source: C. Instance Segmentation on iSAID… · p. 11

D. Object Detection on DIOR Dataset

CATNet outperforms prior methods across the reported DIOR, NWPU VHR-10, and HRSID evaluations. Visual analyses describe its context aggregation, convergence, and feature discrimination.

D. Object Detection on DIOR Dataset

  • On DIOR object detection, CATNet outperforms previous methods with and without augmentation, and ResNet-50 models exceed prior ResNet-101 state of the art.The comparison uses RetinaNet and Faster R-CNN; CATNet† omits HRoIE for one-stage detection.Source: D. Object Detection on DIOR Dataset · p. 11

E. Instance Segmentation on NWPU VHR-10 Dataset

  • On NWPU VHR-10, CATNet outperforms all previous instance-segmentation approaches, including those with heavier backbones.The reported comparison focuses on mask average precision.Source: E. Instance Segmentation on NWPU… · p. 11

F. Instance Segmentation on HRSID Dataset

  • On HRSID SAR imagery, CATNet performs significantly better than strong natural-image and optical-remote-sensing baselines for detection and instance segmentation.SAR images are repeated across three input channels.Source: F. Instance Segmentation on HRSID… · p. 11

G. Visualizations

  • SCP tends to aggregate spatial context from same-category objects and can also connect similar or semantically related categories.The comparison with GCNet notes that GCNet may fuse plane information into large vehicles, whereas SCP tends to aggregate small-vehicle information to large vehicles.Source: G. Visualizations · p. 12
  • Qualitative predictions show attention on object-containing areas and better handling of small, blurry, and clustered objects.The visualizations cover iSAID, DIOR, and HRSID scenes.Source: G. Visualizations · p. 12, Figure 10 · p. 12
  • CATNet converges about 30% faster than the baseline and reaches a 10% lower final loss.The loss comparison uses the 1× schedule and reports the last eight training epochs.Source: G. Visualizations · p. 12
  • Multi-scale context aggregation yields more isolated and cohesive feature clusters, while semantically related categories remain close.The feature comparison samples 1,000 objects per category and reports averaged class-wise variances.Source: G. Visualizations · p. 12, Figure 11 · p. 12

H. Detailed Comparisons and Ablation Study

Ablations find gains from each proposed module and the best results when they are combined. The comparisons also examine module choices, ordering, and inference efficiency.

H. Detailed Comparisons and Ablation Study

  • Across Mask R-CNN, Cascade R-CNN, Mask Scoring R-CNN, and PointRend, the modules steadily improve detection and instance segmentation.DenseFPN gives the largest mAP improvement; SCP and HRoIE add gains with few extra parameters.Source: H. Detailed Comparisons and Ablation… · pp. 12–13
  • DenseFPN outperforms representative feature-pyramid alternatives on detection and segmentation with fewer computational costs, and stacking blocks can further improve performance.A single-layer DenseFPN already brings gains with fewer or comparable computational costs.Source: H. Detailed Comparisons and Ablation… · p. 13
  • CABlock with re-weighted context outperforms GCNet across channel-reduction rates, and placing context modules after FPN yields gains with negligible computational costs.The reported design moves spatial-context modules from the backbone to layers after FPN, where input channels are reduced and feature-map semantics are more consistent.Source: H. Detailed Comparisons and Ablation… · p. 13
  • The best module order is DenseFPN before SCP, consistent with a coarse-to-fine sequence of feature-semantic calibration, spatial aggregation, and instance-specific refinement.The tested alternative orders perform less well on detection and instance segmentation.Source: H. Detailed Comparisons and Ablation… · p. 13
  • HRoIE adaptively fuses multi-layer RoI features for the different feature needs of detection and instance segmentation.Simple sums or concatenations of multi-layer RoI features bring only slight gains over cropping from one feature map.Source: H. Detailed Comparisons and Ablation… · p. 13
  • Combining DenseFPN, SCP, and HRoIE achieves the best results, with stable gains across tasks; the full model is about 20% slower, while the lite version is faster than baseline and performs better.The lite version uses a one-layer DenseFPN with 128 channels.Source: H. Detailed Comparisons and Ablation… · p. 13

V. CONCLUSION AND FUTURE WORK

CATNet combines three lightweight plug-and-play modules to aggregate global context across feature, spatial, and instance domains. The authors report that their collaboration enhances discriminative object features, while noting that each module aggregates context in only one domain.

  • CATNet uses DenseFPN, SCP, and HRoIE to aggregate global visual context in feature, spatial, and instance domains, respectively.Source: §V · p. 13
  • The authors report that collaboration among the modules enhances discriminative object features for object detection and instance segmentation.Source: §V · p. 13
  • The modules are designed as lightweight plug-and-play additions to existing instance segmentation pipelines.Source: §V · p. 13
  • Because each module aggregates context in only one domain, the approach may be suboptimal if cross-domain context is correlated; the authors propose unified frameworks and see potential in transformer-based models.Source: §V · p. 14

Related papers

Loading 2111.11057v4…