PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymmetric Non-local Neural Networks for Semantic Segmentation

Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, Xiang Bai

arXiv:1908.07678v5cs.CVcs.LG

TL;DR

Semantic segmentation benefits from long-range dependencies, but standard non-local modules impose prohibitive computation and GPU-memory costs. The paper introduces APNB and AFNB, using pyramid sampling for efficient non-local modeling and cross-level feature fusion, and reports state-of-the-art benchmark performance with substantial APNB efficiency gains.

  • Problem

    Long-range dependencies can improve semantic segmentation, but standard non-local blocks are prohibitively expensive in computation and GPU memory.

  • Method

    The network combines APNB, which uses pyramid sampling to reduce non-local computation, with AFNB, which fuses features from different network levels through long-range correlations.

  • Results

    The method reports state-of-the-art performance of 81.3%, 45.24% and 52.8% on Cityscapes, ADE20K and PASCAL Context, respectively.

  • Takeaways & Limitations

    APNB is around 6 times faster and 28 times smaller in GPU running memory occupation than a non-local block for a 256 × 128 input.

Abstract

from arXiv · show

The non-local module works as a particularly useful technique for semantic segmentation while criticized for its prohibitive computation and GPU memory occupation. In this paper, we present Asymmetric Non-local Neural Network to semantic segmentation, which has two prominent components: Asymmetric Pyramid Non-local Block (APNB) and Asymmetric Fusion Non-local Block (AFNB). APNB leverages a pyramid sampling module into the non-local block to largely reduce the computation and memory consumption without sacrificing the performance. AFNB is adapted from APNB to fuse the features of different levels under a sufficient consideration of long range dependencies and thus considerably improves the performance. Extensive experiments on semantic segmentation benchmarks demonstrate the effectiveness and efficiency of our work. In particular, we report the state-of-the-art performance of 81.3 mIoU on the Cityscapes test set. For a 256x128 input, APNB is around 6 times faster than a non-local block on GPU while 28 times smaller in GPU running memory occupation. Code is available at: https://github.com/MendelXu/ANN.git.

1. Introduction

Semantic segmentation needs long-range context, but standard non-local blocks are computationally and memory intensive. The proposed asymmetric network uses APNB for efficient context modeling and AFNB for cross-level feature fusion, achieving strong benchmark performance.

  • Motivation: Semantic segmentation remains challenging because accurate pixel-wise labeling requires stronger contextual modeling than current methods provide.The task supports applications including autonomous driving and medical diagnosis.
  • Motivation: Long-range dependencies can improve segmentation, but convolution-only models have limited receptive fields and enlarging them is computationally expensive.The paper therefore considers global operations such as non-local means and spatial pyramid pooling.
  • Motivation: Standard non-local blocks compute pairwise interactions across all N = H · W locations, causing O(CH^2W^2) computation and high GPU memory use.The large spatial resolution typical of segmentation makes these matrix multiplications the primary efficiency bottleneck.
  • Proposed Method: APNB embeds pyramid sampling into non-local blocks, replacing the large key and value representations with fewer representative points while retaining semantic statistics.This reduces matrix-multiplication cost and is reported to outperform simpler sub-sampling strategies.
  • Proposed Method: AFNB adapts APNB to fuse features from different network stages using long-range correlations between low-level and high-level feature maps.The complete network integrates APNB and AFNB into a ResNet-FCN model.
  • Results: 81.3%, 45.24% and 52.8% are reported as state-of-the-art performance on Cityscapes, ADE20K and PASCAL Context, respectively.For a 256 × 128 input, APNB is around 6 times faster and uses 28 times less GPU running memory than a non-local block.

2. Related Work

Prior semantic-segmentation research explores context through encoder-decoder designs, CRFs, specialized convolutions, spatial pyramids, and non-local networks. This work combines pyramid sampling with non-local blocks to capture multi-scale semantic statistics efficiently while maintaining performance.

  • Overview: Related segmentation methods are organized around five directions for exploiting context information.The supplied passages describe encoder-decoder architectures, CRFs, different convolutions, spatial pyramid pooling, and non-local networks.
  • Encoder-Decoder: Encoder-decoder models reduce spatial resolution to enlarge receptive fields, then decode predictions while skip connections restore spatial detail.Skip connections bridge encoding and decoding features to enrich segmentation outputs.
  • CRF: CRF-based approaches combine conditional random fields with CNNs to model context in end-to-end segmentation networks.Gaussian conditional random fields are also integrated into CNNs.
  • Different Convolutions: Dilated and large-kernel convolutions expand contextual coverage, addressing global semantics or the classification-localization trade-off.These approaches modify convolutional receptive fields rather than using non-local operations.
  • Spatial Pyramid Pooling: Spatial pyramid methods use multiple sampling scales to represent multi-scale context, including ASPP and PSPNet-style designs.These methods explicitly account for multiple scales through differently weighted operations or pooling regions.
  • Non-local Network: Non-local networks capture long-range dependencies for segmentation, but this work uniquely combines non-local blocks with pyramid sampling under a smaller computation budget.The stated goal is to preserve the performance of original non-local modules while improving efficiency.

3. Asymmetric Non-local Neural Network

The proposed network makes non-local reasoning more efficient through asymmetric sampling and uses long-range dependencies to fuse multi-level features for semantic segmentation.

  • Overview: APNB and AFNB are the two main components: APNB reduces non-local computation, while AFNB fuses multi-level features using long-range dependencies.APNB targets efficiency; AFNB targets improved learning capacity and segmentation performance.
  • 3.1. Revisiting Non-local Block: Standard non-local blocks are costly because two matrix multiplications require O(ĈN^2) time, with N increasing at high-resolution segmentation outputs.For example, N can equal 96 × 96 = 9216 during training.
  • 3.2. Asymmetric Pyramid Non-local Block: APNB samples S representative anchor points from the key and value branches, preserving output size while replacing dense interactions with asymmetric computation.The sampled anchors are θP ∈ RĈ×S and γP ∈ RĈ×S, where S is much smaller than N.
  • 3.2. Asymmetric Pyramid Non-local Block: APNB’s asymmetric matrix multiplication has complexity O(ĈNS), significantly lower than the O(ĈN^2) complexity of a standard non-local block.For H = 128 and W = 256, the paper reports approximately 298 times less computation for the asymmetric multiplication.
  • 3.2. Asymmetric Pyramid Non-local Block: Pyramid pooling after θ and γ uses multiple output scales to provide global scene statistics while reducing the matrix-multiplication burden.The pooling widths are set to n ∈ {1, 3, 6, 8}, and the pooling results are flattened and concatenated.
  • 3.3. Asymmetric Fusion Non-local Block: AFNB extends the fusion non-local block by combining high-level and low-level feature maps through long-range dependencies, then applies pyramid sampling for efficiency.The resulting output reflects selected contributions from all locations in the low-level feature map.

4. Experiments

Experiments evaluate the method across multiple semantic segmentation benchmarks, efficiency settings, and ablations. The results show strong accuracy alongside reduced computational and memory demands from APNB and performance gains from combining APNB with AFNB.

  • Evaluation settings: The method is evaluated on Cityscapes, ADE20K, and PASCAL Context using mean IoU as the evaluation metric.The study also reports supplementary results on NYUD-V2 and COCO-Stuff-10K.
  • Benchmark performance: 81.3% on Cityscapes, 45.24% on ADE20K, and 52.8% on PASCAL Context are reported as state-of-the-art results.On Cityscapes, the model is also reported to reduce local inconsistency on large objects and improve performance on slim objects relative to DeepLab-V3 and PSPNet.
  • APNB and AFNB ablations: Adding a non-local block improves the baseline from 75.8% to 78.4%, while replacing it with APNB yields 78.6%.This ablation isolates the contribution of non-local modeling and the efficiency-oriented APNB replacement.
  • APNB and AFNB ablations: AFNB reduces computation with a minor performance decrease from 77.3% to 77.1% compared with FNB.Common fusion improves the baseline from 75.8% to 76.5%, and FNB further increases it to 77.3%.
  • Ablation and sampling studies: The full APNB-plus-AFNB model reaches 79.9%, while more anchor points improve performance at the cost of increased computation.Average sampling performs better than max and random sampling in the APNB study.

5. Conclusion

The paper concludes that asymmetric non-local blocks improve semantic segmentation efficiency and performance. APNB reduces the cost of non-local modeling, while AFNB fuses multilevel features using long-range spatial relevance.

  • Conclusion: The asymmetric pyramid non-local block improves efficiency and reduces memory consumption without sacrificing performance.The paper proposes APNB as the core efficiency contribution for non-local neural blocks.
  • Conclusion: The asymmetric fusion non-local block fuses features from different levels while exploring long-range spatial relevance.The conclusion attributes considerable performance improvement over a strong baseline to AFNB.
  • Conclusion: Experiments on Cityscapes, ADE20K, and PASCAL Context report new state-of-the-art performance.The paper identifies applying asymmetric non-local networks to other vision tasks as future work.

A. Quantitative comparisons on COCO-Stuff-10K and NYUD-V2

Additional experiments evaluate the method on COCO-Stuff-10K and NYUD-V2 under single-scale whole-image testing. The reported results are competitive on both datasets.

  • Quantitative comparisons: The method achieves competitive results on COCO-Stuff-10K and NYUD-V2 using single-scale whole-image testing.The protocols follow DeepLab-V2 and RefineNet, with results reported in Table 8.
  • Quantitative comparisons: The experiments assess effectiveness on both a small benchmark, NYUD-V2, and a large, challenging benchmark, COCO-Stuff-10K.The passage characterizes NYUD-V2 as small and COCO-Stuff-10K as challenging and large.

B. More ablation results

Further ablations examine multilevel fusion and the complementarity of APNB and AFNB. The results support fusing later network stages and combining both asymmetric blocks.

  • Qualitative comparisons: The full model shows the best semantic consistency and fewest inconsistency artifacts among the compared variants.The passage reports that +AFNB and +APNB fail in some cases where the full model performs best.
  • Qualitative comparisons: APNB and AFNB are complementary, and their combination improves performance over either component alone.The qualitative comparison identifies the full combination as beneficial for reducing inconsistency artifacts.
  • Selection of fusing layers: The analysis fuses shallow features into deep features in a top-down manner, focusing on the last several layers because they contain semantic information.The selected design combines features from shallow to deep layers rather than testing arbitrary combinations.
  • Selection of fusing layers: Feature visualizations are similar across segmentation and classification networks in the first three stages but differ substantially in the last two.This observation partially supports fusing only the last two stages.

C. Discussions

The discussion reports practical sensitivity to hardware, training choices, and AFNB design, while noting APNB’s stability across architectures. It also frames the supplementary experiences as limited references rather than comprehensive guidance.

  • Practical considerations: The reported experiences are not as thorough as the main draft and may not suit every semantic-segmentation setting.The authors present them as references for interested readers rather than universally applicable guidance.
  • Practical considerations: 0.5 mIoU separates the same model trained on 8 × Titan V from training on 8 × Titan Xp in the authors’ case.The passage attributes the difference to graphics cards, PyTorch, CUDA, and NVIDIA driver versions potentially influencing performance.
  • Architecture: AFNB requires careful architectural modifications, whereas APNB is reported as quite stable across varied architectures.Adding batch normalization after Wo helped AFNB alone but did not necessarily help the Full model.
  • Training: Learning-rate configuration strongly affects segmentation performance, with a substantial boost observed during the last quarter of training iterations.The authors recommend consulting published methods with similar architectures when choosing learning-rate configurations.
  • Training: More training iterations do not always improve performance because changing iteration counts also changes the learning-rate schedule under poly decay.The observation concerns otherwise identical configurations.
Loading 1908.07678v5…