PaperlayerView exact revision on arXiv

Source-linked AI summary

AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing

Qi Song, Kangfu Mei, Rui Huang

arXiv:2103.05930v1cs.CV

TL;DR

Semantic segmentation needs global context and multi-level semantics, but capturing both can be computationally expensive for real-time scenarios. AttaNet combines strip attention with attention-based feature fusion to pursue both efficiently. It reports speed/accuracy trade-offs on Cityscapes and leading performance on Cityscapes and ADE20K.

  • Problem

    Capturing global context and multi-level semantics is computationally expensive, although both are important for segmentation accuracy and real-time applications.

    Source: Introduction · p. 1

  • Method

    AttaNet combines strip attention for efficient long-range context with attention-weighted cross-level feature fusion.

    Source: Introduction · p. 2, Method · p. 3

  • Results

    AttaNet reports leading performance on Cityscapes and ADE20K, including 79.9% mIoU at 71 FPS on Cityscapes.

    Source: Introduction · p. 2, Conclusions · p. 7

  • Takeaways & Limitations

    AttaNet achieves different speed/accuracy trade-offs on Cityscapes while also reporting outstanding trade-offs on ADE20K.

    Source: Introduction · p. 2, Conclusions · p. 7

Abstract

from arXiv · show

Two factors have proven to be very important to the performance of semantic segmentation models: global context and multi-level semantics. However, generating features that capture both factors always leads to high computational complexity, which is problematic in real-time scenarios. In this paper, we propose a new model, called Attention-Augmented Network (AttaNet), to capture both global context and multilevel semantics while keeping the efficiency high. AttaNet consists of two primary modules: Strip Attention Module (SAM) and Attention Fusion Module (AFM). Viewing that in challenging images with low segmentation accuracy, there are a significantly larger amount of vertical strip areas than horizontal ones, SAM utilizes a striping operation to reduce the complexity of encoding global context in the vertical direction drastically while keeping most of contextual information, compared to the non-local approaches. Moreover, AFM follows a cross-level aggregation strategy to limit the computation, and adopts an attention strategy to weight the importance of different levels of features at each pixel when fusing them, obtaining an efficient multi-level representation. We have conducted extensive experiments on two semantic segmentation benchmarks, and our network achieves different levels of speed/accuracy trade-offs on Cityscapes, e.g., 71 FPS/79.9% mIoU, 130 FPS/78.5% mIoU, and 180 FPS/70.1% mIoU, and leading performance on ADE20K as well.

Introduction

Semantic segmentation needs global context and multi-level semantics for accuracy, but capturing both can be computationally costly for real-time applications. AttaNet introduces SAM and AFM to pursue that combination efficiently, with reported speed/accuracy trade-offs on Cityscapes and leading performance on ADE20K.

Introduction

  • Capturing global context and multi-level semantics is computationally expensive, complicating real-time segmentation despite their importance for accuracy.The paper notes that global context is often captured with heavy networks and multi-level representations combine high-level semantics with low-level spatial details.Source: Introduction · p. 1
  • AttaNet combines global context and multi-level representations while aiming to retain computational efficiency for real-time segmentation.The paper reports experiments on Cityscapes and ADE20K and top performance on both.Source: Introduction · pp. 1–2
  • Non-local and position-attention approaches use O((H × W) × (H × W)) computation, while CCNet reduces complexity but still has too much overhead for real-time use.AttaNet’s strip attention is introduced to address this efficiency challenge.Source: Introduction · pp. 1–2
  • SAM introduces vertical striping to reduce self-attention complexity to O((H × W) × W) while encoding global context and harvesting horizontal long-range relations.The authors motivate this design with the prevalence of vertical strip areas in low-accuracy classes and note that horizontal striping is also possible.Source: Introduction · p. 2
  • AFM uses cross-level aggregation and pixel-wise attention to weight multi-level features, applying fusion only between the backbone’s last two stages to limit computation.The design addresses differences between high-level semantic features and low-level spatial details.Source: Introduction · p. 2
  • 79.9% mIoU at 71 FPS, 78.5% mIoU at 130 FPS, and 70.1% mIoU at 180 FPS on the Cityscapes test set illustrate AttaNet’s speed/accuracy trade-offs.The authors also report leading performance on Cityscapes and ADE20K.Source: Introduction · p. 2

Related Work

Prior work seeks efficient long-range context, effective multi-level feature fusion, and fast inference through architectural compromises. AttaNet’s related-work discussion positions SAM and AFM as alternatives that target those efficiency and representation challenges.

Self-attention Model.

  • Self-attention methods such as OCNet and DANet model long-range dependencies but incur overhead from large attention maps; SAM uses directional striping to reduce affinity-map size.CCNet creates sparse connections, while ACFNet exploits class-level context along channel dimensions.Source: Self-attention Model · p. 2

Multi-level Feature Fusion.

  • Multi-branch and cross-level fusion approaches combine multi-level representations, but may ignore feature representation gaps or fail to limit computation; AFM uses lightweight attention to bridge the gap.The passage contrasts these approaches with AFM’s emphasis on adaptability and efficiency.Source: Multi-level Feature Fusion · p. 2

Real-time Segmentation.

  • Real-time methods use multi-resolution inputs, detail and semantic branches, or lightweight backbones; AttaNet instead aims to use large backbones while reserving semantic and spatial information.Figure 2 presents the overall architecture with ResNet18 as an example backbone.Source: Real-time Segmentation · pp. 2–3, Figure 2 · p. 3

Method

AttaNet combines Strip Attention Module (SAM) and Attention Fusion Module (AFM) in a cross-level aggregation network to capture long-range context and fuse multi-level features efficiently. SAM uses striping and horizontal affinity, while AFM uses attention to guide adjacent-level feature fusion.

Method

  • AttaNet uses a cross-level aggregation architecture with SAM for efficient long-range relations and AFM for feature aggregation.The network uses a pretrained ResNet backbone by default, though other CNN backbones can also be chosen.Source: Method · p. 3

Network Architecture

  • Deep supervision applies the principal loss to the network output and auxiliary losses to res3 and AFM outputs, balanced by λ.The joint loss uses cross-entropy losses, with K = 2 and λ = 1 in the implementation.Source: Network Architecture · p. 3

Strip Attention Module

  • SAM strips feature K with H × 1 average pooling, then computes horizontal affinities to capture banded context and long-range dependencies.The authors motivate vertical striping by the prevalence of vertical strip areas in their natural images; SAM adds few parameters and uses little GPU memory.Source: Strip Attention Module · pp. 3–4

Attention Fusion Module

  • AFM upsamples one feature map, predicts a relative attention mask for adjacent features, and fuses their pixel-wise weighted responses.The mask guides each adjacent feature’s response, bridging the semantic and resolution gap compared with simple combination.Source: Attention Fusion Module · p. 4

Experiments

Experiments on Cityscapes and ADE20K report leading AttaNet performance on both benchmarks. The evaluation includes Cityscapes comparisons and ablations, followed by ADE20K results.

Experiments

  • AttaNet obtains leading performance on both Cityscapes and ADE20K.The experiments include a series of Cityscapes comparisons and ablations and report results on ADE20K.Source: Experiments · p. 4
  • The evaluation uses the Cityscapes and ADE20K datasets.Source: Experiments · p. 4
  • The Cityscapes experiments include comparisons and ablation experiments, with ADE20K results reported afterward.Source: Experiments · p. 4

Datasets

Cityscapes and ADE20K provide the two scene-parsing benchmarks used in the experiments, with different dataset sizes and annotation scopes.

Cityscapes.

  • Cityscapes has 5,000 fine-annotated and 20,000 coarsely annotated urban images, with 19 semantic classes at 1024 × 2048 resolution.The fine-annotation split contains 2,975 training, 500 validation, and 1,525 test images.Source: Cityscapes · p. 4

ADE20K.

  • ADE20K contains 20K training and 2K validation images densely labeled across 150 stuff/object categories, drawn from varied scenes with more scale variations.Source: ADE20K · p. 5

Implementation Details

The implementation uses a pretrained ResNet18 backbone, trains with dataset-specific settings, and evaluates full images using segmentation-accuracy and speed metrics.

Implementation Details

  • ImageNet-pretrained ResNet18 supplies the backbone, with feature fusion between the res4 and res5 block outputs.Source: Implementation Details · p. 5

Training Settings.

  • Training uses SGD with batch sizes of 16 for Cityscapes and 32 for ADE20K, plus augmentation and dataset-specific crop sizes and iteration counts.Both datasets use momentum 0.9, weight decay 5e(−4), and a poly learning-rate policy; training runs 200k iterations for Cityscapes and 250K for ADE20K.Source: Training Settings · p. 5

Inference.

  • Inference uses a full image resized following BiSeNetV2, and evaluation reports mIOU for accuracy and FPS, FLOPs, and parameter count for speed.Source: Inference · p. 5

Experiments on Cityscapes

On Cityscapes, AttaNet reports leading speed/accuracy trade-offs, gains from both proposed modules, and improvements across varied backbones with modest computational cost.

Comparisons with state of the art.

  • 70.1% mIoU at 180 FPS, 78.5% mIoU at 130 FPS, and 79.9% mIoU at 71 FPS are reported for AttaNet variants on the Cityscapes test set.The 180 FPS model uses eight convolution layers; the 130 FPS and 71 FPS models use ResNet18 and DF2, respectively.Source: Table 1 · p. 5, Comparisons with state of the art · p. 5

Ablation study on proposed modules.

Robustness on different backbones.

  • AttaNet achieves considerably better mIoU on both lightweight and heavy backbones with only slightly increased computational cost.The tested backbones include ShuffleNetV2, DF1, DF2, ResNet50, and ResNet101.Source: Robustness on different backbones · p. 6

Experiments on ADE20K

On ADE20K, AttaNet is compared with state-of-the-art models on the validation set and reports two mIoU results with much less computation.

Experiments on ADE20K

Conclusions

AttaNet combines efficient global-context modeling with attention-based fusion of multi-level features, and reports strong speed/accuracy trade-offs on Cityscapes and ADE20K.

Conclusions

  • AttaNet achieves outstanding speed/accuracy trade-offs on Cityscapes and ADE20K.Source: Conclusions · p. 7
  • The Strip Attention Module uses striping to reduce self-attention computation while exploiting long-range dependencies among pixels.Source: Conclusions · p. 7
  • The Attention Fusion Module uses attention to let each pixel select contextual information from multi-level features efficiently.Source: Conclusions · p. 7
  • AttaNet targets real-time semantic segmentation by combining global context and multi-level representations with computational efficiency.Source: Conclusions · p. 7

Related papers

Loading 2103.05930v1…