PaperlayerView exact revision on arXiv

Source-linked AI summary

DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation

Hanchao Li, Pengfei Xiong, Haoqiang Fan, Jian Sun

arXiv:1904.02216v1cs.CV

TL;DR

Real-time semantic segmentation must preserve accuracy and receptive field under tight computation limits, especially for high-resolution images. DFANet uses a lightweight Xception-based architecture with sub-network and sub-stage feature aggregation, achieving strong Cityscapes and CamVid performance with substantially reduced computation.

  • Problem

    High-resolution semantic segmentation needs efficient inference without sacrificing accuracy, but existing designs incur high computation or lose spatial detail and feature communication.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    DFANet uses lightweight Xception backbones with cross-level aggregation that reuses high-level and stage-wise features to refine representations and expand receptive fields.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    71.3% Mean IOU is achieved on Cityscapes with 3.4G FLOPs at 1024×1024 input, while smaller configurations achieve 70.3% and 67.1% Mean IOU with 1.7G and 2.1G FLOPs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DFANet provides a lightweight feature-aggregation pipeline that balances segmentation speed, computation, receptive field, and accuracy on Cityscapes and CamVid.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

This paper introduces an extremely efficient CNN architecture named DFANet for semantic segmentation under resource constraints. Our proposed network starts from a single lightweight backbone and aggregates discriminative features through sub-network and sub-stage cascade respectively. Based on the multi-scale feature propagation, DFANet substantially reduces the number of parameters, but still obtains sufficient receptive field and enhances the model learning ability, which strikes a balance between the speed and segmentation performance. Experiments on Cityscapes and CamVid datasets demonstrate the superior performance of DFANet with 8$\times$ less FLOPs and 2$\times$ faster than the existing state-of-the-art real-time semantic segmentation methods while providing comparable accuracy. Specifically, it achieves 70.3\% Mean IOU on the Cityscapes test dataset with only 1.7 GFLOPs and a speed of 160 FPS on one NVIDIA Titan X card, and 71.3\% Mean IOU with 3.4 GFLOPs while inferring on a higher resolution image.

1. Introduction

DFANet addresses the tension between high-resolution segmentation accuracy and real-time efficiency by aggregating multi-level features through interconnected lightweight network components. It reports strong Cityscapes and CamVid results with substantially reduced computation.

  • High-resolution semantic segmentation requires both efficient inference and high accuracy for applications such as autonomous driving, surveillance, and robot sensing.Source: §1 · p. 1
  • Existing real-time methods reduce computation but can lose boundary and small-object details, while multi-branch designs limit speed and feature communication.Source: §1 · p. 1
  • DFANet aggregates features across network and processing-stage levels to combine semantic information with structural details and strengthen feature representation.Source: §1 · p. 2
  • The architecture uses lightweight Xception backbones, sub-network and sub-stage aggregation, and a fully-connected attention layer with limited additional computation.Source: §1 · p. 2
  • 71.3% Mean IOU is achieved with 3.4G FLOPs at 1024×1024 input, while smaller configurations retain 70.3% and 67.1% Mean IOU with 1.7G and 2.1G FLOPs.Source: §1 · p. 2
  • The method is reported as up to 8× smaller in FLOPs and 2× faster than existing works while providing better accuracy.Source: §1 · p. 2

2. Related Work

Related work reduces segmentation cost through lightweight backbones, multi-branch processing, spatial pyramid pooling, attention, and feature reuse. These approaches balance context, spatial detail, representation quality, and computation in different ways.

Real-time Segmentation

  • Real-time segmentation methods use compact architectures, reduced downsampling, efficient spatial pyramid modules, and multi-scale inputs to limit computation.Source: §2 · p. 2

Depthwise Separable Convolution

  • Depthwise separable convolution combines depthwise and pointwise convolutions to reduce computation and parameters while maintaining similar or slightly better performance.Source: §2 · p. 2

High-level Features

  • Spatial pyramid pooling enriches multi-scale context and object representation but typically requires substantial computing resources.Source: §2 · pp. 2–3

Structure Comparison

  • DFANet differs by reusing features at network and stage levels to enrich high-level context through feature communication rather than independent parallel branches.Source: Figure 2 · p. 3

Context Encoding

  • Attention mechanisms perform channel-wise selection or context encoding to improve feature representation and per-pixel prediction.Source: §2 · p. 3

Feature Aggregation

  • Feature aggregation methods extend simple skip connections through refinement modules, dense connections, and other strategies for combining features between network blocks.Source: §2 · p. 3

3. Deep Feature Aggregation Network

DFANet aggregates features across both subnetworks and stages to combine semantic context with spatial detail while retaining an efficient encoder-decoder design.

Observations

  • The approach responds to real-time segmentation challenges in which high-resolution processing is costly and reducing computation can remove boundary and small-object details.The paper also identifies limited feature discriminativeness in shallow networks and weak communication between parallel branches.Source: §3.1 · p. 3

Sub-stage Aggregation

  • The design addresses single-path feature loss by delivering low-level spatial information to semantic understanding and enabling multi-stage context.The authors motivate this connection because deeper networks can lose spatial details, while parallel branches may lack information communication.Source: §3.1 · p. 4, §3.2 · p. 4

Deep Feature Aggregation Network

  • DFANet combines sub-network and sub-stage aggregation to fuse features from different network depths and stages.The architecture uses cascaded backbones and same-resolution stage connections to enhance feature representation and information flow.Source: §3.2 · p. 4, §3.1 · p. 4

Sub-network Aggregation

  • Sub-network aggregation stacks lightweight backbones so each subsequent encoder further processes the preceding backbone's high-level features.The paper describes this as a refinement process that reassesses higher-order spatial relationships.Source: §3.2 · p. 4

Sub-stage Aggregation

  • Sub-stage aggregation passes each stage's output from a previous subnetwork to the corresponding stage of the next subnetwork.Concatenation fuses features when corresponding stage outputs share the same resolution, preserving information across networks.Source: §3.2 · pp. 4–5

Network Architecture

  • The encoder consists of three Xception backbones with both aggregation methods, while a simple bilinear-upsampling decoder fuses low- and high-level features.The decoder intentionally avoids a complicated design and directly combines selected representations for real-time inference.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate lightweight modified Xception backbones, feature aggregation, decoder choices, and speed–accuracy trade-offs on Cityscapes and CamVid. DFANet combines efficient computation with competitive segmentation accuracy through cascaded aggregation and simple decoding.

Experimental Setup

  • Cityscapes and CamVid provide challenging high-resolution benchmarks for evaluating real-time semantic segmentation.Cityscapes images reach 2048×1024, while CamVid reaches 960×720.Source: §4 · p. 5, §4.1 · p. 5

Lightweight Backbone Networks

  • Xception A reduces computation from 9.3G FLOPs for ResNet-50 to 1.6G FLOPs, but its standalone Cityscapes accuracy falls from 68.3% to 59.2%.Adding ASPP improves the lightweight backbone, but the paper notes that ASPP has high computational complexity.Source: §4.1.1 · p. 6
  • FC attention improves Xception A and B accuracy by 4–6% with almost unchanged computation, supplying high-dimensional global context efficiently.The authors use Backbone A and B as basic units for constructing DFANet.Source: §4.1.1 · p. 6

Feature Aggregation

  • 66.3% accuracy is obtained with one sub-network aggregation, whereas repeating aggregation twice reduces accuracy to 65.1%, attributed to amplified noise.The authors associate the degradation with Backbone A x2 already having a receptive field larger than the image.Source: §4.1.2 · p. 6
  • Sub-stage aggregation progressively smooths predictions, enlarges the receptive field, introduces global context, and restores structural details across stacked backbones.The third aggregation backbone combines detail and contextual information in the prediction.Source: §4.1.2 · p. 7, §4.1.3 · p. 7

The Whole DFA Architecture

  • 71.9% mIoU is achieved on Cityscapes validation with 3.4 GFLOPs, while Backbone B reaches 68.4% mIoU at 2.1 GFLOPs.The decoder combines high- and low-level features with only a slight computational increase.Source: §4.1.3 · p. 7

5. Conclusion

The paper proposes deep feature aggregation for real-time semantic segmentation on high-resolution images. Experiments on Cityscapes and CamVid evaluate the method’s effectiveness.

5. Conclusion

  • Deep feature aggregation connects convolution layers to refine high-level and low-level features without specially designed operations.The proposed approach targets real-time semantic segmentation on high-resolution images.Source: §5 · p. 8
  • Quantitative and qualitative experiments on Cityscapes and CamVid demonstrate the effectiveness of the proposed method.Source: §5 · p. 8

Related papers

Loading 1904.02216v1…