Source-linked AI summary
DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation
Hanchao Li, Pengfei Xiong, Haoqiang Fan, Jian Sun
TL;DR
Real-time semantic segmentation must preserve accuracy and receptive field under tight computation limits, especially for high-resolution images. DFANet uses a lightweight Xception-based architecture with sub-network and sub-stage feature aggregation, achieving strong Cityscapes and CamVid performance with substantially reduced computation.
Problem
High-resolution semantic segmentation needs efficient inference without sacrificing accuracy, but existing designs incur high computation or lose spatial detail and feature communication.
Method
DFANet uses lightweight Xception backbones with cross-level aggregation that reuses high-level and stage-wise features to refine representations and expand receptive fields.
Results
71.3% Mean IOU is achieved on Cityscapes with 3.4G FLOPs at 1024×1024 input, while smaller configurations achieve 70.3% and 67.1% Mean IOU with 1.7G and 2.1G FLOPs.
Takeaways & Limitations
DFANet provides a lightweight feature-aggregation pipeline that balances segmentation speed, computation, receptive field, and accuracy on Cityscapes and CamVid.
Abstract
from arXiv · showhide
This paper introduces an extremely efficient CNN architecture named DFANet for semantic segmentation under resource constraints. Our proposed network starts from a single lightweight backbone and aggregates discriminative features through sub-network and sub-stage cascade respectively. Based on the multi-scale feature propagation, DFANet substantially reduces the number of parameters, but still obtains sufficient receptive field and enhances the model learning ability, which strikes a balance between the speed and segmentation performance. Experiments on Cityscapes and CamVid datasets demonstrate the superior performance of DFANet with 8$\times$ less FLOPs and 2$\times$ faster than the existing state-of-the-art real-time semantic segmentation methods while providing comparable accuracy. Specifically, it achieves 70.3\% Mean IOU on the Cityscapes test dataset with only 1.7 GFLOPs and a speed of 160 FPS on one NVIDIA Titan X card, and 71.3\% Mean IOU with 3.4 GFLOPs while inferring on a higher resolution image.
1. Introduction
DFANet addresses the tension between high-resolution segmentation accuracy and real-time efficiency by aggregating multi-level features through interconnected lightweight network components. It reports strong Cityscapes and CamVid results with substantially reduced computation.
- High-resolution semantic segmentation requires both efficient inference and high accuracy for applications such as autonomous driving, surveillance, and robot sensing.
- Existing real-time methods reduce computation but can lose boundary and small-object details, while multi-branch designs limit speed and feature communication.
- DFANet aggregates features across network and processing-stage levels to combine semantic information with structural details and strengthen feature representation.
- The architecture uses lightweight Xception backbones, sub-network and sub-stage aggregation, and a fully-connected attention layer with limited additional computation.
- 71.3% Mean IOU is achieved with 3.4G FLOPs at 1024×1024 input, while smaller configurations retain 70.3% and 67.1% Mean IOU with 1.7G and 2.1G FLOPs.
- The method is reported as up to 8× smaller in FLOPs and 2× faster than existing works while providing better accuracy.
2. Related Work
Related work reduces segmentation cost through lightweight backbones, multi-branch processing, spatial pyramid pooling, attention, and feature reuse. These approaches balance context, spatial detail, representation quality, and computation in different ways.
- Real-time Segmentation: Real-time segmentation methods use compact architectures, reduced downsampling, efficient spatial pyramid modules, and multi-scale inputs to limit computation.
- Depthwise Separable Convolution: Depthwise separable convolution combines depthwise and pointwise convolutions to reduce computation and parameters while maintaining similar or slightly better performance.
- High-level Features: Spatial pyramid pooling enriches multi-scale context and object representation but typically requires substantial computing resources.
- Structure Comparison: DFANet differs by reusing features at network and stage levels to enrich high-level context through feature communication rather than independent parallel branches.
- Context Encoding: Attention mechanisms perform channel-wise selection or context encoding to improve feature representation and per-pixel prediction.
- Feature Aggregation: Feature aggregation methods extend simple skip connections through refinement modules, dense connections, and other strategies for combining features between network blocks.
3. Deep Feature Aggregation Network
DFANet aggregates features across both subnetworks and stages to combine semantic context with spatial detail while retaining an efficient encoder-decoder design.
- Observations: The approach responds to real-time segmentation challenges in which high-resolution processing is costly and reducing computation can remove boundary and small-object details.The paper also identifies limited feature discriminativeness in shallow networks and weak communication between parallel branches.
- Sub-stage Aggregation: The design addresses single-path feature loss by delivering low-level spatial information to semantic understanding and enabling multi-stage context.The authors motivate this connection because deeper networks can lose spatial details, while parallel branches may lack information communication.
- Deep Feature Aggregation Network: DFANet combines sub-network and sub-stage aggregation to fuse features from different network depths and stages.The architecture uses cascaded backbones and same-resolution stage connections to enhance feature representation and information flow.
- Sub-network Aggregation: Sub-network aggregation stacks lightweight backbones so each subsequent encoder further processes the preceding backbone's high-level features.The paper describes this as a refinement process that reassesses higher-order spatial relationships.
- Sub-stage Aggregation: Sub-stage aggregation passes each stage's output from a previous subnetwork to the corresponding stage of the next subnetwork.Concatenation fuses features when corresponding stage outputs share the same resolution, preserving information across networks.
- Network Architecture: The encoder consists of three Xception backbones with both aggregation methods, while a simple bilinear-upsampling decoder fuses low- and high-level features.The decoder intentionally avoids a complicated design and directly combines selected representations for real-time inference.
4. Experiments
Experiments evaluate lightweight modified Xception backbones, feature aggregation, decoder choices, and speed–accuracy trade-offs on Cityscapes and CamVid. DFANet combines efficient computation with competitive segmentation accuracy through cascaded aggregation and simple decoding.
- Experimental Setup: Cityscapes and CamVid provide challenging high-resolution benchmarks for evaluating real-time semantic segmentation.Cityscapes images reach 2048×1024, while CamVid reaches 960×720.
- Lightweight Backbone Networks: Xception A reduces computation from 9.3G FLOPs for ResNet-50 to 1.6G FLOPs, but its standalone Cityscapes accuracy falls from 68.3% to 59.2%.Adding ASPP improves the lightweight backbone, but the paper notes that ASPP has high computational complexity.
- Lightweight Backbone Networks: FC attention improves Xception A and B accuracy by 4–6% with almost unchanged computation, supplying high-dimensional global context efficiently.The authors use Backbone A and B as basic units for constructing DFANet.
- Feature Aggregation: 66.3% accuracy is obtained with one sub-network aggregation, whereas repeating aggregation twice reduces accuracy to 65.1%, attributed to amplified noise.The authors associate the degradation with Backbone A x2 already having a receptive field larger than the image.
- Feature Aggregation: Sub-stage aggregation progressively smooths predictions, enlarges the receptive field, introduces global context, and restores structural details across stacked backbones.The third aggregation backbone combines detail and contextual information in the prediction.
- The Whole DFA Architecture: 71.9% mIoU is achieved on Cityscapes validation with 3.4 GFLOPs, while Backbone B reaches 68.4% mIoU at 2.1 GFLOPs.The decoder combines high- and low-level features with only a slight computational increase.
5. Conclusion
The paper proposes deep feature aggregation for real-time semantic segmentation on high-resolution images. Experiments on Cityscapes and CamVid evaluate the method’s effectiveness.
- 5. Conclusion: Deep feature aggregation connects convolution layers to refine high-level and low-level features without specially designed operations.The proposed approach targets real-time semantic segmentation on high-resolution images.
- 5. Conclusion: Quantitative and qualitative experiments on Cityscapes and CamVid demonstrate the effectiveness of the proposed method.