Source-linked AI summary
Real-time Semantic Segmentation with Fast Attention
Ping Hu, Federico Perazzi, Fabian Caba Heilbron, Oliver Wang, Zhe Lin, Kate Saenko, Stan Sclaroff
TL;DR
Real-time semantic segmentation must combine rich spatial context with high-resolution detail without prohibitive computation. FANet addresses this through efficient fast attention, intermediate spatial reduction, and spatial-temporal extensions, achieving strong accuracy and speed across image and video benchmarks.
Problem
High-accuracy semantic segmentation requires rich context and high-resolution detail, but both increase computational cost and hinder real-time processing.
Method
FANet uses fast attention, intermediate spatial reduction, and spatial-temporal context aggregation for efficient image and video segmentation.
Results
FANet achieves accuracy comparable to state-of-the-art methods with the highest efficiency across Cityscapes, CamVid, and COCO-Stuff, including 75.5% mIoU at 58 fps on Cityscapes.
Takeaways & Limitations
The method supports real-time segmentation of high-resolution images and videos while retaining rich contextual information and spatial details.
Abstract
from arXiv · showhide
In deep CNN based models for semantic segmentation, high accuracy relies on rich spatial context (large receptive fields) and fine spatial details (high resolution), both of which incur high computational costs. In this paper, we propose a novel architecture that addresses both challenges and achieves state-of-the-art performance for semantic segmentation of high-resolution images and videos in real-time. The proposed architecture relies on our fast spatial attention, which is a simple yet efficient modification of the popular self-attention mechanism and captures the same rich spatial context at a small fraction of the computational cost, by changing the order of operations. Moreover, to efficiently process high-resolution input, we apply an additional spatial reduction to intermediate feature stages of the network with minimal loss in accuracy thanks to the use of the fast attention module to fuse features. We validate our method with a series of experiments, and show that results on multiple datasets demonstrate superior performance with better accuracy and speed compared to existing approaches for real-time semantic segmentation. On Cityscapes, our network achieves 74.4$\%$ mIoU at 72 FPS and 75.5$\%$ mIoU at 58 FPS on a single Titan X GPU, which is~$\sim$50$\%$ faster than the state-of-the-art while retaining the same accuracy.
I. INTRODUCTION
Real-time semantic segmentation must balance rich context and high-resolution detail against computational cost. FANet addresses this trade-off with fast attention, spatial reduction, and video extensions for efficient segmentation.
- Compact networks and low-resolution inputs accelerate prior methods but can reduce receptive fields, object boundaries, or small-object details.
- Fast attention changes the computation of self-attention to reduce cost while preserving contextual aggregation.The supplied passage states that it is c times more efficient than standard self-attention when n is much larger than c.
- Additional spatial reduction is applied to intermediate feature maps rather than input images, retaining high-resolution details while reducing computation.
- FANet introduces fast attention for non-local context aggregation and extends it to spatial-temporal context for video segmentation.
- FANet is presented as achieving state-of-the-art accuracy with higher efficiency for real-time image and video segmentation.
II. RELATED WORK
Prior segmentation work improves context through receptive-field expansion, multi-scale features, and self-attention, while efficiency methods trade capacity or detail for speed. Video methods similarly trade frame reuse efficiency against spatial alignment or temporal-context costs.
- Self-attention captures long-range dependencies but can incur intensive computation, motivating more efficient context-aggregation approaches.
- Bilinear feature-pooling methods approximate pixel affinity, whereas fast attention is described as an equivalent accurate-affinity formulation with efficient video feature reuse.
- Video segmentation methods either reuse keyframe features efficiently while risking spatial misalignment or capture temporal context at higher computational cost.
III. FAST ATTENTION NETWORK
The Fast Attention Network replaces quadratic self-attention computation with an associative, normalized formulation that aggregates spatial context more efficiently. Its architecture combines this module with spatial reduction to support real-time high-resolution segmentation.
- B. Network Architecture: FANet applies fast attention across feature stages and uses additional intermediate spatial reduction to process high-resolution inputs efficiently.
- A. Fast Attention Module: Self-attention forms weighted sums of features at all positions, but its O(n^2c) cost creates high computational and memory demands for high-resolution segmentation.
- A. Fast Attention Module: The module uses normalized cosine similarity and changes matrix-multiplication order so K_hat^T · V is computed before multiplication by Q_hat.
- A. Fast Attention Module: Fast attention has O(nc^2) complexity instead of self-attention's O(n^2c), reducing computation when spatial size n greatly exceeds channel size c.
- A. Fast Attention Module: Unlike channel attention, fast attention aggregates contextual information across the spatial domain rather than only across channels at each pixel.
B. Network Architecture
FANet uses an encoder-decoder architecture with a lightweight ResNet-18 encoder, stage-wise fast attention, and progressively merged features for prediction.
- B. Network Architecture: FANet combines an encoder, context-aggregation module, and decoder in an encoder-decoder architecture.
- B. Network Architecture: A ResNet18 backbone extracts feature maps at multiple semantic levels, with successive blocks downsampling spatial resolution.
- B. Network Architecture: Fast attention modules enhance feature maps at each stage before the network merges features from deep to shallow levels for prediction.
C. Extra Spatial Reduction for Real-time Speed
FANet preserves high-resolution spatial details while reducing computation by down-sampling intermediate feature stages instead of the input images.
- High-resolution inputs improve accuracy through spatial details, whereas smaller inputs reduce computation but worsen results by losing those details.
- Additional down-sampling of intermediate feature stages reduces computation while allowing lower layers to learn from high-resolution spatial details.
- The strategy supports real-time efficiency and effectiveness with full-resolution input.
D. Extending to Video Semantic Segmentation
The paper extends fast attention to aggregate spatial-temporal context for video segmentation, reusing computations from previous frames so the cost remains independent of the temporal window length.
- Video segmentation benefits from temporal context, and the spatial-temporal extension improves video accuracy without increasing computational costs.
- Traditional self-attention over a t-frame window has complexity O(tn^2c), which is t times higher than single-frame spatial attention.
- Fast spatial-temporal attention reuses previously computed terms, adds the current frame’s term, and multiplies by the target-frame query.
- The spatial-temporal fast-attention module has complexity independent of t, matching single-frame fast attention and avoiding increased computation with longer temporal ranges.
- The method sequentially segments video frames using features enhanced with spatial-temporal context.
A. Datasets and Evaluation
The evaluation uses several semantic-segmentation datasets and compares efficiency through controlled implementation settings, including visualizing spatial-temporal attention and measuring computational cost.
- The evaluation includes a visualization of fast spatial-temporal attention and a GFLOPs comparison between the non-local and fast-attention modules.
- CamVid has 11 classes with 367, 101, and 233 annotated frames for training, validation, and testing, respectively.
- COCO-Stuff provides 9,000 training and 1,000 testing images, evaluated at 640×640 resolution across 182 classes.
- The experiments use ResNet-18/34 encoders pretrained on ImageNet and train the remaining modules with mini-batch SGD.
C. Method Analysis
FANet’s method analysis shows that fast attention substantially reduces computation while preserving segmentation quality, and that spatial-reduction choices strongly affect the accuracy–speed trade-off.
- Fast Attention: More than 94% less computation is achieved by fast attention than by standard self-attention across different input-feature sizes.
- Fast Attention: 75.0% mIoU is achieved with L2-normalized fast attention, compared with 74.1% without L2 normalization on Cityscapes validation.The authors attribute this difference to bounded cosine-similarity affinities.
- Fast Attention: 2.4% mIoU improvement from original self-attention reduces speed from 83 fps to 8 fps, whereas fast attention provides slightly lower quality at much lower computation cost.
- Spatial Reduction: Down-sampling the input reduces computation but loses critical spatial details, while extra reduction at Res-4 performs better than applying no additional down-sampling.
- Spatial Reduction: 75.0% mIoU is obtained with stride-convolution down-sampling, versus 72.9% with average pooling and 74.2% with max pooling.The paper suggests stride convolution preserves spatial details while maintaining sizable receptive fields.
D. Image Semantic Segmentation
On Cityscapes, FANet achieves state-of-the-art real-time image segmentation accuracy with high throughput, including a faster lightweight variant and a stronger full-resolution model.
- Cityscapes: 75.5% test mIoU at 58 fps is achieved by FANet-34 on full-resolution 1024×2048 Cityscapes inputs.Its validation mIoU is 76.3%.
- Cityscapes: 74.4% test mIoU at 72 fps is achieved by FANet18, nearly twice the speed of recent methods such as ShelfNet and SwiftNet.Its validation mIoU is 75.0%.
- Cityscapes: The evaluation compares FANet with recent state-of-the-art real-time methods on Cityscapes, with GFLOPs reported for 1-million-pixel inputs.
- Cityscapes: FANet-34 achieves better accuracy and speed than ICNet, while FANet-18 further accelerates inference with comparable mIoU.
E. Video Semantic Segmentation
For video semantic segmentation on Cityscapes, FANet combines spatial-temporal context with low-latency processing and outperforms prior methods in the reported comparisons. Its temporal variants achieve state-of-the-art accuracy at substantially faster speeds than competing approaches.
- Video Semantic Segmentation: The method captures both spatial and temporal context without significantly increasing computational cost, improving accuracy while reducing latency.The evaluation compares FANet with recent state-of-the-art video semantic segmentation approaches on Cityscapes.
- Video Semantic Segmentation: FANet18+Temp and FANet34+Temp achieve state-of-the-art accuracy with much faster speed than prior video segmentation methods on Cityscapes.FANet18+Temp is reported as more than 200× more efficient than Netwarp, while FANet34+Temp is 40× faster than PEARL.
- Video Semantic Segmentation: More than 200× better efficiency is reported for FANet18+Temp than Netwarp, while FANet34+Temp is 40× faster than PEARL.
- Video Semantic Segmentation: The proposed network extends fast attention to spatial-temporal context and uses spatial reduction at intermediate feature stages for efficient high-resolution processing.These design choices are presented as enabling rich contextual information and high-resolution details while maintaining real-time speed.