Source-linked AI summary
Hierarchical Multi-Scale Attention for Semantic Segmentation
Andrew Tao, Karan Sapra, Bryan Catanzaro
TL;DR
Semantic segmentation must combine predictions from multiple image scales because different scales resolve different failure modes, while averaging or max pooling can be suboptimal. The paper introduces hierarchical multi-scale attention and hard auto-labelling, achieving state-of-the-art results on Cityscapes and Mapillary Vistas.
Problem
Different inference scales are better suited to fine details or large structures, but averaging and max pooling can combine scales suboptimally.
Method
The paper uses hierarchical attention to learn relative pixel-level weighting between adjacent-scale predictions and hard auto-labelling for coarse Cityscapes images.
Results
The method achieves state-of-the-art results on Cityscapes and Mapillary Vistas, including 85.1 IOU and 61.1 IOU, respectively.
Takeaways & Limitations
Hierarchical multi-scale attention improves segmentation accuracy while remaining memory- and computationally efficient across Cityscapes and Mapillary.
Abstract
from arXiv · showhide
Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).
1 Introduction
Semantic segmentation faces a scale-dependent trade-off: higher-resolution inference helps fine details, while lower-resolution inference provides context for large structures. The paper proposes hierarchical attention to combine multi-scale predictions selectively and efficiently, alongside hard auto-labelling for Cityscapes.
- Scale-dependent segmentation failures: Higher inference resolutions better capture object edges and thin structures, whereas lower resolutions provide context that helps segment large structures and reduce class confusion.Figure 1 illustrates both cases: thin posts improve at 2.0x, while a large road/divider region improves at 0.5x.
- Limitations of standard multi-scale inference: Averaging multi-scale predictions can mix strong and weak predictions, while max-pooling selects one scale even when a weighted combination would be better.These limitations motivate learning pixel-level scale weights rather than combining scales uniformly or selecting only one.
- Hierarchical multi-scale attention: Hierarchical attention learns relative weights between adjacent scales, requiring only one extra training scale and permitting additional inference scales more flexibly than prior methods.The method is designed to reduce training requirements while adapting scale combinations at the pixel level.
- Cityscapes auto-labelling: Hard auto-labelling of coarse Cityscapes images increases dataset variance while reducing label storage and disk-I/O costs compared with soft labelling.The strategy uses thresholded labels to improve training throughput and support generalization.
- Reported outcomes: The paper reports state-of-the-art results of 85.1 IOU on Cityscapes and 61.1 IOU on Mapillary Vistas.The contribution summary identifies both benchmark results as state of the art for the respective datasets.
2 Related Work
Prior semantic segmentation systems use multi-scale context and inference to balance fine detail against global context, but common scale-combination methods are fixed or uniformly weighted. Related work also includes attention-based scale fusion and auto-labelling with hard thresholded labels for efficiency.
- Multi-scale context methods: Low-output-stride trunks preserve fine detail but shrink receptive fields, making large objects harder to predict; pyramid pooling adds multi-scale context to counter this.Pyramid pooling assembles features from multiple scales, but the passage describes this related approach only generally.
- Relational context methods: Relational context methods learn relationships between pixels, allowing context to adapt to image composition and non-square semantic regions rather than fixed square regions.The learned context can better reflect structures such as long trains or tall objects.
- Multi-scale inference: Multi-scale inference commonly combines predictions through averaging or max pooling, with average pooling equally weighting scales despite potentially sub-optimal results.This limitation motivates attention-based alternatives for combining predictions.
- Attention-based scale fusion: Prior attention methods learn scale-combination masks across fixed training scales, using either final-layer features or features from multiple network layers.The cited approaches differ in which features supply the attention mechanism.
- Auto-labelling: Cityscapes work often uses roughly 20,000 coarsely labelled images, but substantial portions of those images remain unlabelled because the annotations are coarse.This motivates auto-labelling strategies for exploiting the coarse data more fully.
- Auto-labelling: Hard auto-labelling assigns one top class per pixel using a threshold, reducing storage requirements and improving training speed compared with continuous soft labels.The paper generates hard dense labels for coarse Cityscapes images.
3 Hierarchical multi-scale attention
The method combines predictions from multiple image scales using relative attention between adjacent scales, chained hierarchically at inference. This design supports flexible scale selection and improves accuracy over averaging and explicit attention in the reported evaluation.
- 3 Hierarchical multi-scale attention: Hierarchical attention predicts relative pixel-wise weighting between adjacent scales rather than learning separate masks for every fixed scale.Training uses adjacent scale pairs, with semantic logits and attention masks produced by a shared network trunk.
- 3 Hierarchical multi-scale attention: Inference chains adjacent-scale attention so lower-scale attention determines the contribution of the next higher scale.The architecture combines multiple scales hierarchically instead of weighting all scales independently.
- 3 Hierarchical multi-scale attention: The method can add inference scales such as 0.25x or 2.0x without retraining, unlike approaches restricted to scales used during training.A model trained with 0.5x and 1.0x can be evaluated flexibly across additional scales.
- 3.2 Analysis: 51.6 accuracy was achieved with hierarchical attention, compared with 49.4 for averaging and 51.4 for explicit attention on the reported Mapillary evaluation.The comparison uses DeepLab V3+ with a ResNet-50 trunk.
- 3.2 Analysis: Adding 0.25x reduced averaging accuracy by 0.7 IOU but increased the proposed method’s accuracy by 0.6 IOU.The attention mechanism can avoid using coarse predictions around edges while leveraging lower scales for large regions.
- 3.2 Analysis: The architecture was also tested with attention features from both adjacent scales, but this produced no significant accuracy difference from using only the lower scale.The authors therefore retained a single feature set for the attention head.
4 Auto Labelling on Cityscapes
For Cityscapes, the authors use auto-labelling to improve the quality and effective size of supervision from coarse images. They choose thresholded hard labels to reduce storage and training-throughput costs.
- 4 Auto Labelling on Cityscapes: Cityscapes provides 20,000 coarsely labelled images alongside 3,500 finely labelled images, but the coarse labels contain many unlabelled pixels.The auto-labelling strategy is intended to improve coarse-label quality and increase effective dataset size.
- 4 Auto Labelling on Cityscapes: Soft auto-labels would require roughly 3.2TB of disk space for the Cityscapes coarse images, potentially slowing training through label I/O.The estimate covers per-pixel class probabilities for 20,000 images at 2048 × 1024 resolution and 19 classes.
- 4 Auto Labelling on Cityscapes: Hard labelling keeps a teacher’s top class only when its probability exceeds 0.9; otherwise, the pixel is assigned the ignore class.This thresholded strategy manages label storage size and supports training throughput by reducing disk I/O.
5 Results
Experiments evaluate the implementation on Cityscapes and Mapillary Vistas, including ablations, multi-scale attention, auto-labelling, and benchmark comparisons.
- Implementation Protocol: Cityscapes training uses HRNet-OCR with auto-labelled coarse images, while inference evaluates scales {0.5, 1.0, 2.0} with image flipping.The protocol also uses train + val images and samples from the auto-labelled pool with 50% probability.
- Cityscapes Ablation: 0.5% IOU comes from multi-scale attention, 1.1% IOU from auto-labelling, and combining both techniques yields a 1.4% IOU gain on Cityscapes validation.These are ablation improvements over the baseline HRNet-OCR architecture with average pooling.
- Results on Cityscapes: 85.1 IOU is achieved on the Cityscapes test set, beating the best previous score by 0.6 IOU.The paper reports this as the best reported Cityscapes test score and reports top per-class scores in all but three classes.
- Results on Mapillary Vistas: 61.1 is achieved on the Mapillary validation set by a single-model method, 2.4 higher than Panoptic Deeplab’s 58.7 ensemble result.Mapillary Vistas contains 25,000 high-resolution images annotated into 66 object categories.
6 Conclusion
The hierarchical multi-scale attention approach improves segmentation accuracy while addressing memory and computational efficiency. Experiments show consistent improvement on Cityscapes and Mapillary.
- The proposed approach improves semantic-segmentation accuracy while remaining memory- and computationally efficient.These efficiencies address practical constraints on research speed and trainable crop size.
- Training efficiency affects how quickly research can be conducted, while GPU memory efficiency affects the crop sizes networks can use.The latter can also limit network accuracy.
- The method shows consistent improvement on both Cityscapes and Mapillary.