Source-linked AI summary
Adaptive Context Network for Scene Parsing
Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, Hanqing Lu
TL;DR
Scene parsing methods commonly apply contextual information uniformly, although pixels can require different balances of global and local context. ACNet adapts that balance through pixel-aware fusion, and the paper reports state-of-the-art performance across scene-parsing datasets.
Problem
Scene-parsing methods often apply context uniformly despite differing pixel-level needs for global and local information.
Source: §1 · pp. 1–2Method
ACNet competitively fuses global and local features according to pixel-specific context demands, using adaptive context blocks at different network levels.
Source: §1 · p. 2Results
Takeaways & Limitations
Global cues help large semantic regions, while local cues support sharp boundaries and details; ACNet combines both adaptively for scene parsing.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Recent works attempt to improve scene parsing performance by exploring different levels of contexts, and typically train a well-designed convolutional network to exploit useful contexts across all pixels equally. However, in this paper, we find that the context demands are varying from different pixels or regions in each image. Based on this observation, we propose an Adaptive Context Network (ACNet) to capture the pixel-aware contexts by a competitive fusion of global context and local context according to different per-pixel demands. Specifically, when given a pixel, the global context demand is measured by the similarity between the global feature and its local feature, whose reverse value can be used to measure the local context demand. We model the two demand measurements by the proposed global context module and local context module, respectively, to generate adaptive contextual features. Furthermore, we import multiple such modules to build several adaptive context blocks in different levels of network to obtain a coarse-to-fine result. Finally, comprehensive experimental evaluations demonstrate the effectiveness of the proposed ACNet, and new state-of-the-arts performances are achieved on all four public datasets, i.e. Cityscapes, ADE20K, PASCAL Context, and COCO Stuff.
1. Introduction
Scene parsing with FCNs can lose spatial detail and misclassify pixels when context is limited or applied uniformly. ACNet adapts the balance of global and local context to per-pixel demands, and reports state-of-the-art results on scene-parsing datasets.
1. Introduction
- FCNs can lose spatial detail through subsampling and produce rough boundaries, miss small objects, or misclassify large objects and stuff.Limited receptive fields or local context can make dense per-pixel classification ambiguous.Source: §1 · p. 1
- Earlier context methods apply context uniformly, although local cues help edges and small objects while global cues help large objects and stuff.The paper points to Figure 1 as showing different improvements from local and global context.Source: §1 · pp. 1–2, Figure 1 · p. 1
- ACNet fuses global image-level and middle-layer local context competitively, allocating more weight to one as the other receives less.Its adaptive context features vary across pixels according to their differing context demands.Source: §1 · p. 2
- The global module gauges demand by matching global and per-pixel features, while the local module uses the reverse gate and progressively reuses local features.The proposed modules are combined into adaptive context blocks at different network levels.Source: §1 · p. 2
- 82.3% Mean IoU on the Cityscapes test set without coarse data and 45.90% on ADE20K validation are reported as state-of-the-art results.The paper also reports state-of-the-art performance on various scene-parsing datasets.Source: §1 · p. 2
2. Related Work
Prior scene-parsing work draws on global context, local visual features, and attention or gating to refine predictions. ACNet distinguishes itself by using data-driven gates to adapt global and local context to pixel-level demands.
2. Related Work
- Global-context methods use image-level or pooled features to guide categorization of large semantic regions.Examples include augmenting each location with pooled features or selectively highlighting class-dependent feature maps.Source: §2 · p. 2
- Local-context approaches use low- and middle-level visual features to refine boundaries and small details, while attention and gating methods select or aggregate features.The cited approaches include encoder-decoder refinement, multiplicative boundary gating, and multi-scale aggregation.Source: §2 · p. 3
- ACNet introduces data-driven gating that captures global and local context according to pixel-aware context demand.This contrasts with the distinct global-context, local-context, and attention or gating approaches reviewed in this section.Source: §2 · pp. 2–3
3. Adaptive Context Network
ACNet replaces uniform context fusion with per-pixel competition between global and local context. Its modules use pixel-specific gates, combine coarse semantic guidance with local detail, and progressively refine features.
3.1. Overview
- ACNet uses a pretrained dilated ResNet and three adaptive context blocks at different resolutions to progressively produce a high-resolution segmentation map.The backbone removes downsampling operations and uses dilated convolutions in its last ResNet blocks, producing features at 1/16 input resolution.Source: §3.1 · p. 3
3.2. Global Context Module
- The Global Context Module matches each pixel feature to a global feature and uses pixel-specific gating to add global context, selectively improving consistency for large objects and stuff.The global feature is weighted by the gate and summed with the original features.Source: §3.2 · pp. 3–4
3.3. Local Context Module
- The Local Context Module reverses the upsampled global gate to emphasize local features where spatial-detail demand is greater, then concatenates them with upsampled features for refinement.A convolution fuses the concatenated features; the local gate reduces emphasis on pixels associated with large objects.Source: §3.3 · p. 4
- Reusing gated local features through three concatenation-and-convolution steps is described as recurrent learning that adds spatial detail and yields coarse-to-fine improvement.Source: §3.3 · p. 4
3.4. Adaptive Context Block
- The Adaptive Context Block cascades global-context selection with upsampling and local-context refinement, competitively fusing both context types according to their gated coefficients.Source: §3.4 · pp. 4–5
4. Experiments
The experiments evaluate ACNet on Cityscapes, ADE20K, PASCAL Context, and COCO Stuff, reporting new state-of-the-art performance on these datasets.
4.1. Datasets
- The experiments use Cityscapes, ADE20K, PASCAL Context, and COCO Stuff, with dataset-specific splits and class counts.Cityscapes has 2,979 training, 500 validation, and 1,525 test images; no coarse data is used.Source: §4.1 · p. 5
4.2. Implementation Details
- Implementation uses a pretrained dilated ResNet backbone, low-level features for local context in the first two blocks, and only a global context module in the last block.Training uses a poly learning-rate policy, synchronized batch normalization, random cropping, and horizontal flipping.Source: §4.2 · p. 5
4.3. Results on Cityscape dataset
- On Cityscapes validation, GCM outperforms GC in both tested settings, especially at 1/8 output resolution; local-context fusion raises performance from 72.45% to 74.67%.Three adaptive context blocks (ACB#3) improve performance to 76.53%.Source: §4.3 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
- Figure 4 maps show large global gated coefficients over dominant stuff and large objects, while small coefficients align with small objects and edges.The visualizations include coefficient maps at 1/16, 1/8, and 1/4 resolution.Source: §4.3 · p. 6, Figure 4 · p. 6
- On the Cityscapes testing set, ACNet achieves new state-of-the-art performance of 82.3% and surpasses Deeplabv3+ [4] at 82.1%, which uses extra coarse annotations in training.Source: Table 4 · p. 7, §4.3 · p. 7
4.5. Results on PASCAL Context Dataset
- ACNet obtains 54.1% Mean IoU on PASCAL Context, surpassing previously published methods, including approaches using ResNet-152 and Xception-71.The authors report a relatively large margin over those methods.Source: Table 7 · p. 8, §4.5 · p. 8
4.4. Results on ADE20K dataset
- 45.90% mIoU and 81.96% PixelAcc on ADE20K validation set establish new state-of-the-art performance with ResNet101.The single-model ACNet-101 test score is 55.84%.Source: §4.4 · pp. 7–8, Table 5 · p. 8
4.6. Results on COCO stuff Dataset
- ACNet achieves 40.1% Mean IoU on COCO Stuff, outperforming other state-of-the-art methods.Source: Table 8 · p. 8, §4.6 · p. 8
5. Conclusion
ACNet combines global and local context in an adaptive block to produce pixel-aware features for scene parsing. The paper reports strong performance against state-of-the-art methods and suggests the block could extend to other vision tasks.
5. Conclusion
- ACNet jointly uses global and local context modules in an adaptive context block to fuse contexts for each position.The design reflects global features’ role in categorizing semantically confused regions and local features’ role in producing sharp boundaries and clear details.Source: §5 · p. 8
- Extensive experiments demonstrate ACNet’s outstanding performance compared with other state-of-the-art methods.Source: §5 · p. 8
- The authors suggest extending the adaptive context block to object detection, pose estimation, and fine-grained recognition.Source: §5 · p. 8