PaperlayerView exact revision on arXiv

Source-linked AI summary

GFF: Gated Fully Fusion for Semantic Segmentation

Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Kuiyuan Yang

arXiv:1904.01803v2cs.CV

TL;DR

Semantic segmentation needs features that combine high-level semantics with fine spatial detail, but top- and bottom-layer features differ and simple fusion can mix useful and useless information. GFF uses gates to selectively fuse features across levels and achieves state-of-the-art performance on Cityscapes, Pascal Context, COCO-stuff, and ADE20K.

  • Problem

    Semantic segmentation needs feature maps that are both high-resolution and high-level, but top- and bottom-layer features differ and simple fusion can drown useful information.

    Source: Introduction · p. 1

  • Method

    GFF uses learned gates to selectively fuse feature maps from every level, regulating information flow across levels rather than only between adjacent levels.

    Source: Gated Fully Fusion · pp. 3–4

  • Results

    GFF achieves state-of-the-art performance on four semantic-segmentation benchmarks: Cityscapes, Pascal Context, COCO-stuff, and ADE20K.

    Source: Introduction · p. 2, Results on Other Datasets · pp. 7–8

  • Takeaways & Limitations

    The authors conclude that fusing missing low-level features into each pyramid level indicates GFF can handle small and thin objects in scenes.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Semantic segmentation generates comprehensive understanding of scenes through densely predicting the category for each pixel. High-level features from Deep Convolutional Neural Networks already demonstrate their effectiveness in semantic segmentation tasks, however the coarse resolution of high-level features often leads to inferior results for small/thin objects where detailed information is important. It is natural to consider importing low level features to compensate for the lost detailed information in high-level features.Unfortunately, simply combining multi-level features suffers from the semantic gap among them. In this paper, we propose a new architecture, named Gated Fully Fusion (GFF), to selectively fuse features from multiple levels using gates in a fully connected way. Specifically, features at each level are enhanced by higher-level features with stronger semantics and lower-level features with more details, and gates are used to control the propagation of useful information which significantly reduces the noises during fusion. We achieve the state of the art results on four challenging scene parsing datasets including Cityscapes, Pascal Context, COCO-stuff and ADE20K.

Introduction

GFF selectively fuses multi-level features to combine semantic strength with spatial detail, while DFP adds context; together they are reported to achieve state-of-the-art segmentation results across four benchmarks.

Introduction

  • Semantic segmentation requires high-level, high-resolution pixel representations, but ConvNets’ high-level features are coarse, while low-level features offer detail yet predict few pixels on small patterns.Pixels may belong to tiny or large objects, distant or near objects, or object interiors and boundaries; simply combining feature levels can drown useful information in noise.Source: Introduction · p. 1
  • GFF uses pixelwise gates to selectively propagate useful information between feature levels, enabling fully connected multi-level fusion while minimizing redundancy and noise.At each layer, gates send useful information to other layers or receive it when the current layer’s information is less useful.Source: Introduction · pp. 1–2
  • The dense feature pyramid (DFP) encodes and reuses contextual information across feature levels after GFF, complementing GFF’s capture of detailed backbone features.The paper combines both components in a single end-to-end network.Source: Introduction · p. 2
  • GFF achieves state-of-the-art performance on Cityscapes, Pascal Context, COCO-stuff, and ADE20K, including 82.3% mIoU on Cityscapes test with ResNet101 and 83.3% with WiderResNet trained only on fine-labeled data.The paper also reports detailed analysis and visualization of learned gates to illustrate GFF’s information-regulation mechanism.Source: Introduction · p. 2

Related Work

Prior work addresses missing context and fine detail through context modeling, multi-level fusion, or gating, but GFF combines multi-level feature maps simultaneously through gates. The paper distinguishes this approach from earlier methods and reports that it surpasses state-of-the-art approaches.

Context modeling

Multi-level feature fusion

  • Earlier feature-fusion methods combine levels through skips, pyramids, or adjacent-level fusion, but lack awareness of all maps’ usefulness, limiting useful-feature propagation.Source: Related Work · p. 2
  • Top-layer features lack fine detail, motivating methods that incorporate middle- or low-level features to improve segmentation of detailed structures.Source: Related Work · p. 2

Gating mechanism

  • Gating has been used to control information flow in recurrent and deep networks, fuse multi-task features, adapt pooling to depth, and encode boundary information.Examples include LSTM, GRU (Cho et al. 2014), PAD-Net, DepthSeg (Kong and Fowlkes 2018), and GSCNN.Source: Related Work · p. 2

GFF distinction

  • GFF simultaneously fuses multi-level feature maps through gating, distinguishing its approach from prior methods and surpassing state-of-the-art approaches.Source: Related Work · p. 2

Method

GFF addresses semantic and resolution gaps across backbone feature levels by gating their fusion, while a dense feature pyramid adds context before prediction. The method builds on PSPNet and supports end-to-end training with only slightly increased computation.

Multi-level Feature Fusion

  • High-level features provide semantics but lack detail, while shallow features retain resolution; concatenation, addition, and FPN can mix useful and non-informative features.The paper identifies indiscriminate fusion without measuring feature usefulness as the limitation of these basic strategies.Source: Multi-level Feature Fusion · p. 3

Gated Fully Fusion

  • GFF gates addition-based fusion across all feature levels, sending information only when the sender gate is high and the receiver gate is low.Each level’s gate map is estimated by a convolution followed by sigmoid; duplex gating suppresses useless information at both sender and receiver sides.Source: Gated Fully Fusion · pp. 3–4, Figure 2 · p. 3

Dense Feature Pyramid

  • DFP densely connects feature pyramids top-down, reusing higher-level context at lower levels; each pyramid receives preceding outputs, and their outputs feed final prediction.Its fusion function is a single convolution, and both DFP and GFF can be plugged into FCNs for end-to-end training with slightly extra computation.Source: Dense Feature Pyramid · p. 4

Network Architecture and Implementation

  • The network uses PSPNet with a ResNet backbone, dilated convolutions in the last two stages, GFF over stage features, and concatenated DFP outputs for segmentation.All backbone features are reduced to 256 channels before GFF; training is end-to-end with cross-entropy and an auxiliary stage-3 loss weighted 0.4.Source: Figure 3 · p. 3, Network Architecture and Implementation · p. 4

Experiment

Experiments evaluate GFF’s Cityscapes performance, fusion and inference variants, learned gates, and results on ADE20K, Pascal Context, and COCO-Stuff.

Implementation Details

  • Training uses SGD, poly learning-rate scheduling, synchronized batch normalization, and dataset-specific crop sizes and iteration counts.Cityscapes uses 864 × 864 crops, 100K iterations, and batch size 8; ADE20K, COCO-Stuff, and Pascal Context use 512 × 512 crops, 150K iterations, and batch size 16, with random flips, crops, color jitter, and scaling.Source: Implementation Details · pp. 4–5

Experiments on Cityscapes Dataset

  • GFF reaches 80.4% mIoU on Cityscapes validation, outperforming concatenation, addition, and FPN fusion methods.Concatenation and addition only slightly improve the PSPNet baseline; FPN performs best among those three, while GFF’s results support the value of gating and fully fusing feature levels.Source: Experiments on Cityscapes Dataset · p. 5, Table 1 · p. 5
  • DFP adds 0.8% mIoU, while multi-scale inference reaches 81.8% mIoU, exceeding DeepLabv3+ at 79.55% by 2.25%.The five evaluation scales are 0.75, 1, 1.25, 1.5, and 1.75; DFP also produces more consistent segmentation within large objects.Source: Experiments on Cityscapes Dataset · p. 5, Figure 5 · p. 5, Table 2 · p. 6
  • GFF and DFP improve results across two alternative backbones; PSPNet with ResNet18 improves 5.9% point over its baseline.The study describes the results as evidence of the modules’ generality across architectures.Source: Experiments on Cityscapes Dataset · p. 5, Table 4 · p. 6
  • The modules add 7.7% computational cost and 6.3% parameters compared with baseline PSPNet.The paper concludes that this overhead allows the method to be plugged into existing segmentation methods with little extra computational cost.Source: Experiments on Cityscapes Dataset · p. 5, Table 3 · p. 6
  • 80.9% mIoU on Cityscapes test using train-fine data beats PSANet by 2.3%; fine-tuning on train-fine plus val-fine reaches 82.3% mIoU.GFF also achieves the highest IoU on 15 of 19 categories, with large improvements in small/thin categories including poles, street lights/signs, people, and riders.Source: Experiments on Cityscapes Dataset · pp. 5–6, Table 5 · p. 6, Table 7 · p. 7

Visualization of Gates

  • Higher-level gates favor large structures and spread information broadly, whereas lower-level gates mainly receive information for local details and boundaries.Setting gates to zero highlights the roles of G1/G2 for object boundaries and G3/G4 for large patterns such as cars.Source: Visualization of Gates · pp. 6–7, Figure 7 · p. 7

Conclusion

Gated Fully Fusion (GFF) uses learned gate maps to fuse multi-level features, bridging high-resolution features with low semantics and low-resolution features with high semantics. The paper reports new state-of-the-art results on four challenging scene-parsing datasets and finds that fusing missing low-level features into each pyramid level can handle small and thin objects.

Conclusion

  • GFF achieves new state-of-the-art results on four challenging scene-parsing datasets.The conclusion reports this result without specifying dataset names or metric values.Source: Conclusion · p. 8
  • GFF fully fuses multi-level feature maps using learned gate maps to bridge high-resolution, low-semantic and low-resolution, high-semantic features.Source: Conclusion · p. 8
  • Fusing missing low-level features into each pyramid feature level indicates that GFF can handle small and thin objects in scenes.Source: Conclusion · p. 8

More visualization of Gates

The section presents additional visualizations of learned gate maps on Cityscapes and ADE20K.

More visualization of Gates

  • Additional gate-map visualizations are shown for Cityscapes and ADE20K.The examples appear in Figures 9 and 10, respectively.Source: More visualization of Gates · p. 9
  • On Cityscapes, the learned gate maps illustrate how gates control information propagation.Figure 9 labels G_i as the output of the ith layer’s gate.Source: Figure 9 · p. 10
  • On ADE20K, the visualization shows gate maps for the network layers.Figure 10 identifies G_i as the gate map of the ith layer.Source: Figure 10 · p. 10

Related papers

Loading 1904.01803v2…