PaperlayerView exact revision on arXiv

Source-linked AI summary

ExFuse: Enhancing Feature Fusion for Semantic Segmentation

Zhenli Zhang, Xiangyu Zhang, Chao Peng, Dazhi Cheng, Jian Sun

arXiv:1804.03821v1cs.CV

TL;DR

Semantic segmentation feature fusion can be ineffective when low-level and high-level features differ substantially in semantic level and spatial resolution. ExFuse addresses this gap by enriching low-level features with semantic information and high-level features with spatial details, yielding a 4% total improvement and a reported 87.9% mean IoU on PASCAL VOC 2012.

  • Problem

    Feature fusion in U-Net-style segmentation can be ineffective when low-level and high-level features have large semantic or resolution gaps.

  • Method

    ExFuse bridges the gap by adding semantic information to low-level features and spatial information to high-level features through several proposed fusion enhancements.

  • Results

    4% total improvement is obtained by combining the approaches, and ExFuse achieves 87.9% mean IoU on the PASCAL VOC 2012 test dataset.

  • Takeaways & Limitations

    Enhanced feature fusion improves segmentation with original low-level features and produces a new state-of-the-art result on the PASCAL VOC 2012 benchmark.

  • Takeaways & Limitations

    Generalization of the perspective and techniques to other vision tasks remains an open question left for future work.

Abstract

from arXiv · show

Modern semantic segmentation frameworks usually combine low-level and high-level features from pre-trained backbone convolutional models to boost performance. In this paper, we first point out that a simple fusion of low-level and high-level features could be less effective because of the gap in semantic levels and spatial resolution. We find that introducing semantic information into low-level features and high-resolution details into high-level features is more effective for the later fusion. Based on this observation, we propose a new framework, named ExFuse, to bridge the gap between low-level and high-level features thus significantly improve the segmentation quality by 4.0\% in total. Furthermore, we evaluate our approach on the challenging PASCAL VOC 2012 segmentation benchmark and achieve 87.9\% mean IoU, which outperforms the previous state-of-the-art results.

1 Introduction

U-Net-style segmentation fuses low-level high-resolution and high-level low-resolution features, but semantic and resolution gaps can make this fusion ineffective. ExFuse bridges these gaps by enriching low-level features with semantics and high-level features with spatial information, achieving a 4% total improvement and 87.9% mean IoU on PASCAL VOC 2012.

  • 1 Introduction: U-Net frameworks fuse high-level low-resolution features with low-level high-resolution features to generate high-resolution semantic results.The encoder extracts features at different semantic levels and resolutions, while the decoder progressively fuses them.
  • 1 Introduction: “Pure” low-level and high-level features are difficult to fuse because they have significant semantic and resolution gaps.Low-level features provide spatial details but lack semantic information, whereas high-level features have the opposite characteristics.
  • 1 Introduction: ExFuse bridges the gap by introducing semantic information into low-level features and spatial information into high-level features.Its proposed solutions include layer rearrangement, semantic supervision, semantic embedding, explicit channel resolution embedding, and densely adjacent prediction.
  • 1 Introduction: 4% total improvement is obtained by combining ExFuse’s approaches, and the method achieves 87.9% mean IoU on the PASCAL VOC 2012 test dataset.The reported score surpasses previous state-of-the-art methods.
  • 1 Introduction: The paper presents bridging semantic and resolution gaps through more effective feature fusion as a perspective for improving semantic segmentation.The fully equipped model is reported to achieve a new state-of-the-art test result.

2 Related Work

Related work situates ExFuse among feature-fusion, deeply supervised learning, and upsampling methods used in semantic segmentation. The paper’s architecture is presented as an extension of a GCN-based framework with proposed components distinguished from backbone components.

  • 2 Related Work: Semantic segmentation methods commonly fuse low-level high-resolution and high-level low-resolution features for complementary information.Other feature-fusion methods combine multi-scale features, bounding-box features, or segmentation maps for different purposes.
  • 2 Related Work: Deeply supervised learning adds extra losses to ease optimization in very deep neural networks.The approach is discussed as a precedent for auxiliary supervision in segmentation and classification models.
  • 2 Related Work: Semantic segmentation uses bilinear interpolation, deconvolution, and sub-pixel convolution as three main upsampling approaches.The paper places its work within the broader development of decoder upsampling operations.
  • 2 Related Work: Figure 2 distinguishes proposed ExFuse components from solid-box components belonging to the GCN backbone framework.The figure labels semantic supervision, explicit channel resolution embedding, semantic embedding branch, and densely adjacent prediction.

3 Approach

ExFuse studies feature fusion in U-Net-style segmentation, where encoder features trade semantic level against spatial resolution. Using GCN as a baseline, it finds that adding low-level feature maps yields only marginal gains, motivating methods to bridge this gap.

  • Feature Fusion: U-Net-style frameworks use pretrained encoders to produce low-level, high-resolution and high-level, low-resolution features for decoder fusion.The decoder mixes encoder features to predict segmentation results.
  • Baseline Analysis: GCN extracts four feature levels with spatial resolutions {128, 64, 32, 16} for a 512 × 512 input.Higher feature levels have more semantic information but lower spatial resolution.
  • Motivation: The baseline analysis attributes ineffective fusion to a semantic and resolution gap between low-level and high-level features.ExFuse therefore introduces solutions that add semantic information to low-level features and spatial details to high-level features.
  • Baseline Analysis: 0.24% for ResNet 50 and 0.05% for ResNeXt 101 are the marginal improvements contributed by the lowest two feature levels.Segmentation quality increases with more feature levels but saturates quickly.
  • Baseline Settings: The baseline uses the fully equipped GCN architecture with a ResNeXt 101 encoder pretrained on ImageNet.Evaluation uses PASCAL VOC 2012 and the Semantic Boundaries Dataset, with performance measured by mean intersection-over-union.

3.1 Introducing More Semantic Information into Low-level Features

ExFuse adds semantic information to low-level features through layer rearrangement, semantic supervision, and semantic embedding. These methods are motivated by placing early features closer to semantic supervision so they can support later fusion more effectively.

  • Approach: ExFuse proposes layer rearrangement, semantic supervision, and semantic embedding branch to add semantic information to low-level features.The methods are motivated by the tendency of features near semantic supervision to encode more semantic information.
  • Layer Rearrangement: Layer rearrangement changes ResNeXt 101 stage assignments from {3, 4, 23, 3} to {8, 8, 9, 8} while preserving overall computational complexity.The rearrangement places more layers in early stages.
  • Semantic Supervision: Semantic Supervision attaches auxiliary supervisions to early encoder stages, forcing low-level features to encode more semantic concepts for later fusion.The auxiliary branches are attached during pretraining and removed before fine tuning.
  • Semantic Supervision: 1.5% improvement is obtained when fine tuning extracts features from the second convolutional layer in the auxiliary module.The broader semantic-supervision method boosts segmentation by 1.1%, while the second-layer extraction produces more improvement.
  • Comparison: The paper distinguishes its semantic-supervision use from PSPNet, where deeply supervised learning primarily aims to ease optimization rather than explicitly supervise extracted features.The supplied comparison notes that the architectures use auxiliary supervision for different purposes.

Semantic Embedding Branch

The Semantic Embedding Branch generalizes decoder fusion by conditioning low-level feature refinement on high-level features. It is applied to Levels 1–3 and improves segmentation performance in the reported experiment.

  • Semantic Embedding Branch: Semantic Embedding Branch generalizes fusion so the low-level residual depends on low-level and multiple higher-level feature maps.The formulation uses y_l = Upsample(y_l+1) + F(x_l, x_l+1, . . . , x_L).
  • Semantic Embedding Branch: The branch introduces semantic information from high-level features to guide resolution fusion.Its design is detailed in Fig. 4.
  • Results: 0.7% improvement is reported when SEB is applied to feature Levels 1–3.The experiment compares the corresponding ablation settings in Table 3.
  • Implementation: Element-wise multiplication combines feature maps in the Semantic Embedding Branch, with each high-level feature map upsampled before the component produces its output.This behavior is specified in the figure caption.

3.2 Embedding More Spatial Resolution into High-level Features

ExFuse embeds spatial resolution information into high-level features through Explicit Channel Resolution Embedding and Densely Adjacent Prediction. The design emphasizes explicit channel encoding rather than relying solely on learned upsampling.

  • Motivation: High-level backbone features have limited spatial resolution, motivating methods that encode more spatial details into their channels.For a 224 × 224 input, the top-most ResNet or ResNeXt feature map can be 7 × 7.
  • ECRE: Explicit Channel Resolution Embedding replaces learned deconvolution with parameter-free Sub-pixel Upsample so auxiliary supervision explicitly impacts feature channels.Sub-pixel upsampling enlarges feature maps by reshaping spatial and channel dimensions.
  • ECRE Ablation: Replacing deconvolution without auxiliary supervision performs worse than the baseline, indicating that sub-pixel upsampling alone is insufficient in the reported ablation.The comparison is reported in Table 4, setting #3.
  • Evaluation: Table 3 evaluates SS, LR, ECRE, SEB, and DAP using mean IoU on the PASCAL VOC 2012 validation set.The baseline is the authors’ implementation of GCN.
  • DAP: Densely Adjacent Prediction predicts results at adjacent positions and averages associated predictions to generate the final segmentation map.The mechanism is designed to encode spatial information into decoder feature channels.

Densely Adjacent Prediction

Densely Adjacent Prediction (DAP) embeds spatial information into decoder feature channels by predicting adjacent-position results and averaging associated scores. With k = 3, DAP improves performance by 0.6%.

  • Densely Adjacent Prediction: DAP predicts results at adjacent positions, then averages associated scores to generate the final segmentation map.Feature channels are divided into k × k groups for a window of size k × k.
  • Densely Adjacent Prediction: DAP operates on the decoder output and uses k = 3 in the experiment.The decoder output channels are increased to 189 because DAP requires k × k times more feature channels.
  • Densely Adjacent Prediction: 0.6% performance improvement is obtained with DAP, while the counterpart without DAP achieves only marginal gain.The baseline comparison controls for the increased number of feature channels.

3.3 Discussions

The discussion attributes ineffective baseline fusion to the semantic and resolution gap between low-level and high-level features. ExFuse addresses this gap by enriching low-level semantics and high-level spatial details, with high-level enhancement also improving segmentation directly.

  • 3.3 Discussions: Baseline fusion produces only marginal improvements from the lowest feature levels, indicating a semantic and resolution gap between low-level and high-level features.The issue is observed for fusion involving Level 1 and Level 2 features.
  • 3.3 Discussions: ExFuse introduces semantic information into low-level features and spatial details into high-level features to improve later feature fusion.The proposed solutions are described in Sections 3.1 and 3.2.
  • 3.3 Discussions: ExFuse remains much better than the baseline when only top-most Level 4 feature maps are used.This result implies superior high-level feature quality in the proposed model.
  • 3.3 Discussions: Methods that enhance high-level features contribute most of the improvement and also benefit segmentation performance directly.This conclusion is based on the further study reported alongside Table 6.

Do techniques work in a vanilla U-Net?

The proposed techniques generalize from the GCN-based U-Net structure to a vanilla U-Net. On PASCAL VOC 2012, performance rises from 72.7 to 79.6 mIoU, while broader generalization remains open for future work.

  • Do techniques work in a vanilla U-Net?: 79.6 mIoU is achieved on PASCAL VOC 2012 with the vanilla U-Net, compared with 72.7 mIoU for the original model.The gain is 6.9 percentage points, larger than the 4.0-point gain reported for the GCN structure.
  • Do techniques work in a vanilla U-Net?: The larger 6.9-point improvement shows that the techniques generalize well beyond the GCN-based U-Net structure.The comparison is made between the vanilla U-Net and its original model.
  • Do techniques work in a vanilla U-Net?: Whether the perspective and techniques generalize to other vision tasks remains for future work despite positive ablation results.The paper specifically raises low-level vision and detection as other possible tasks.

4 PASCAL VOC 2012 Experiment

The fully equipped ExFuse system is evaluated on PASCAL VOC 2012 through progressively stronger backbones, pretraining, and training strategies. It reaches 87.9% mIoU on the test set without DenseCRF post-processing and surpasses previous state-of-the-art results.

  • 4 PASCAL VOC 2012 Experiment: 80.8% mIoU is achieved using a ResNeXt 131 backbone trained on 10,582 PASCAL VOC 2012 and SBD images.The deeper backbone provides a 0.8% performance gain and is reported as 2.3% better than DeepLabv3 at the same setting.
  • 4 PASCAL VOC 2012 Experiment: COCO pretraining is incorporated before staged training on COCO, SBD, and PASCAL VOC 2012 images.The first stage mixes 109,892 images, followed by stages using SBD and PASCAL VOC 2012 and then the standard PASCAL VOC 2012 training set.
  • 4 PASCAL VOC 2012 Experiment: 85.8% mIoU is obtained on the PASCAL VOC 2012 validation set after averaging predictions with the horizontally flipped image.This is reported as 2.3% better than DeepLabv3+.
  • 4 PASCAL VOC 2012 Experiment: 87.9% mIoU is achieved on the PASCAL VOC 2012 test set without DenseCRF post-processing, surpassing previous state-of-the-art results.Using a standard ResNet101 yields 86.2% mIoU under the same comparison setting.
  • 4 PASCAL VOC 2012 Experiment: ExFuse produces more precise boundaries than the GCN baseline in representative visualizations.The comparison is shown through qualitative results in Fig. 7.

5 Conclusions

ExFuse addresses ineffective fusion between high-resolution low-level and low-resolution high-level features, improving segmentation performance and establishing a new PASCAL VOC 2012 state-of-the-art result.

  • ExFuse tackles ineffective feature fusion by bridging the gap between high-level low-resolution and low-level high-resolution features.
  • 85.4% mIoU is achieved by ExFuse on the PASCAL VOC 2012 validation examples, compared with 81.0% for the GCN baseline.
  • ExFuse achieves new state-of-the-art performance on the PASCAL VOC 2012 benchmark.
Loading 1804.03821v1…