PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Pyramid Transformer

Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, Qianru Sun

arXiv:2007.09451v1cs.CV

TL;DR

Existing visual recognition systems provide spatial context but omit non-local interactions across feature scales. FPT actively transforms a feature pyramid through self-level, top-down, and bottom-up interactions, and experiments report consistent gains across detection and segmentation tasks.

  • Problem

    Existing non-local spatial interactions do not capture contexts of objects or parts residing at different scales.

  • Method

    FPT transforms a feature pyramid into another pyramid of the same size using Self-, Grounding-, and Rendering-Transformer interactions across self-level, top-down, and bottom-up directions.

  • Results

    FPT consistently improves baselines and state-of-the-art methods across object detection, instance segmentation, and semantic segmentation.

  • Takeaways & Limitations

    FPT is a generic, plug-and-play feature interaction approach applicable to multiple visual recognition tasks without changing feature-pyramid size.

Abstract

from arXiv · show

Feature interactions across space and scales underpin modern visual recognition systems because they introduce beneficial visual contexts. Conventionally, spatial contexts are passively hidden in the CNN's increasing receptive fields or actively encoded by non-local convolution. Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales. To this end, we propose a fully active feature interaction across both space and scales, called Feature Pyramid Transformer (FPT). It transforms any feature pyramid into another feature pyramid of the same size but with richer contexts, by using three specially designed transformers in self-level, top-down, and bottom-up interaction fashion. FPT serves as a generic visual backbone with fair computational overhead. We conduct extensive experiments in both instance-level (i.e., object detection and instance segmentation) and pixel-level segmentation tasks, using various backbones and head networks, and observe consistent improvement over all the baselines and the state-of-the-art methods.

1 Introduction

Modern recognition systems use visual context from CNN receptive fields and active spatial interactions, but existing non-local methods omit cross-scale interactions. FPT addresses this gap by transforming feature pyramids with self-level, top-down, and bottom-up interactions, improving recognition across detection and segmentation tasks.

  • CNNs passively encode broader visual context through pooling, stride, dilated convolution, and increasingly large receptive fields.
  • Image pyramids represent objects at corresponding resolutions but multiply CNN computation, whereas in-network feature pyramids provide multiple scales without repeated CNN passes.
  • Non-local convolution and self-attention actively model long-range spatial co-occurrence, but existing approaches do not interact across feature scales.
  • FPT converts a feature pyramid into an equally sized pyramid whose levels encode non-local interactions across space and scales before task-specific prediction.
  • FPT uses Self-Transformer, Grounding Transformer, and Rendering Transformer for self-level, top-down, and bottom-up feature interactions, respectively.
  • 8.5% box-AP and 6.0% mask-AP gains were reported on MS-COCO, alongside mIoU gains on Cityscapes, PASCAL VOC 2012, ADE20K, and LIP.The reported semantic-segmentation gains were 1.6%, 1.2%, 1.7%, and 2.0%, respectively.

2 Related Work

Prior feature-pyramid methods support multiple visual tasks, but the paper argues that non-local interaction at one uniform scale is insufficient. It therefore targets interactions at the corresponding scales of objects or parts.

  • Feature pyramids are widely used to boost object detection, instance segmentation, and semantic segmentation across visual tasks.
  • The paper identifies uniform-scale non-local interaction as insufficient for representing contexts involving objects or parts at different scales.

3 Feature Pyramid Transformer

Feature Pyramid Transformer transforms feature pyramids through self-level, top-down, and bottom-up interactions across space and scales. Its components use distinct interaction patterns to enrich feature maps while preserving pyramid structure.

  • Overview: FPT transforms a feature pyramid into a same-sized pyramid with richer feature maps encoding non-local interactions across space and scales.It is designed as a feature transformer that can precede task-specific head networks.
  • Self-Transformer: Self-Transformer captures co-occurring object features within one feature map and preserves the input scale.It modifies non-local interaction by using Mixture of Softmaxes for normalization.
  • Grounding Transformer: Grounding Transformer performs top-down interaction by grounding higher-level concepts in lower-level pixels, producing an output at the lower-level scale.It uses negative Euclidean distance as the similarity function for feature maps with different semantic information.
  • Grounding Transformer: Locality-constrained Grounding Transformer restricts each low-level query to a local square region of higher-level keys and values for semantic segmentation.The design reflects the greater informativeness of local context around the query position in this setting.
  • Rendering Transformer: Rendering Transformer performs bottom-up interaction by combining high-level concepts with low-level visual attributes through feature-map-level channel-wise attention and convolutional refinement.The low-level value map is down-sampled before being summed with the refined attended high-level map.

4 Experiments

Experiments evaluate FPT on instance-level and pixel-level recognition tasks through ablations, state-of-the-art comparisons, and visualizations. FPT consistently improves recognition performance while adding fair computational overhead.

  • Experiments cover object detection, instance segmentation, and semantic segmentation with ablations, extensive state-of-the-art comparisons, and representative visualizations.
  • Instance-Level Recognition: 38.0% bounding box AP and 36.8% mask AP are achieved by the full ST, GT, and RT combination on MS-COCO 2017 validation.These results are 6.4% and 6.9% higher than BFP for detection and instance segmentation, respectively.
  • Pixel-Level Recognition: FPT visualizations report segmentation mIoU values on MS-COCO 2017 and PASCAL VOC 2012 validation samples, with red rectangles marking better predicted areas.
  • Pixel-Level Recognition: 1.4% training mIoU and 2.6% validation mIoU improvements over UFP are obtained by the best model on Cityscapes.Adding FPT transformers increases model parameters and GFLOPs, described as a fair computational overhead.
  • Pixel-Level Recognition: 1.6%, 1.2%, 1.7%, and 1.8% mIoU improvements are reported on Cityscapes, ADE20K, LIP, and PASCAL VOC 2012, respectively.The comparisons use the same ResNet-101 backbone and report new state-of-the-art performance over previous methods.

5 Conclusion

The conclusion presents FPT as an efficient, plug-and-play feature interaction approach for feature pyramids. Across three visual recognition tasks, quantitative and qualitative experiments show consistent improvements over baselines and state-of-the-art methods.

  • FPT uses three transformers to encode explicit self-level, top-down, and bottom-up information in a feature pyramid.
  • FPT preserves the feature pyramid size, making it generic and easy to plug into modern deep networks.
  • Quantitative and qualitative results across three visual recognition tasks show consistent improvements over baselines and state-of-the-art methods.

Supplementary Materials

The supplementary materials provide additional analyses of effectiveness, hyperparameters, complexity, quantitative comparisons, and qualitative results.

  • Supplementary sections cover Feud effectiveness, hyperparameters, FPT complexity, quantitative comparisons, and additional qualitative results.

A Effectiveness of Feud

Feud improves Grounding Transformer performance over the conventional Fsim similarity, while MoS settings affect ST and GT results. For LGT, square size 5 performs best on Cityscapes.

  • Effectiveness of Feud: Feud with MoS achieves the best performance, improving object detection by up to 3.1% box AP and instance segmentation by up to 3.3% mask AP.The comparison evaluates Feud against Fsim with and without Mixture of Softmaxes.
  • Influence of N: N=2 brings the best performance for Self-Transformer across the evaluated object detection and instance segmentation cases.N=1 corresponds to the classical softmax without MoS.
  • Influence of N: N=4 achieves the best performance for Grounding Transformer in both object detection and instance segmentation.The result is reported in the GT normalization study.
  • Square size in LGT: LGT with square size 5 achieves the best performance on Cityscapes semantic segmentation.The study reports both training and validation mIoU.

B.4 DropBlock in Instance-level Tasks

The supplementary experiments examine DropBlock hyperparameters for instance-level and pixel-level tasks, with the available semantic-segmentation result favoring block size 3 and keep probability 0.9.

  • Instance-level Tasks: DropBlock hyperparameters are investigated for object detection and instance segmentation using ResNet-50.The instance-level table reports bounding-box AP and mask AP.
  • Pixel-level Tasks: Block size 3 and keep probability 0.9 achieve the best validation mIoU for Cityscapes semantic segmentation.The experiment uses dilated ResNet-101 and reports validation-set mIoU.

C FPT Complexity

FPT provides the reported instance-segmentation improvements with lower average parameter and FLOP increases per AP point than the non-local operation.

  • Complexity Analysis: FPT complexity is compared with its components and the non-local operation on MS-COCO instance segmentation using a ResNet-50 backbone.The analysis uses a Mask R-CNN head and reports model parameters and FLOPs.
  • Complexity Analysis: 0.21× Params and 0.15× FLOPs are the average increases required by FPT per improved AP point.These increases are lower than the non-local operation's 0.27× Params and 0.27× FLOPs.

D.1 Results on Stronger Backbones

On stronger NL-, GC-, and AA-ResNet backbones, BFP+FPT continues to outperform the compared cross-scale interaction methods.

  • Results on Stronger Backbones: BFP+FPT achieves better performance than BFP+FPN, BFP+BPA, and BFP+BFI on NL-, GC-, and AA-ResNet backbones.The comparison evaluates stronger backbone networks beyond standard ResNet.
  • Results on Stronger Backbones: Up to 40.8% bounding box AP and 38.7% mask AP are achieved by BFP+FPT on the stronger backbones.The corresponding maxima for BFP+FPN, BFP+BPA, and BFP+BFI are lower in both reported metrics.

D.2 Results on Deeper Backbones

Results on different backbones show that FPT remains competitive with fewer parameters, including against deeper-backbone alternatives. On ResNet-50, BFP + FPT surpasses BFP + FPN and BFP + BPA on both detection and instance segmentation metrics.

  • Evaluation setup: Table S9 compares bounding box detection and instance segmentation results across different backbones on the MS-COCO 2017 validation set.The results are split into detection and instance segmentation metrics around a dashed separator.
  • Backbone comparisons: BFP + FPT on ResNet-50 achieves 38.0%/36.8% AP, surpassing BFP + FPN and BFP + BPA on ResNet-101 under similar 88 M parameters.BFP + FPN achieves 36.2%/35.7% AP, while BFP + BPA achieves 37.3%/36.3% AP.
  • Backbone comparisons: Compared with BFP on ResNet-152, FPT still achieves higher results with fewer parameters.BFP + FPN on ResNet-152 can slightly outperform FPT on ResNet-50, but uses more parameters.

E More Qualitative Results

Qualitative visualizations show high-quality FPT predictions across object detection, instance segmentation, and semantic segmentation. The examples highlight accurate recognition of distant or thin objects, improved larger-object segmentation, and dataset-related failure cases.

  • Object detection: FPT correctly detects small distant objects such as persons and sheep in MS-COCO test-set examples.The visualizations include additional object detection results in Fig. S1.
  • Semantic segmentation: FPT produces precise segmentation for thin objects and enhances segmentation quality for larger objects in PSACAL VOC 2012 examples.Examples include a biker’s foot, cat’s tail, distant man, woman’s arm, sofa, bottle, and dining table.
  • Failure cases: Failure examples include objects in unannotated backgrounds being recognized as real objects.Examples include a cat in a painting, a bike on a wall, and a man in a mirror.
  • Instance segmentation: Additional instance segmentation visualizations use red rectangles to highlight areas where FPT makes better predictions.These samples come from the MS-COCO 2017 test set.
Loading 2007.09451v1…