Source-linked AI summary
GraphFPN: Graph Feature Pyramid Network for Object Detection
Gangming Zhao, Weifeng Ge, Yizhou Yu
TL;DR
Feature-pyramid methods use fixed topologies, while interactions between nonadjacent scales are indirect. GraphFPN builds an image-specific graph from a superpixel hierarchy and uses graph layers and local channel attention to enhance convolutional feature pyramids. Integrated with Faster R-CNN, it outperforms existing state-of-the-art detection methods on MS-COCO 2017 validation and test datasets.
Problem
Feature-pyramid methods use fixed topologies, and interactions between nonadjacent scales are indirect, potentially weakening essential features during propagation.
Source: §1 · pp. 1–2Method
GraphFPN builds an image-specific graph from a superpixel hierarchy, uses contextual and hierarchical layers and local channel attention, and fuses its features with a conventional feature pyramid.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported results support GraphFPN as a feature-pyramid enhancement for Faster R-CNN object detection on MS-COCO 2017.
Source: §5 · p. 9With COB hierarchy construction included, COB models have 21 (+21%) million parameters, 13.1 (+3.4%) GFLOPs, and 0.12 (+76.4%) seconds time cost; construction uses CPU contour detection and hierarchical segmentation.
Source: 4.3.LearnableParametersand… · p. 8
Abstract
from arXiv · showhide
Feature pyramids have been proven powerful in image understanding tasks that require multi-scale features. State-of-the-art methods for multi-scale feature learning focus on performing feature interactions across space and scales using neural networks with a fixed topology. In this paper, we propose graph feature pyramid networks that are capable of adapting their topological structures to varying intrinsic image structures and supporting simultaneous feature interactions across all scales. We first define an image-specific superpixel hierarchy for each input image to represent its intrinsic image structures. The graph feature pyramid network inherits its structure from this superpixel hierarchy. Contextual and hierarchical layers are designed to achieve feature interactions within the same scale and across different scales. To make these layers more powerful, we introduce two types of local channel attention for graph neural networks by generalizing global channel attention for convolutional neural networks. The proposed graph feature pyramid network can enhance the multiscale features from a convolutional feature pyramid network. We evaluate our graph feature pyramid network in the object detection task by integrating it into the Faster R-CNN algorithm. The modified algorithm outperforms not only previous state-of-the-art feature pyramid-based methods with a clear margin but also other popular detection methods on both MS-COCO 2017 validation and test datasets.
1. Introduction
Feature pyramids support multiscale image understanding, but fixed topologies and indirect interactions between nonadjacent scales may limit feature learning. GraphFPN uses an image-specific superpixel hierarchy, graph layers, and local channel attention; experiments on MS-COCO 2017 report better detection performance than existing state-of-the-art methods.
1. Introduction
- Existing feature-pyramid methods use fixed topologies, and interactions between nonadjacent scales are indirect, potentially weakening essential feature information during propagation.Multiscale features are desired because objects may appear at varying scales.Source: §1 · pp. 1–2
- GraphFPN inherits its image-specific structure from a superpixel hierarchy whose coarser regions group finer regions into part-whole relationships.The hierarchy is extracted from hierarchical segmentation; pixels within a superpixel typically belong to the same semantic object or part, reducing feature mixing between foreground objects and background clutter.Source: §1 · p. 2
- Contextual and hierarchical graph layers support feature interactions within the same scale and across scales, respectively.The graph pyramid's final features are fused with conventional FPN features to produce enhanced multiscale features.Source: §1 · p. 2
- The paper introduces two types of local channel attention for graph neural networks by generalizing global channel attention mechanisms from convolutional neural networks.Source: §1 · p. 2
- Experiments on MS-COCO 2017 validation and test datasets show GraphFPN improves over state-of-the-art detection methods, both feature-pyramid-based and other methods.The reported ablation studies also verify the effectiveness of the proposed network components.Source: §1 · p. 2
2. Related Work
Prior work develops feature pyramids and graph networks for multiscale vision, while hierarchical segmentation provides image-specific part-whole structures. GraphFPN combines these ideas by adapting its graph topology to each image's intrinsic structure and enabling interactions across scales.
2. Related Work
- FPN [30] builds top-down feature maps with lateral connections, PANet [33] adds bottom-up path augmentation, and FPT [54] performs feature interaction across space and scales with transformers.ZigZagNet also enriches multilevel context through top-down, bottom-up, and zig-zag aggregation.Source: §2 · p. 2
- GraphFPN maps backbone features to graph nodes on a superpixel hierarchy, applies contextual and hierarchical layers, then maps features back for fusion with conventional FPN maps.Source: Figure 1 · p. 3
- GraphFPN differs from prior feature-pyramid work by adapting its topology to each input image and enabling simultaneous interactions across all scales.Its graph pyramid is built on a superpixel hierarchy rather than a fixed grid topology.Source: §2 · p. 3
- Graph neural networks model dependencies among nodes in irregular structures, while GraphFPN's node merge and split operations follow image-specific superpixel hierarchies.Graph U-Net instead uses trainable similarity measures for graph pooling and saved information for unpooling.Source: §2 · p. 3
3. Graph Feature Pyramid Networks
GraphFPN builds an image-specific graph pyramid from hierarchical superpixels, then applies contextual and hierarchical layers before mapping and fusing graph features with the convolutional FPN.
3.1. Superpixel Hierarchy
- COB produces a hierarchical segmentation by recursively grouping pixels or smaller superpixels into larger regions, forming the image’s part-whole hierarchy.The selected levels are chosen so each higher level has one quarter as many superpixels as the preceding level.Source: §3.1 · p. 4
3.2. Multi-scale Graph Pyramid
- Superpixels become graph nodes, with same-level contextual edges and ancestor-descendant hierarchical edges; the latter are pruned by removing the lowest-similarity 50%.The hierarchy’s selected levels decrease in superpixel count by a factor of four at each step.Source: §3.2 · p. 4, §3.1 · p. 4
3.3. Graph Neural Network Layers
- Contextual layers use within-level edges, while hierarchical layers use pruned cross-level edges; local channel attention is neighborhood-specific and spatially adaptive.The architecture places contextual layers before and after hierarchical layers; local channel attention is reported to reduce computation and increase spatial adaptivity.Source: §3.3 · pp. 4–5
3.4. Feature Mapping between GNN and CNN
- Backbone grid cells are assigned to superpixels by largest overlap and max/min pooled to initialize graph nodes; graph features are copied to FPN cells and fused by concatenation.A 1×1 convolution adjusts the concatenated feature map to the FPN’s channel count.Source: §3.4 · p. 5
3.5. Object Detection
- The fused feature pyramid replaces conventional FPN in Faster R-CNN, which is trained end to end.The authors evaluate the integrated method in object detection.Source: §3.5 · p. 5
4. Experiments
Experiments evaluate GraphFPN for object detection through comparisons with feature-pyramid methods and other detectors, computational-cost analysis, and ablations of its components.
4. Experiments
- Experiments use COCO 2017 detection, Faster R-CNN, 36-epoch training, and 800/1,000-pixel resizing; superpixel hierarchies are precomputed in 0.120 seconds per image.Models are trained on the detection training set; reported comparisons use the standard test set and ablations use the validation set.Source: §4 · pp. 5–6
4.1. Comparison with State-of-the-Art Methods
- 43.7% AP is highest, at least 1.1% above other state-of-the-art algorithms; against FPN alone, AP improves 7.5%, with leads on AP50, AP75, APS, and APM.Results are on MS-COCO 2017 test-dev.Source: Table 1 · p. 6, §4.1 · pp. 6–7
4.2. Comparison with Other Object Detectors
- With the same ResNet-101 backbone, Faster R-CNN + FPN + GraphFPN outperforms all listed popular detectors across six performance metrics.The comparison uses the MS-COCO 2017 validation set.Source: Table 2 · p. 6, §4.2 · p. 7
4.3.LearnableParametersand Computational Cost
- GraphFPN+FPN has 1.89 times more learnable parameters, 1.21 times more GFLOPs, and 12.9% longer test time; CPU hierarchy construction costs 0.12 (+76.4%) seconds.The authors note that implementing hierarchical segmentation with CUDA and running it on a GPU could significantly reduce test time.Source: Table 3 · p. 7, 4.3.LearnableParametersand… · pp. 7–8
4.4. Ablation Studies
- Removing the first or second contextual group drops AP by 0.9% or 0.4%; keeping only contextual or hierarchical layers drops AP by 2.9% or 1.9%, respectively.The authors interpret the two layer types as complementary.Source: §4.4 · p. 8, Table 4 · p. 8
- Three layers in each of the three groups gives the best results on all five metrics, while too many graph layers worsen performance, attributed to gradient vanishing.The best setting has nine graph layers in total.Source: §4.4 · p. 8, Table 6 · p. 8
- Removing spatial self-attention, local average-pooling channel attention, or local channel self-attention drops AP by 1.3%, 1.2%, or 1.5%; removing both channel mechanisms worsens AP by 2%.The authors describe the two local channel-attention mechanisms as complementary.Source: §4.4 · p. 8, Table 5 · p. 8
5. Conclusions
GraphFPN adapts its topology to image-specific structures and enables feature interactions across scales; integrated with Faster R-CNN+FPN, it outperforms existing state-of-the-art object detection methods on MS-COCO 2017 validation and test datasets.
5. Conclusions
- GraphFPN adapts its topology to varying intrinsic image structures and supports simultaneous feature interactions across all scales.Its structure is inherited from a superpixel hierarchy constructed using hierarchical segmentation.Source: §5 · p. 8
- Contextual and hierarchical graph neural network layers handle within-scale and cross-scale interactions, respectively, with two types of local channel attention.The local channel attention mechanisms are introduced for graph neural networks.Source: §5 · pp. 8–9
- GraphFPN-enhanced Faster R-CNN+FPN outperforms existing state-of-the-art object detection methods on MS-COCO 2017 validation and test datasets.The conclusion reports this comparison without giving numerical results.Source: §5 · p. 9
A. Network Architecture and Visual Results
GraphFPN uses a five-level superpixel hierarchy to structure its graph, with contextual and hierarchical layer groups. The section also presents sample detection results from GraphFPN and comparisons with FPN and FPT.
A. Network Architecture and Visual Results
- GraphFPN uses three groups of three layers: contextual, hierarchical, then contextual again, with three attention modules in each layer.The graph nodes are shared across the layers, while contextual and hierarchical layers use different edge sets.Source: A. Network Architecture and Visual… · p. 12
- Each layer has N + N 4 + N 16 + N 64 + N 256 graph nodes, where N is the finest-level superpixel count.The node feature dimension is 256 in all reported experiments.Source: A. Network Architecture and Visual… · p. 12
- The five-level superpixel hierarchy is built by recursively merging superpixels from the finest partition according to contour strengths.The sample hierarchies use the COB hierarchical image segmentation algorithm.Source: A. Network Architecture and Visual… · p. 12
- Figure 1 shows sample detections from FPN, FPT, and GraphFPN, while Figures 5 and 6 show additional GraphFPN detections.The images are sampled from the MS-COCO 2017 validation set.Source: A. Network Architecture and Visual… · p. 12
B. Experiments on Semantic Segmentation
On Cityscapes validation, UFP + GraphFPN achieves clearly better performance than UFP and feature pyramid transformer. The experiments use the same settings as the cited comparison and are presented as evidence of the method's applicability to semantic segmentation.
B. Experiments on Semantic Segmentation
- The semantic segmentation experiment uses the same settings as [54] and adopts UFP as its feature pyramid construction module.UFP is compared with UFP + GraphFPN and feature pyramid transformer.Source: B. Experiments on Semantic Segmentation · p. 12
- The authors present the Cityscapes results as demonstrating GraphFPN's applicability to semantic segmentation.The dataset has 19 classes, and the passage describes it as containing 2,975,500 images for training and validation.Source: B. Experiments on Semantic Segmentation · p. 12
- UFP + GraphFPN achieves clearly better performance than UFP and feature pyramid transformer on Cityscapes validation.Table 3 compares semantic segmentation methods on that dataset; the passage reports no specific scores.Source: B. Experiments on Semantic Segmentation · p. 12, Table 3 · p. 14