PaperlayerView exact revision on arXiv

Source-linked AI summary

CE-FPN: Enhancing Channel Information for Object Detection

Yihao Luo, Xiang Cao, Juntao Zhang, Xiang Cao, Jingjuan Guo, Haibo Shen, Tianjiang Wang, Qi Feng

arXiv:2103.10643v1cs.CV

TL;DR

FPN-based detectors can lose semantic information through channel reduction and suffer aliasing when heterogeneous features are fused. CE-FPN uses sub-pixel fusion and context enhancement together with channel attention, achieving competitive detection performance with modest computational cost.

  • Problem

    FPN-based methods face channel information loss from channel reduction, information decay during fusion, and aliasing in cross-scale fusion.

  • Method

    CE-FPN combines Sub-pixel Skip Fusion, Sub-pixel Context Enhancement, and Channel Attention Guided Module to preserve rich channel information and refine integrated features.

  • Results

    38.8 points and 40.9 points Average Precision (AP) are achieved with Faster R-CNN using ResNet-50 and ResNet-101, respectively, on MS COCO.

  • Takeaways & Limitations

    CE-FPN achieves competitive performance against state-of-the-art FPN-based detectors with only a few computation increases.

Abstract

from arXiv · show

Feature pyramid network (FPN) has been an effective framework to extract multi-scale features in object detection. However, current FPN-based methods mostly suffer from the intrinsic flaw of channel reduction, which brings about the loss of semantical information. And the miscellaneous fused feature maps may cause serious aliasing effects. In this paper, we present a novel channel enhancement feature pyramid network (CE-FPN) with three simple yet effective modules to alleviate these problems. Specifically, inspired by sub-pixel convolution, we propose a sub-pixel skip fusion method to perform both channel enhancement and upsampling. Instead of the original 1x1 convolution and linear upsampling, it mitigates the information loss due to channel reduction. Then we propose a sub-pixel context enhancement module for extracting more feature representations, which is superior to other context methods due to the utilization of rich channel information by sub-pixel convolution. Furthermore, a channel attention guided module is introduced to optimize the final integrated features on each level, which alleviates the aliasing effect only with a few computational burdens. Our experiments show that CE-FPN achieves competitive performance compared to state-of-the-art FPN-based detectors on MS COCO benchmark.

I. INTRODUCTION

FPN-based detectors face channel information loss, information decay during fusion, and aliasing in cross-scale fusion. CE-FPN addresses these issues with sub-pixel enhancement and channel attention, achieving competitive MS COCO detection results with limited computational cost.

  • Limitations: FPN-based methods reduce backbone channels, losing information as high-level features such as 2048 channels become 256-channel maps.Existing methods mainly add modules after channel reduction, trading lower computation for reduced channel information.
  • Limitations: Semantic information can be diluted during top-down fusion, while high-level features are not fully exploited for larger receptive fields.Context extraction is presented as a way to improve representation without directly adding deeper convolutional layers.
  • Limitations: Directly interpolating semantically different cross-scale maps can cause aliasing and confuse localization and recognition.The paper motivates attention-based refinement of integrated features to address this issue.
  • Proposed approach: CE-FPN introduces sub-pixel skip fusion and sub-pixel context enhancement to use rich channel information while performing channel enhancement and upsampling.These methods replace the original 1 × 1 convolution and upsampling approach and extend sub-pixel upsampling to channel-information fusion.
  • Results: 38.8 points and 40.9 points Average Precision (AP) are achieved with Faster R-CNN using ResNet-50 and ResNet-101, respectively, on MS COCO.The results use the 1× schedule and are reported after replacing FPN with CE-FPN.
  • Results: CE-FPN achieves significant improvements over state-of-the-art FPN-based detectors with only slightly increased computational cost.The paper specifically reports competitive performance relative to Libra R-CNN and AugFPN.
  • Proposed approach: A channel attention guided module optimizes integrated features at each level to alleviate aliasing with few computational burdens.The module extracts channel weights from integrated features and applies them to the final feature maps.

A. Deep object detectors

Deep object detectors include two-stage and one-stage designs, while FPN-based research improves multi-scale fusion through pathways, learned connections, context extraction, and attention. This paper focuses on reducing channel information loss and optimizing integrated features.

  • Detector categories: Two-stage detectors generate regions of interest before refining them with classification and regression, whereas one-stage detectors predict locations and classifications directly.The passage contrasts the efficiency and accuracy characteristics of the two detector categories.
  • FPN-based detectors: FPN addresses scale variation by merging features through a top-down pathway, and later methods add bottom-up paths, balanced fusion, or learned connections.PANet, Libra R-CNN, NAS-FPN, EfficientDet, and AugFPN extend or modify FPN fusion.
  • Feature augmentation: Context extraction and attention mechanisms are used to improve feature representation, localization, and classification in vision models.Examples include pyramid pooling for hierarchical global context and attention using global and local features.
  • This paper’s focus: This paper focuses on reducing information loss caused by channel decline in FPN construction and optimizing features after complicated integration.The stated focus connects channel enhancement with post-integration feature refinement.

III. PROPOSED METHODS

CE-FPN addresses channel information loss in FPN by retaining rich backbone channels during fusion and adding three modules: SSF, SCE, and CAG. SSF combines channel fusion with sub-pixel upsampling to enhance the feature pyramid.

  • CE-FPN introduces SSF, SCE, and CAG to alleviate channel information loss and optimize integrated feature maps.
  • FPN reduces high-level backbone outputs such as C5 from 2048 to 256 channels, losing channel information before fusion.
  • Sub-pixel convolution rearranges features from H×W ×C·r^2 to rH×rW ×C after channel dimensions are increased.
  • The resulting pyramid features are produced through element-wise summation and nearest-neighbor upsampling, as in FPN.
  • SSF directly connects C5 to F4 and C4 to F3, performing upsampling and channel fusion simultaneously.

C. Sub-pixel Context Enhancement

SCE enriches the highest-level feature by combining local, larger-field contextual, and global information. Their aggregation enlarges C5's receptive field and strengthens the integration map.

  • The highest-level FPN feature lacks multi-scale context, while higher-resolution inputs require larger receptive fields for large-object representation.
  • SCE generates three context scales from C5 and aggregates them into an integration map through element-wise summation.
  • A 3×3 convolution extracts local information and transforms channels for sub-pixel upsampling in the first pathway.
  • The second pathway combines max-pooling, channel expansion, and 4× sub-pixel upsampling to obtain contextual information for larger receptive fields.
  • Global average pooling produces a 1×1×C feature that is broadcast to the integration-map size for global contextual information.
  • SCE enlarges C5's receptive field and refines the integration map, allowing highest-level semantic information to be used without F5 and P5.

D. Channel Attention Guided Module

CAG is designed to reduce aliasing-related confusion after cross-scale fusion. It derives channel attention from pooled spatial descriptors and applies it to each pyramid level.

  • Cross-scale semantic differences can make integrated features produce aliasing effects that confuse localization and recognition.
  • CAG computes channel attention using independent global average-pooling and global max-pooling descriptors passed through fully connected layers.
  • The two descriptor outputs are summed and passed through a sigmoid function to form the channel-attention function.
  • CAG is intentionally lightweight and aims to reduce misleading aliasing features rather than build a sophisticated feature-discrimination architecture.

A. Dataset and Evaluation Metrics

The study evaluates CE-FPN on MS COCO using standard COCO-style mAP metrics and compares it with baselines and state-of-the-art FPN-based detectors. Experiments cover two-stage and one-stage detectors, multiple backbones, and qualitative detection results.

  • Dataset and metrics: MS COCO contains 80 categories, 115k training images, 5k validation images, and 20k test-dev images without publicly released labels.Models are trained on train-2017, ablations are reported on val-2017, and final comparisons use the test-dev evaluation server.
  • Dataset and metrics: The evaluation uses COCO-style mean Average Precision across IoU thresholds from 0.5 to 0.95 in increments of 0.05.
  • Quantitative evaluation: Replacing FPN with CE-FPN gives Faster R-CNN 38.8 AP with ResNet-50 and 40.9 AP with ResNet-101, respectively.These results are reported on COCO test-dev under the 1× schedule.
  • Quantitative evaluation: CE-FPN raises RetinaNet from 36.3 AP to 37.8 AP and provides improvements across small, medium, and large objects.
  • Comparative evaluation: CE-FPN achieves competitive performance against Libra R-CNN and AugFPN, while qualitative comparisons show stronger detection of objects across multiple scales than FPN.

D. Ablation Experiments

Ablations isolate the contributions of SSF, SCE, and CAG, showing gains from channel-preserving fusion, context enhancement, and channel attention with limited computational overhead.

  • Sub-pixel Skip Fusion: SSF adds 0.5 AP over the corresponding RetinaNet baseline, whereas replacing it with 1 × 1 convolution and linear interpolation provides no performance improvement.The comparison links SSF to reduced aliasing during cross-scale fusion.
  • Sub-pixel Skip Fusion: C4 has 1024 channels, exactly four times the 256-channel feature pyramid, while C5 has 2048 channels and is reduced by half for sub-pixel processing.Three C5 channel-transformation schemes are evaluated, including convolutional squeezing, channel selection, and channel splitting.
  • Sub-pixel Context Enhancement: SCE improves AP by 0.9 points alone, and combining SSF with SCE increases the improvement to 1.1 points over the corresponding baseline.
  • Efficiency: Removing F5 and P5 does not affect performance while reducing computational costs and parameters, and all components together add only slight computational cost.SSF introduces no extra computation or parameters; SCE and CAG add few or slight computational burdens.
  • Sub-pixel Context Enhancement: SCE exceeds PSPNet by 0.3 AP and CEM by 0.2 AP under otherwise matching experimental settings.Both comparison modules use the integration map and linear upsampling.
  • Channel Attention Guided Module: CAG boosts performance by 1.0 AP by optimizing channel information in output features after cross-scale fusion.The module is designed to alleviate negative effects associated with aliasing in integrated features.

E. Inference speed

CE-FPN modestly reduces inference speed relative to FPN but incurs a smaller slowdown than AugFPN under the reported Faster R-CNN setting.

  • Runtime comparison: 9.8 fps is achieved by Faster R-CNN with CE-FPN, compared with 10.5 fps for FPN using ResNet-50 and 1333 × 800 inputs.The runtime is averaged over COCO val-2017 and measured on an NVIDIA Quadro P5000.
  • Runtime comparison: 6.67% is the reported inference-speed decrease when replacing FPN with CE-FPN.
  • Runtime comparison: 17.2% is the reported inference-speed decrease when replacing FPN with AugFPN in Faster R-CNN.The paper uses this comparison to characterize CE-FPN's relative speed advantage.

V. CONCLUSION

The paper concludes that CE-FPN addresses channel information loss and aliasing in FPN-based detectors through three modules. Experiments show significant improvements across detectors with only a few additional computation costs.

  • Conclusion: CE-FPN combines SSF and SCE to use rich channel information through sub-pixel convolution, and CAG to alleviate aliasing across feature levels.
  • Conclusion: CE-FPN generalizes to various FPN-based detectors and improves performance with only a few computation increases.The paper identifies broader evaluation on more backbones and other multi-scale vision tasks as future work.
Loading 2103.10643v1…