Source-linked AI summary
Effective Fusion Factor in FPN for Tiny Object Detection
Yuqi Gong, Xuehui Yu, Yao Ding, Xiaoke Peng, Jian Zhao, Zhenjun Han
TL;DR
Tiny object detection remains challenging, and FPN’s deep-to-shallow fusion can help or hurt performance depending on its strength. The paper introduces fusion factor to control this fusion, estimates effective values statistically from dataset object distributions, and reports improved tiny-object detection performance with proper configuration.
Problem
FPN-based detectors perform poorly in application scenarios involving tiny objects, motivating analysis of how their top-down connections affect detection.
Method
The paper defines fusion factor to control deep-layer information delivered to shallow FPN layers and estimates effective values statistically from dataset object distributions.
Results
Properly configuring fusion factor improves tiny-object detection performance, while TinyPerson and Tiny CityPersons show performance increases followed by decreases as α increases.
Takeaways & Limitations
Adjusting adjacent-layer fusion factors can adapt FPN shallow layers toward learning tiny objects and provide positive influence for tiny object detection.
Takeaways & Limitations
The proposed estimation considers dataset-dependent attributes, including object sizes, dataset volume, and object distribution across FPN layers, as hypothesized influences on α.
Abstract
from arXiv · showhide
FPN-based detectors have made significant progress in general object detection, e.g., MS COCO and PASCAL VOC. However, these detectors fail in certain application scenarios, e.g., tiny object detection. In this paper, we argue that the top-down connections between adjacent layers in FPN bring two-side influences for tiny object detection, not only positive. We propose a novel concept, fusion factor, to control information that deep layers deliver to shallow layers, for adapting FPN to tiny object detection. After series of experiments and analysis, we explore how to estimate an effective value of fusion factor for a particular dataset by a statistical method. The estimation is dependent on the number of objects distributed in each layer. Comprehensive experiments are conducted on tiny object detection datasets, e.g., TinyPerson and Tiny CityPersons. Our results show that when configuring FPN with a proper fusion factor, the network is able to achieve significant performance gains over the baseline on tiny object detection datasets. Codes and models will be released.
1. Introduction
FPN-based detectors succeed on general detection benchmarks but perform poorly on tiny-object datasets. The paper attributes this to the two-sided effects of top-down fusion and proposes controlling deep-to-shallow information with a fusion factor.
- Tiny object detection supports applications including surveillance, driving assistance, and quick maritime rescue.
- FPN-based detectors perform well on MS COCO, PASCAL VOC, and CityPersons but poorly on TinyPerson and Tiny CityPersons.
- Figure 1 shows performance first increasing and then decreasing as deeper-layer information delivered to shallow layers increases.The fusion factor is the coefficient weighted on the deeper layer when adjacent FPN features are fused.
- FPN fusion creates an indirect multi-task learning effect in which each layer nearly learns objects across all sizes rather than only its corresponding scale.
- Across TinyPerson and Tiny CityPersons, performance under APall 50 fluctuates with fusion factor, whereas CityPersons, PASCAL VOC, and MS COCO remain relatively stable.
- The conventional FPN uses fusion factor 1, while an effective dataset-specific value is estimated statistically from object counts distributed across FPN layers.
2. Related Work
Related work addresses small-object detection through scale matching, scale regularization, super resolution, contextual enrichment, and feature-fusion strategies. These methods target representation quality, scale balance, and semantic consistency across feature layers.
- TinyPerson targets tiny person detection, with an average absolute object size of nearly 18 pixels.
- Scale matching aligns object scales between pretraining and target datasets, while SNIP and SNIPER regularize object sizes across image resolutions.
- Super resolution is used to recover information from low-resolution objects for small-object detection.
- PANet adds bottom-top connections, while other methods use attention, feature reconstruction, multi-branch architectures, or adaptive feature weighting.
- FPN combines deep semantic features with shallow geometric details through top-down and lateral connections to build a feature pyramid.
3. Effective fusion factor
The paper studies how fusion factor affects FPN performance for tiny-object detection and proposes estimating it from object distributions across adjacent feature layers. Experiments compare fixed, learned, attention-based, brute-force, and statistic-based settings, while analyses connect fusion-factor effects to object size, gradient propagation, and dataset scale.
- Motivation: FPN’s fusion proportion between adjacent feature layers is identified as an overlooked factor affecting tiny-person detection performance.The conventional FPN uses α = 1, while the paper investigates whether different fusion proportions better suit tiny-object detection.
- Effect of fusion factor: On TinyPerson and Tiny CityPersons, performance first increases and then decreases as α grows, indicating an optimal value range.Cross-scale datasets are less sensitive to α except when α = 0, whereas tiny-object datasets show a clear peak.
- Dataset factors: Object distribution across FPN layers determines training-sample sufficiency, with many tiny objects concentrated in P2 and P3 and fewer samples in deep layers.The differing CityPersons, Tiny CityPersons, and TinyPerson trends motivate examining layer-wise object distribution alongside object size.
- Mathematical explanation: The fusion factor reweights gradient propagation so deep layers participate differently in shallow-layer learning, creating a compromise between deep- and shallow-layer objectives.Too much deviation from the compromise value favors one layer’s task and risks losing information useful to the other.
- Estimating an effective fusion factor: The statistic-based S-α method computes α from object-number proportions between adjacent FPN layers, using dataset-wide statistics.Positive anchors are counted per layer from ground-truth assignments, and the resulting statistics determine the fusion factors.
- Estimating an effective fusion factor: All non-fixed α settings outperform the α = 1 baseline, while S-α achieves performance comparable to brute-force search without requiring the same redundant computation.Attention-based estimation also increases computation, whereas brute-force search is limited by redundant evaluations.
4. Experiment
Experiments evaluate the proposed S-α configuration on TinyPerson and related tiny-object datasets, showing gains across detectors, backbones, and datasets.
- Experimental setting: TinyPerson experiments use RetinaNet with a pretrained ResNet-50, 12 training epochs, specified anchors, horizontal flips, and IoU = 0.5 as the main threshold.Images containing more than 200 objects are excluded, and large images are cropped into overlapping sub-images.
- TinyPerson comparison: S-α improves RetinaNet on TinyPerson without adding network parameters and performs better than the compared detectors except SM/MSM.SM/MSM uses COCO scale-matching pretraining, whereas RetinaNet with S-α uses only ImageNet pretraining.
- Backbone comparison: ResNet-101 does not outperform ResNet-50 for tiny-person detection, while its additional stage-4 blocks increase computational burden.Tiny objects are mainly distributed in FPN levels P2 and P3, whereas the extra ResNet-101 blocks are deeper.
- Other tiny datasets: RetinaNet with S-α also improves results on Tiny CityPersons and COCO100 and remains as effective as the best brute-force fusion-factor result.The reported result supports validity beyond TinyPerson.
5. Conclusion
The paper analyzes fusion factor in FPN and estimates effective values to improve tiny object detection across varied experimental settings.
- The paper analyzes why fusion factor affects tiny object detection and estimates effective values for stronger positive influence.The analysis includes a mathematical explanation based on gradient propagation in FPN.
- Adjusting adjacent-layer fusion factors adaptively encourages shallow layers to focus on tiny objects, improving detection performance.
- Extensive experiments demonstrate effectiveness across different detectors, backbones, and datasets.
- The authors plan to extend the method to other object scales and difficult tasks such as occluded or truncated detection.