PaperlayerView exact revision on arXiv

Source-linked AI summary

HS-FPN: High Frequency and Spatial Perception FPN for Tiny Object Detection

Zican Shi, Jing Hu, Jie Ren, Hengkang Ye, Xuyang Yuan, Yan Ouyang, Jia He, Bo Ji, Junyu Guo

arXiv:2412.10116v3cs.CV

TL;DR

Tiny object detection remains challenging because tiny targets have weak, easily disrupted features and FPN lacks direct tiny-object enhancement and spatial perception. HS-FPN addresses these gaps with high-frequency perception and spatial-dependency modules, and experiments report improved detector performance on AI-TOD and DOTAmini10.

  • Problem

    Tiny object detection is limited by weak, interference-prone features and by FPN’s lack of direct tiny-object enhancement and spatial perception.

  • Method

    HS-FPN enhances FPN using high-frequency responses for channel and spatial feature refinement and pixel-level spatial-dependency learning between adjacent feature maps.

  • Results

    Faster R-CNN with ResNet50 achieves 20.3 AP versus 18.3 with FPN on AI-TOD, while HS-FPN also improves Cascade R-CNN from 20.2 to 23.6.

  • Takeaways & Limitations

    HS-FPN provides a feature-pyramid replacement that improves tiny-object detection across detectors and datasets within the reported experiments.

  • Takeaways & Limitations

    The spatial dependency module assumes feature-map sizes are multiples of one another and uses blocks matched to C5 for perfect division.

Abstract

from arXiv · show

The introduction of Feature Pyramid Network (FPN) has significantly improved object detection performance. However, substantial challenges remain in detecting tiny objects, as their features occupy only a very small proportion of the feature maps. Although FPN integrates multi-scale features, it does not directly enhance or enrich the features of tiny objects. Furthermore, FPN lacks spatial perception ability. To address these issues, we propose a novel High Frequency and Spatial Perception Feature Pyramid Network (HS-FPN) with two innovative modules. First, we designed a high frequency perception module (HFP) that generates high frequency responses through high pass filters. These high frequency responses are used as mask weights from both spatial and channel perspectives to enrich and highlight the features of tiny objects in the original feature maps. Second, we developed a spatial dependency perception module (SDP) to capture the spatial dependencies that FPN lacks. Our experiments demonstrate that detectors based on HS-FPN exhibit competitive advantages over state-of-the-art models on the AI-TOD dataset for tiny object detection.

1 Introduction

Tiny object detection is difficult because tiny targets retain few, weak, interference-prone features, while standard FPN neither specifically enhances them nor models their spatial dependencies. HS-FPN addresses these gaps with high-frequency feature refinement and spatial-dependency modeling, improving detection performance on AI-TOD.

  • 1 Introduction: Tiny objects retain few feature-map pixels after repeated downsampling, producing weak responses that hinder precise detection and localization.FPN combines multi-scale features but does not directly increase tiny-object feature content.
  • 1 Introduction: FPN applies uniform processing and pixel-wise feature addition, so it provides neither special attention to tiny objects nor robust spatial alignment across layers.Recursive upsampling can offset tiny-target features between upper and lower layers.
  • 1 Introduction: Filtering low-frequency components can highlight tiny objects because they appear mainly as image details and edges, and moderate filtering significantly improves SCR.Figure 1 shows SCR first increasing and then decreasing as the filtering range expands.
  • 1 Introduction: HFP uses predefined high-pass filters to generate responses that refine original FPN features through channel and spatial masks, while SDP learns dependencies between adjacent upper and lower maps.HFP emphasizes channels and locations containing tiny-object features; SDP uses pixel-level similarity to enrich them with spatial dependencies.
  • 1 Introduction: 20.3 AP versus 18.3 with FPN is achieved by Faster R-CNN with ResNet50 and HS-FPN on AI-TOD, while Cascade R-CNN improves from 20.2 to 23.6.HS-FPN has a similar overall structure to FPN and can be embedded into models requiring FPN.
  • 1 Introduction: HS-FPN replaces FPN with high frequency perception and spatial dependency perception modules to address tiny-object feature and spatial-perception limitations.The paper integrates HS-FPN into various detection models and evaluates it on tiny-object detection datasets.

2 Related Work

Related work in tiny object detection emphasizes benchmark construction, label assignment, and multi-scale learning, while frequency-domain analysis motivates using high-frequency information for tiny-object features.

  • 2 Related Work: Tiny object detection research mainly advances through benchmark construction, improved label assignment, and multi-scale learning.These directions address dataset coverage, matching quality, and integration of semantic and detailed features.
  • 2 Related Work: Datasets such as AI-TOD, SODA, and AI-TOD-v2 supplement COCO with more tiny-object instances and evaluation metrics.The paper identifies insufficient tiny-object instances in generally used datasets such as COCO.
  • 2 Related Work: NWD, RFLA, and Dot Distance improve label assignment because IoU is highly sensitive to localization deviations for tiny objects.The cited methods modify or replace conventional IoU-based matching considerations in anchor-based detectors.
  • 2 Related Work: FPNs use top-down pathways and lateral connections to combine deep semantic features with shallow detailed information for tiny object detection.This multi-scale design is the architectural context that HS-FPN extends.
  • 2 Related Work: Frequency-domain methods use transformations such as DCT and Fourier analysis to incorporate frequency information into deep learning features.Prior work includes channel weighting based on DCT, although not all cited studies specifically target tiny-object detection.

3 Method

HS-FPN extends FPN with high-frequency feature refinement and spatial dependency modeling to enrich tiny-object representations and address feature misalignment.

  • HS-FPN Framework: HS-FPN collects four backbone features, reduces them to 256 channels, and builds {P2, P3, P4, P5} through a top-down pathway.The input strides are {4, 8, 16, 32} pixels, respectively.
  • High Frequency Perception Module: HFP applies a high-pass filter to each input feature and uses the resulting response in channel and spatial paths to generate attention weights.The channel path produces channel weights, while the spatial path produces a spatial mask for refining the original feature map.
  • High Frequency Perception Module: High-frequency responses increase the relative proportion of tiny-object features by suppressing low-frequency components and homogeneous background.This makes channel-attention estimation more accurate and highlights spatial regions containing tiny-object features.
  • High Frequency Perception Module: The HFP channel path combines global average and max pooling, ReLU, and grouped 1×1 convolutions to produce final channel attention weights.Pooling preserves spatially integrated and maximum activation information before channel scoring.
  • Spatial Dependency Perception Module: SDP computes cross-attention between pixels in corresponding blocks of adjacent feature maps, then aggregates the outputs and adds them to the lower-level feature.The upper feature is upsampled first; Q comes from Ci, while K and V come from the upsampled Pi+1.
  • Spatial Dependency Perception Module: Unlike ViT, SDP uses separate similarity matrices within feature blocks rather than one matrix across feature blocks.This design models pixel-level spatial dependencies between adjacent pyramid features.

4 Experiments and Analysis

Experiments on AI-TOD and DOTAmini10 evaluate HS-FPN through component ablations, filter-rate studies, benchmark comparisons, and computational analysis. HS-FPN consistently improves tiny-object detection, with HFP and SDP providing complementary gains at modest additional cost.

  • 4.3 Ablation Study: 2.2 AP points over baseline are obtained by adding HFP, while APt and APs increase by 2.6 and 2.0 points, respectively.The ablation uses Cascade R-CNN with ResNet50 and evaluates progressively added HFP and SDP components on AI-TOD.
  • 4.3 Ablation Study: 1.4 AP points are gained by either CP or SP alone, while their combination adds 0.8 points beyond using only one branch.This supports a synergistic effect between HFP’s channel and spatial paths.
  • 4.3 Ablation Study: Detection performance first improves and then declines as α increases from 0 to 1, leading to α = 0.25 for AI-TOD and DOTAmini10.The trend parallels SCR behavior, indicating that an appropriate degree of low-frequency filtering benefits tiny-object detection.
  • 4.3 Ablation Study: 3.4 AP points over FPN result when HFP and SDP are combined, exceeding HFP alone by 1.2 and SDP alone by 2.3 points.The combined modules enrich tiny-object features while suppressing some high-frequency noise amplified by HFP.
  • 4.5 Computational Complexity: HS-FPN slightly increases computational cost while significantly improving performance over FPN in Cascade R-CNN with ResNet50 and 800 × 800 inputs.HFP’s cost mainly comes from the 3 × 3 convolutions, with CP and SP having similar computational costs.

5 Conclusion

The paper proposes HS-FPN to improve tiny-object detection by addressing weak feature representation and missing spatial dependencies. HFP enhances frequency-domain features, SDP enriches spatial context, and experiments on two datasets validate the method.

  • 5 Conclusion: HS-FPN combines HFP for frequency-domain feature enhancement with SDP for modeling spatial dependencies between adjacent pixels in upper and lower feature maps.The modules target the small scale and low-quality feature representations of tiny objects.
  • 5 Conclusion: Experiments on two tiny-object detection datasets validate the effectiveness of HS-FPN.

A.1 Motivation of HFP

The HFP motivation is that tiny objects correspond to high-frequency details but occupy a small, noisy portion of feature maps. HFP uses filtered responses to derive spatial and channel emphasis for tiny-object features.

  • A.1 Motivation of HFP: As α increases, low-frequency components are progressively removed and tiny-object features become increasingly prominent in the high-frequency response F2.At α = 0, F2 remains identical to C2 and retains substantial low-frequency background information.
  • A.1 Motivation of HFP: Channel responses differ because convolutional kernels emphasize different aspects, and channels contain varying proportions of tiny-object features.These differences affect each channel’s contribution to detection performance.
  • A.1 Motivation of HFP: HFG filters low-frequency components before channel weighting, increasing the proportion of tiny-object features used to identify informative channels.The resulting channel weights are dynamically adjusted according to tiny-object features rather than remaining uniform.
  • A.1 Motivation of HFP: HFP and SDP each expand tiny-object feature dimensions and enhance feature responses at the P2 level.The visualizations compare processing by HS-FPN, HFP alone, and SDP alone.

A.2 Differences between SDP and ViT

SDP differs from ViT by computing pixel-level attention within feature blocks rather than region-level attention across blocks. This local design enriches spatial context while avoiding the much larger cost of global pixel attention.

  • A.2 Differences between SDP and ViT: SDP computes similarity among pixels within each feature block, producing one (hw) × (hw) similarity matrix per block.Its computational complexity is O(n(hw)^2c), whereas ViT computes one n × n matrix across feature blocks.
  • A.2 Differences between SDP and ViT: SDP learns spatial dependencies between pixels and their local surroundings, enriching spatial context information for tiny objects.ViT instead implements region-level attention across feature blocks.
  • A.2 Differences between SDP and ViT: SDP’s computational cost is (hw)/n times ViT’s and increases as patch size hw becomes larger.
  • A.2 Differences between SDP and ViT: Global pixel attention is avoided because it would create an (nhw) × (nhw) similarity matrix with complexity O((nhw)^2c).The paper considers global attention unnecessary because tiny-object feature sizes are typically small.
  • A.2 Differences between SDP and ViT: The SDP attention calculation is considered effective while keeping computational complexity within an acceptable range.

A.3 Datasets and Metrics

DOTAmini10 was developed to provide balanced tiny-object instances for detection research, while evaluation uses AI-TOD’s scale-specific AP categories.

  • DOTAmini10 Dataset: DOTAmini10 reports train and validation category counts separately for its ten selected categories.
  • DOTAmini10 Dataset: DOTAmini10 selects 10 balanced categories from DOTA and processes their annotations and images for tiny-object detection.The construction includes image scaling and a stride-based procedure, though the supplied passage truncates the final implementation details.
  • Metrics: AI-TOD evaluates average precision using scale classes of 2–8, 8–16, 16–32, and above 32 pixels, reported as APvt, APt, APs, and APm.

A.4 Train Details

The experiments use standardized detector settings centered on a pretrained ResNet50 backbone and a 256-channel FPN.

  • Model setting: Unless otherwise specified, models use a pretrained ResNet50 backbone.
  • Model setting: The region proposal network uses 3000 proposals, and the feature pyramid network uses 256 channels.
  • Model setting: Inference filters background boxes at a confidence threshold of 0.05 before non-maximum suppression.

A.5 Visualization Results.

Visualization comparisons on AI-TOD and DOTAmini10 show that replacing FPN with HS-FPN reduces false-negative and false-positive predictions for tiny objects.

  • Visualization Results: Replacing FPN with HS-FPN significantly reduces both false-negative and false-positive predictions in Cascade R-CNN visualizations on AI-TOD and DOTAmini10.Both comparisons use ResNet50 backbones; green, blue, and red boxes denote true positives, false positives, and false negatives, respectively.
  • AI-TOD: The AI-TOD visualization compares Cascade R-CNN with FPN and HS-FPN across alternating rows, using colored boxes for TP, FP, and FN predictions.
  • DOTAmini10: The DOTAmini10 visualization uses the same alternating-row comparison between Cascade R-CNN with FPN and HS-FPN.
Loading 2412.10116v3…