PaperlayerView exact revision on arXiv

Source-linked AI summary

A DeNoising FPN With Transformer R-CNN for Tiny Object Detection

Hou-I Liu, Yu-Wen Tseng, Kai-Cheng Chang, Pin-Jyun Wang, Hong-Han Shuai, Wen-Huang Cheng

arXiv:2406.05755v4cs.CV

TL;DR

Tiny object detection is challenging because aerial-image objects have very limited pixel representation and FPN fusion can introduce noisy features. DNTR combines contrastive-learning-based DN-FPN with the Transformer-based Trans R-CNN, and reports stronger performance on AI-TOD and VisDrone benchmarks. The framework also supports a DN-FPN-integrated YOLOv8s variant that improves accuracy without changing its original speed.

  • Problem

    Tiny objects in aerial imagery are difficult to detect because their limited pixel representation challenges feature representation and conventional FPN fusion can generate noisy multiscale features.

  • Method

    DNTR combines DN-FPN, which uses geometric-semantic contrastive learning to reduce FPN fusion noise, with Trans R-CNN, which models local and global information within RoIs.

  • Results

    DNTR outperforms contemporaries on AI-TOD and VisDrone, recording AP scores of 26.2% and 33.1%, respectively.

  • Takeaways & Limitations

    DN-FPN can be integrated into YOLOv8s while matching its original speed and increasing AP, AP50, and APvt by 61.2%, 59.1%, and 97.1%, respectively.

Abstract

from arXiv · show

Despite notable advancements in the field of computer vision, the precise detection of tiny objects continues to pose a significant challenge, largely owing to the minuscule pixel representation allocated to these objects in imagery data. This challenge resonates profoundly in the domain of geoscience and remote sensing, where high-fidelity detection of tiny objects can facilitate a myriad of applications ranging from urban planning to environmental monitoring. In this paper, we propose a new framework, namely, DeNoising FPN with Trans R-CNN (DNTR), to improve the performance of tiny object detection. DNTR consists of an easy plug-in design, DeNoising FPN (DN-FPN), and an effective Transformer-based detector, Trans R-CNN. Specifically, feature fusion in the feature pyramid network is important for detecting multiscale objects. However, noisy features may be produced during the fusion process since there is no regularization between the features of different scales. Therefore, we introduce a DN-FPN module that utilizes contrastive learning to suppress noise in each level's features in the top-down path of FPN. Second, based on the two-stage framework, we replace the obsolete R-CNN detector with a novel Trans R-CNN detector to focus on the representation of tiny objects with self-attention. Experimental results manifest that our DNTR outperforms the baselines by at least 17.4% in terms of APvt on the AI-TOD dataset and 9.6% in terms of AP on the VisDrone dataset, respectively. Our code will be available at https://github.com/hoiliu-0801/DNTR.

I. INTRODUCTION

Tiny object detection in aerial imagery is difficult because objects occupy few pixels, while conventional FPN fusion can distort geometric and semantic features. DNTR addresses these issues with contrastive-learning-based DN-FPN and the Transformer-based Trans R-CNN detector.

  • Motivation: FPN combines low-resolution semantic features with high-resolution geometric features, but channel reduction and upsampling can introduce noise and information loss.These distortions arise during multiscale feature fusion in the top-down path.
  • DN-FPN: DN-FPN uses geometric-semantic contrastive learning to reduce redundant fusion features while preserving geometric and semantic information.Geometric and semantic encoders supervise feature representations from different FPN layers.
  • Trans R-CNN: Trans R-CNN decomposes each proposal into patch tokens and uses shuffle unfolding to aggregate surrounding tokens for richer local information.The detector is designed to address the limited pixel representation of tiny objects and capture long-range dependencies.
  • Framework: DNTR combines less noisy multiscale features with local and global RoI modeling through DN-FPN, shuffle unfolding, and Trans R-CNN.The framework is presented as an alternative to conventional two-stage and DETR-like detection pipelines.

II. RELATED WORKS

Tiny-object detection is challenged by limited pixel representation, feature distortion during FPN fusion, and weak use of local and global RoI information. DNTR addresses these issues with DN-FPN for contrastive noise reduction and Trans R-CNN for richer RoI representations.

  • Challenges: Tiny objects are difficult to detect because their representations contain limited pixel information and are sensitive to noise.Remote-sensing imagery also contains blurry objects that can be confused with background.
  • DN-FPN: DN-FPN uses geometric and semantic encoders with contrastive learning to preserve source information and reduce noise during FPN fusion.Its loss brings fusion representations closer to the corresponding lateral or upper-level sources while separating unrelated levels and batches.
  • Feature pyramid networks: FPN combines low-level geometric information from lateral features with high-level semantic information from upper-level features.Channel reduction can distort geometric information, while upsampling can distort semantic information.
  • Trans R-CNN: Trans R-CNN decomposes each proposal into patch tokens and aggregates surrounding tokens to capture local information and long-range dependencies within an RoI.This design targets more discriminative representations for tiny objects than traditional whole-proposal processing.
  • Contrastive relations: The geometric contrastive relation treats corresponding lateral and fusion representations as positive pairs and representations from different levels or images as negatives.The geometric relation is learned with InfoNCE to improve consistency between fusion and lateral features.
  • Consequences: DN-FPN preserves geometric and semantic consistency in fusion features, supporting improved tiny-object detection.The module uses the FPN’s multiscale feature structure and can be integrated without additional inference FLOPs or parameters.

B. Trans R-CNN

Trans R-CNN enhances tiny-object representations by combining shuffle unfolding, a mask transformer encoder, and task token selection. These components diversify local token features, capture global information, and separate tokens for classification and box regression.

  • Trans R-CNN: Trans R-CNN combines shuffle unfolding, a mask transformer encoder, and task token selection to improve tiny-object representations and global information.The detector is designed as a replacement within a two-stage R-CNN framework.
  • Shuffle Unfolding: Shuffle unfolding oversamples neighboring patch-token combinations, increasing feature diversity while preserving spatial correlation within each RoI.The oversampling rate r controls the number of generated unfolded tokens, and subsequent attention and fully connected layers preserve their order.
  • Shuffle Unfolding: The shuffle order increases the variety of unfolded-token combinations compared with raster scanning, especially when oversampling is applied.The paper reports similar AP at the basic token number but greater AP gains for shuffle unfolding with oversampling.
  • Mask Transformer Encoder: The mask transformer encoder concatenates unfolded tokens with learnable class and box tokens, then uses masked self-attention to obtain global information.The attention mask ignores interactions between the class and box tokens because they are treated as irrelevant to each other.
  • Task Token Selection: Task token selection ranks unfolded tokens by attention scores and assigns them to class-related or box-related groups for classification and regression.The class-related group supports classification, while the box-related group supports bounding-box regression.

C. Overall Objective

The overall objective combines classification, bounding-box regression, and DN-FPN feature optimization into a single loss. A balance parameter controls the contribution of the geometric and semantic losses.

  • Overall Objective: The overall loss combines cross-entropy classification loss, smooth L1 box-regression loss, geometric loss, and semantic loss.The geometric and semantic terms optimize DN-FPN features alongside the detector objectives.
  • Overall Objective: λ = 0.1 empirically balances the effects of the different loss terms.The balance parameter is specified in the overall loss formulation.
  • Task Token Selection: Algorithm 1 initializes class-related and box-related groups before sorting unfolded tokens by descending attention score.The procedure iterates through the sorted tokens and returns the two task-specific groups.

1) Dataset:

The experiments use AI-TOD and VisDrone as datasets focused on tiny objects, and COCO to assess the generalizability of DNTR.

  • AI-TOD: AI-TOD contains 28,036 aerial images and 700,621 instances, with 86% of objects smaller than 16 pixels and an average object size of 12.8 pixels.AI-TOD is split into 11,214 training, 2,804 validation, and 14,018 testing images.
  • VisDrone: VisDrone contains 10,209 drone-shot images across 10 categories at 2000×1500-pixel resolution.The dataset is divided into 6,471 training, 548 validation, and 3,190 test images; evaluation uses the validation set because the test set is unavailable.
  • COCO: COCO contains 80 categories, 118k training images, and 5k validation images for evaluating general object detection.The paper uses COCO to demonstrate DNTR’s generalizability.

2) Implementation Details:

DNTR is implemented in MMDetection with dataset-specific input sizes and training schedules. The model uses DetectoRS with a ResNet50 backbone and Cascade R-CNN as its detector.

  • Training Setup: Training uses input sizes of 800×800 for AI-TOD, 1333×800 for VisDrone, and 1280×800 for COCO.The model is optimized with SGD using a learning rate of 0.01, weight decay of 0.0001, and batch size 2.
  • Architecture: DNTR is based on DetectoRS with a ResNet50 backbone and uses Cascade R-CNN instead of HTC because segmentation annotations are not required.The DN-FPN projection encoders contain two convolution layers and two linear layers.

3) Evaluation Metrics:

The evaluation uses size-aware detection metrics across AI-TOD, VisDrone, and COCO, with results emphasizing DNTR’s performance on tiny objects and the effects of backbone and resolution choices.

  • Evaluation metrics: AP, AP50, and AP75 evaluate detection on AI-TOD, VisDrone, and COCO, while size-specific metrics vary by dataset.COCO additionally uses APs, APm, and APL; AI-TOD uses APvt, APt, APs, and APm.
  • AI-TOD results: DNTR exceeds the second-best AI-TOD model by at least 3.5 APvt (37.6%) and 1.6 APt (6.5%).The gain is larger for smaller objects, consistent with the paper’s focus on tiny-object detection.
  • AI-TOD results: CNN-based baselines remain below 11 APvt on AI-TOD, while DNTR improves tiny-object detection without the same tiny-versus-large-object trade-off.The passage attributes baseline weakness to extreme object size, FPN noise, and limited global-information capture.
  • Transformer comparisons: DETR-like models remain below 10 APvt without multiscale deformable attention, whereas Trans R-CNN uses masked box and class tokens to improve performance.The passage links the limitation to noisy multiscale features and joint box-class processing in the decoder.
  • Backbone and resolution: Larger backbones add only 0.2 AP in the best case, while increasing resolution improves DNTR by 2.8 AP and 4.5 APvt.The larger-backbone result is reported with ResNet101; resolution increases images from 800 × 800 to 1000 × 1000.

2) Experiments on VisDrone:

On VisDrone, DNTR achieves strong performance without cropping, while combining DN-FPN with a cropping-based pipeline recovers state-of-the-art performance under that setting.

  • VisDrone comparison: Without cropping strategies, DNTR outperforms DetectoRS by 10.8 AP (48.4%) and 13.7 AP50 (34.2%) on VisDrone.The comparison is conducted on the VisDrone validation set under the stated official evaluation settings.
  • Cropping strategies: With cropping strategies, DNTR slightly trails UFPMP because UFPMP uses separate networks for mosaic packaging and object detection.Applying DN-FPN to UFPMP’s cropping-based pipeline achieves state-of-the-art performance.
  • Generalization: DNTR remains competitive on COCO, with APL outperforming CNN-based and DETR-like models under the matched 1280 × 800 setting.The COCO evaluation uses ResNet50 and the same resolution for all compared methods.
  • DN-FPN effectiveness: DN-FPN improves FPN without additional parameters or FLOPs, increasing AP and APvt over FA2M by 10.7 (71.8%) and 7.9 (316.0%), respectively.Its encoders operate during training and are discarded during inference.
  • DN-FPN effectiveness: DN-FPN benefits one-stage and two-stage detectors as a plug-and-play module, improving DetectoRS by 10.8 AP and 10.4 APvt.DetectoRS can benefit twice because it uses a recursive feature pyramid with two FPNs.

4) Efficiency Analysis of the DNTR:

DNTR prioritizes detection accuracy over inference speed, while DN-FPN can improve faster detectors without sacrificing their original speed.

  • Accuracy-speed trade-off: DNTR achieves superior AP across tested models but does not provide the highest inference speed.The comparison uses FPS and FLOPs to assess efficiency alongside accuracy.
  • Accuracy-speed trade-off: Replacing YOLOv8s’s PAN-FPN with DN-FPN preserves its original speed while increasing AP, AP50, and APvt by 61.2%, 59.1%, and 97.1%.This integration is presented as a competitive speed-preserving alternative to DNTR.
  • Inference cost: DN-FPN is eliminated during inference, adding no FPS or FLOPs cost in the efficiency comparison.The training-stage denoising design supports its deployment as an efficient FPN enhancement.
  • Token design: Shuffle unfolding improves AP more under token oversampling than raster scanning because it provides greater variety in object representations.The selected configuration uses shuffle order with 64 tokens.

D. Visualization

The visualizations evaluate DNTR’s denoising behavior through PSNR-based response features and compare its detections with competing models across AI-TOD, VisDrone, and COCO. DNTR shows stronger response alignment and fewer detection errors in the reported comparisons.

  • Noise analysis: PSNR evaluates feature-map noise by comparing DNTR responses with target features derived from ground-truth object centers.The target feature is formed by projecting box centers and applying a Gaussian kernel; DNTR responses are obtained from the bottom FPN feature map.
  • Noise analysis: DNTR significantly outperforms Deformable-DETR and NWD-RKA in average PSNR on both AI-TOD and VisDrone.The paper uses higher average PSNR as evidence of improved denoising capability.
  • Detection visualization: DNTR better detects tiny bounding boxes than competing methods, producing more true positives and fewer false negatives.The comparison specifically contrasts DNTR with Deformable-DETR and NWD-RKA visualizations.
  • Detection visualization: Deformable-DETR produces many background responses and false positives, consistent with its lack of RPN-based filtering of unimportant regions.The visualization identifies background activation as a distinctive weakness of the DETR-like comparison.
  • Cross-dataset robustness: Across VisDrone and COCO visualizations, DNTR consistently yields fewer false positives and false negatives than DetectoRS.The response features of DetectoRS retain strong activation on cluttered regions, whereas DNTR’s detections are reported as cleaner.
  • Overall design: DNTR combines DN-FPN noise mitigation with Trans R-CNN global information aggregation in a two-stage model for tiny object detection.The conclusion presents these components as the basis of DNTR’s reported performance on AI-TOD and VisDrone.
Loading 2406.05755v4…