PaperlayerView exact revision on arXiv

Source-linked AI summary

QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection

Chenhongyi Yang, Zehao Huang, Naiyan Wang

arXiv:2103.09136v2cs.CV

TL;DR

Small-object detection remains difficult because high-resolution images or feature maps improve representation but substantially increase computation. QueryDet uses Cascade Sparse Query to predict coarse locations on low-resolution features and sparsely guide high-resolution detection, improving accuracy and inference speed on COCO and VisDrone. The paper reports 3.0× average high-resolution acceleration on COCO and 2.3× on VisDrone, alongside improved detection results.

  • Problem

    Small-object detection has unsatisfactory performance and efficiency, while using high-resolution images or feature maps increases computational cost.

  • Method

    QueryDet recursively predicts coarse small-object locations on lower-resolution feature maps and uses them to guide sparse computation on higher-resolution features.

  • Results

    On COCO, QueryDet improves the RetinaNet baseline by 1.1 AP and 2.0 APS with 3.0× average high-resolution speedup, while VisDrone gains 2.3× speedup and state-of-the-art mAP.

  • Takeaways & Limitations

    QueryDet enables high-resolution small-object detection at lower cost and is presented as practical for real-time applications such as autonomous driving.

  • Takeaways & Limitations

    The reported failure cases include missed localization after correct coarse querying and false activation on large objects that slows inference.

Abstract

from arXiv · show

While general object detection with deep learning has achieved great success in the past few years, the performance and efficiency of detecting small objects are far from satisfactory. The most common and effective way to promote small object detection is to use high-resolution images or feature maps. However, both approaches induce costly computation since the computational cost grows squarely as the size of images and features increases. To get the best of two worlds, we propose QueryDet that uses a novel query mechanism to accelerate the inference speed of feature-pyramid based object detectors. The pipeline composes two steps: it first predicts the coarse locations of small objects on low-resolution features and then computes the accurate detection results using high-resolution features sparsely guided by those coarse positions. In this way, we can not only harvest the benefit of high-resolution feature maps but also avoid useless computation for the background area. On the popular COCO dataset, the proposed method improves the detection mAP by 1.0 and mAP-small by 2.0, and the high-resolution inference speed is improved to 3.0x on average. On VisDrone dataset, which contains more small objects, we create a new state-of-the-art while gaining a 2.3x high-resolution acceleration on average. Code is available at https://github.com/ChenhongyiYang/QueryDet-PyTorch.

1. Introduction

Small-object detection remains difficult because downsampling degrades relevant features, while high-resolution processing is computationally expensive. QueryDet addresses this with cascaded sparse queries that guide high-resolution computation toward likely object locations.

  • 1. Introduction: RetinaNet achieves 44.1 mAP on medium objects and 51.2 mAP on large objects but only 24.1 mAP on small objects on COCO.The cited passage attributes this degradation to downsampling, mismatched receptive fields, and localization difficulty.
  • 1. Introduction: Increasing input or feature-map resolution can improve small-object detection but incurs considerable computation costs.Feature pyramids reduce this burden by assigning objects of different scales to different feature levels.
  • 1. Introduction: QueryDet exploits sparse small-object layouts and coarse low-resolution predictions to avoid dense computation over background regions.The method infers rough object locations on low-resolution features before applying detection computation selectively at higher resolution.
  • 1. Introduction: Cascade Sparse Query recursively predicts queries on lower-resolution feature maps and guides sparse computation on higher-resolution maps.Sparse convolution reduces detection-head computation on low-level features while retaining small-object detection accuracy.
  • 1. Introduction: QueryDet is evaluated on COCO and VisDrone to assess acceleration and detection performance, including on a dataset containing many small objects.The paper presents QueryDet as a mechanism for reducing computation costs across feature-pyramid-based detectors.
  • 1. Introduction: 1.1 AP and 2.0 APS improvements over RetinaNet are reported on COCO, with 3.0× average high-resolution speedup; VisDrone reaches state-of-the-art mAP and 2.3× speedup.These results are reported when using high-resolution features and CSQ.

2. Related Works

Related work covers object-detector architectures, small-object recognition strategies, and sparse computation methods. QueryDet is positioned as a scale-aware sparse approach that targets redundant high-resolution feature processing.

  • 2. Related Works: Object detectors are commonly divided into two-stage and one-stage streams, with QueryDet implemented on RetinaNet and FCOS to demonstrate generalization.RetinaNet uses feature pyramids for multi-scale detection, while FCOS is an anchor-free one-stage detector.
  • 2. Related Works: Small-object recognition methods include increasing feature resolution, oversampling and augmentation, context incorporation, and scale-aware training.These categories address the challenge of low-resolution features in small-object recognition.
  • 2. Related Works: Sparse-computation methods use masks, gating, halting scores, or region filtering to reduce CNN computation.The cited approaches include Perforated-CNN, Dynamic Convolution, SACT, and SBNet.
  • 2. Related Works: RetinaNet’s high-resolution P3 accounts for 43% of total computation, while adding P2 raises the combined share of P2 and P3 to 74%.QueryDet is reported to reduce computation on those features by 99%.

3. Methods

QueryDet accelerates high-resolution FPN detection by predicting coarse small-object locations and selectively computing higher-resolution features at guided positions. Its Cascade Sparse Query design propagates sparse queries across pyramid levels while retaining standard detection training with rebalanced layer losses.

  • Sparse query pipeline: QueryDet uses coarse-to-fine sparse computation: low-resolution features predict rough small-object locations, guiding intensive computation on corresponding high-resolution locations.The rough locations act as query keys, while high-resolution features used for detection act as query values.
  • Query generation: Query heads generate heatmaps whose locations represent the probability that a grid position contains a small object.During training, small objects are assigned according to level-specific scale thresholds, and the query head uses focal loss.
  • Sparse query pipeline: The FPN produces multi-scale features, and each cascade layer applies query operations before sparse detection and query heads predict boxes and next-layer key positions.The pipeline begins at a query start layer such as P5 and proceeds toward higher-resolution levels.
  • Query generation: At inference, scores above threshold σ become queries, which map to four nearest-neighbor key positions on the preceding pyramid level.Those key positions index sparse value features, allowing the detection, regression, and query heads to process only selected locations.
  • Cascade Sparse Query: Cascade Sparse Query generates queries for a finer level only from the preceding level’s key positions, avoiding the exponentially increasing key sets caused by independently mapping queries.Sparse convolution kernels reuse the dense detection-head weights to compute results on selected positions.
  • Training: Training retains RetinaNet’s classification and regression objectives, adds focal-loss supervision for query scores, and rebalances pyramid-level losses with β_l.Rebalancing prevents the many P2 samples from dominating learning as higher-resolution features are added.

4. Experiments

Experiments show that QueryDet improves small-object detection while accelerating high-resolution inference across datasets, backbones, detector types, and query settings. Ablations and analyses identify the effectiveness and trade-offs of CSQ, high-resolution features, query placement, and context selection.

  • COCO and VisDrone: QueryDet improves RetinaNet on COCO from 37.46 AP and 22.64 APS to 38.53 AP and 24.64 APS using high-resolution features.These gains are 1.1 AP and 2.0 APS over the baseline.
  • COCO and VisDrone: 2.1 AP and 3.2 AP50 improvements are achieved on VisDrone, while inference speed rises from 1.16 FPS to 2.75 FPS, a 2.3× increase.VisDrone contains many small objects, making the gains especially significant in this evaluation.
  • Effectiveness and ablations: Adding high-resolution P2 initially drops AP by 1.34, while loss re-balancing raises AP to 38.11 and QueryHead adds 0.42 AP and 1.58 APS.The QueryHead provides extra objectiveness supervision, and the complete configuration reaches 38.53 AP and 24.64 APS.
  • CSQ design choices: P4 is the best CSQ starting layer because querying from higher-level layers loses speed, whereas starting higher increases AP loss.Very low-resolution features are fast but make small objects difficult to distinguish.
  • CSQ design choices: A 5x5 context patch provides enough context for small-object detection; more context slightly improves AP but reduces acceleration, while less context harms AP.This evaluates the context processed around queried positions.
  • Broader applicability: QueryDet averages 4.1× and 3.8× high-resolution speed improvements with MobileNet V2 and ShuffleNet V2, respectively, supporting edge-device deployment.The acceleration is more visible with lightweight backbones because backbone inference takes less time.
  • Broader applicability: CSQ improves high-resolution speed by 1.8× on average when applied to FCOS, demonstrating applicability beyond RetinaNet.QueryDet is presented as applicable to FPN-based detectors, including anchor-free detectors.

5. Conclusion

QueryDet uses Cascade Sparse Query to accelerate feature pyramid-based dense object detectors, targeting accurate and fast small-object detection. The method is presented as practical for real-time applications and as a basis for future 3D detection work.

  • QueryDet uses Cascade Sparse Query (CSQ) to accelerate inference for feature pyramid-based dense object detectors.
  • QueryDet is designed to detect small objects accurately at low cost, supporting deployment in real-time applications such as autonomous driving.
  • Future work extends QueryDet to 3D object detection, where LiDAR point clouds are sparser and 3D convolutions require more computational resources.
Loading 2103.09136v2…