PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Query as Budgeting for Tiny Object Detection

Yi Zhang, Di Xiong, Ge Gao, Xiangyue Zhang, Yihang Qiu, Yuxuan Zhou, Shuo Chen

arXiv:2610.33430v1cs.CV

TL;DR

Tiny-object DETR detectors face density imbalance, while existing dynamic-query designs entangle query budgeting with query selection. The paper analyzes these effects and proposes BUTR, which estimates an input-dependent query budget from encoder confidences while retaining standard Top-K initialization. Experiments report improved accuracy and efficiency over Dome-DETR on AI-TOD-v2 and VisDrone, with inference-only COCO integration reducing queries and computation without degrading overall AP.

  • Problem

    A fixed decoder budget can be insufficient for rare crowded scenes yet impose unnecessary overhead on mostly sparse images, while dynamic-query designs conflate how many queries are decoded with which queries are retained.

    Source: §1 · pp. 1–2

  • Method

    BUTR uses a lightweight, non-learned budgeter to estimate a per-image decoder query budget from encoder confidences and retains standard Top-K initialization.

    Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6

  • Results

    BUTR improves overall accuracy and efficiency over Dome-DETR on AI-TOD-v2 and VisDrone; inference-only COCO integration reduces queries and computation without degrading overall AP.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The evaluated evidence suggests that adaptive decoder budgeting explains much of dynamic query’s benefit, while inference-only integration also applies the approach to COCO.

    Source: §1 · p. 2, §6 · p. 10

    Budget allocation depends on encoder-confidence scale and may be affected by calibration, proposal count, or domain; other dense settings remain unevaluated.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent DETR-based tiny object detectors adopt dynamic query mechanisms to handle severe density imbalance. However, existing designs entangle two effects: how many and which queries are decoded. This entanglement obscures what drives the gains and complicates dynamic-query design; additional learned predictors and heuristic filtering also introduce overhead and potential selection errors. We disentangle these effects and show that dynamic query primarily acts as \emph{budgeting}---allocating decoder query capacity according to object density---rather than intricate query selection.Motivated by this view, we formulate a monotonic and conservative budgeting principle and propose a budgeting-based DETR (BUTR), a streamlined framework that uses a lightweight, non-learned budgeter to estimate an input-dependent query budget $K(x)$ from encoder confidences. BUTR retains standard Top-$K(x)$ query initialization without additional learned modules or heuristic filtering in the budgeter.At $800\times800$ resolution, BUTR-L achieves 35.7 AP at 47.8 FPS on AI-TOD-v2 and 29.8 AP at 42.1 FPS on VisDrone on an NVIDIA T4 GPU. Inference-time integration on COCO further demonstrates its applicability to general scenario detection. Code is available at https://github.com/zhangyi-magic/BUTR.

1 Introduction

Tiny-object detection must balance a fixed query budget that serves mostly sparse images against the needs of rare crowded scenes. The paper argues that dynamic query mainly helps by budgeting decoder capacity, then proposes BUTR, a lightweight non-learned per-image budgeter that retains standard Top-K initialization.

Motivation

  • Tiny objects occupy few pixels and are ambiguous amid clutter, while aerial images range from mostly sparse scenes to rare images with hundreds or more instances.A fixed budget can underserve crowded cases; uniformly increasing it adds overhead and can increase false positives in sparse scenes.Source: §1 · p. 1
  • Existing dynamic-query approaches add predictors or heuristic filtering, and selection-heavy designs entangle query count with which queries survive.DQ-DETR uses density prediction and preset budget bins; Dome-DETR combines a learned window-mask predictor with heuristic filtering.Source: §1 · pp. 1–2

Evidence

  • On AI-TOD-v2, fixed Top-600 produces highly similar detections to Dome-DETR’s dynamic initialization, while Top-300 is insufficient in the extreme-density example.The figure color-codes true positives green, false positives blue, and false negatives red.Source: Figure 1 · p. 2

Core finding

  • Dynamic query primarily adapts decoder capacity to object density: gains concentrate on rare high-density images, while confidence-ranked Top-K at a comparable budget closely matches PAQI across density subsets.At fixed query count, spatially distinct DDQ-style selection and Top-K achieve the same overall AP on AI-TOD-v2 and VisDrone.Source: §1 · p. 2

Proposed approach

  • BUTR estimates a per-image query budget K(x) from encoder-proposal density and uses standard Top-K(x) initialization, without extra learned modules or heuristic filtering in the budgeter.The budgeter is lightweight and non-learned.Source: §1 · p. 2

2 Related Work

Fixed-query DETRs retain a global query budget despite density imbalance, while dynamic-query methods adapt decoder capacity per input. The mechanism question is whether dynamic query helps through query count or through query selection.

2.1 Fixed-Query DETRs

  • Fixed-query DETRs retain a fixed budget, typically 300–900 queries, which is efficient for sparse images but may be insufficient for rare crowded scenes.The related-work discussion notes strong performance from DETR-style and transformer-augmented two-stage baselines on VisDrone and AI-TOD-v2.Source: §2.1 · pp. 2–3

2.2 Dynamic-Query DETRs

  • Dynamic-query methods include DQ-DETR’s learned density prediction and preset budget bins, and Dome-DETR’s multi-stage filtering of an enlarged candidate set.Related adaptive-query approaches also appear in remote sensing, including FDA-DETR and CGAQ-DETR.Source: §2.2 · p. 3

3 Rethinking Dynamic Query: A Budgeting View

Inference-time interventions on Dome-DETR suggest that dynamic query’s benefit comes mainly from decoder budget rather than intricate selection. Fixed Top-K can reproduce PAQI’s accuracy at a comparable average query count, and changing query identity at fixed count leaves overall AP unchanged in the reported comparison.

PAQI interventions

  • Removing SPLIT causes substantial AP drops, while removing MASK or NMS with SPLIT retained leaves overall AP unchanged despite query-count variation.SPLIT preserves Top-Kmin proposals, including high-confidence foreground proposals that filtering may discard; PAQI can therefore simplify to confidence-ranked Top-K retention.Source: §3 · p. 3

Budget sweep

  • At comparable average query counts, fixed Top-K matches PAQI’s overall AP on both datasets: AI-TOD-v2 AP 35.1 with mean |Q|=630 versus AP 35.0 with fixed K=600, and VisDrone AP 29.3 with mean |Q|=388 versus AP 29.3 with fixed K=400.Budget matching is approximate at the average-query level, not an exact per-image match.Source: §3 · pp. 3–4

Interpretation

  • PAQI overlaps substantially with confidence-ranked Top-|Q| proposals: mean overlap is 71.5% ± 8.1% on AI-TOD-v2 and 84.6% ± 1.9% on VisDrone.The paper also notes substantial encoder-proposal redundancy around foreground objects.Source: §3 · p. 4

Density-stratified analysis

  • The density-stratified figure plots AP against fixed decoder budget K within Minimal, Sparse, Moderate, and Dense buckets, marking PAQI’s mean query count and ±σ alongside budget-matched Top-K baselines.The bucket boundaries are defined by ground-truth object counts.Source: Figure 2 · p. 4

Selection versus budgeting

  • At fixed query count, DDQ-style spatially distinct selection and original Top-K achieve the same overall AP on AI-TOD-v2 and VisDrone.The two D-FINE-L variants used each initializer during training and inference.Source: §3 · p. 5

4 BUTR

BUTR keeps standard Top-K query initialization but makes its truncation point adaptive, using a non-learned budgeter to allocate decoder capacity from encoder-proposal confidences.

4.1 Preliminaries

  • BUTR sets Q = T_K(x)(P), adapting the query budget per image while retaining standard Top-K initialization without a learned predictor or post-Top-K selection pipeline.This is presented as a drop-in modification for two-stage DETRs.Source: §4.1 · p. 5

4.2 Budgeter: Estimating an Adaptive Query Budget

  • A reasonable budget should increase with image density and lean mildly toward sufficient capacity, because under-allocation is more harmful than slight over-allocation in the observed saturation regime.The budgeter need not recover exact object counts; it needs a reliable density ordering translated into a conservative budget.Source: §4.2 · pp. 5–6
  • BUTR sums proposal ranking scores as a density proxy, using each proposal’s maximum sigmoid-activated class confidence as its score.More high-scoring proposals indicate a likely denser image and a need for more decoder queries.Source: §4.2 · p. 6
  • The budget mapping is linear, K(x) = γe(x), with γ ≥ 1 controlling how conservatively the density signal becomes decoder capacity.The estimator correlates with object count on AI-TOD-v2 (Pearson r = 0.94, Spearman ρ = 0.82) and VisDrone (r = 0.82, ρ = 0.79; all p-values < 10−5).Source: §4.2 · p. 6
  • The budgeter adds no learnable parameters, auxiliary supervision, extra loss, or budgeter-specific optimization, and supports plug-in use without retraining.The host detector continues to use its standard detection objectives.Source: §4.2 · p. 6

5 Experiments

Across AI-TOD-v2 and VisDrone, BUTR improves on dynamic-query baselines in accuracy and efficiency, with density-stratified results highlighting gains in crowded scenes; evaluations also cover ablations and broader detection settings.

5.1 Setup

  • The comparisons use AI-TOD-v2 and VisDrone, report AP and efficiency measures, and align query-budget ranges and training schedules across methods where specified.The main comparisons use 800 × 800 inputs, with T4 TensorRT latency measured at batch size 1.Source: §5.1 · pp. 6–7

5.2 Comparisons

  • BUTR achieves higher AP than Dome-DETR on AI-TOD-v2 and VisDrone, with 1.6–1.9× faster T4 inference on AI-TOD-v2 and 1.5–1.8× faster T4 inference on VisDrone.On AI-TOD-v2, BUTR-L reaches 35.7 AP versus 35.2 for D-FINE-L, with 316 versus 600 average queries; on VisDrone, BUTR has a slight AP gain over D-FINE with 1.7–2.0 ms additional deployment latency.Source: Table 3 · p. 7, §5.2 · p. 7, Table 4 · p. 8

5.3 Density-Stratified Evaluation

  • On AI-TOD-v2, dynamic-query benefits are more pronounced on the extreme-density subset than in overall AP; BUTR remains competitive with Dome-DETR on APext and exceeds it on VisDrone APden across scales.D-FINE APext declines from 27.0 at S to 23.4 at M and 22.0 at L; BUTR-L also shows lower APext variability than Dome-DETR-L across the reported repeated and matched-seed runs.Source: §5.3 · p. 8, Figure 5 · p. 9

5.4 Ablations

  • BUTR is insensitive to the tested proposal-ranking function and robust to γ from 1.0–3.0; discrete budget bins achieve comparable accuracy but require manually designed thresholds.The default ranking proxy is Max-sigmoid, and the default safety factor is γ = 2.0.Source: §5.4 · p. 9, Table 5 · p. 9

5.5 Generality beyond Tiny Object Scenarios

  • On Sorghum-Grain-Count, BUTR-L matches or exceeds fixed-query baselines using 1100 queries while averaging 378; on COCO, plug-in budgeting reduces queries and GFLOPs while maintaining or slightly improving AP without retraining.The broader evaluations include dense agricultural imagery and fixed-checkpoint inference-time integration on COCO.Source: §5.5 · pp. 9–10, Table 6 · p. 10

6 Conclusion

Adaptive budgeting largely explains Dome-DETR’s accuracy in the evaluated settings, motivating BUTR’s per-image decoder-budget estimates from encoder confidences. BUTR improves overall accuracy and efficiency over Dome-DETR on AI-TOD-v2 and VisDrone, while COCO integration reduces queries and computation without degrading overall AP.

6 Conclusion

  • BUTR improves overall accuracy and efficiency over Dome-DETR on AI-TOD-v2 and VisDrone, while COCO integration reduces queries and computation without degrading overall AP.The COCO integration is inference-only.Source: §6 · p. 10
  • Adaptive budgeting largely explains Dome-DETR’s accuracy in the evaluated settings.BUTR estimates a per-image decoder budget from encoder confidences while retaining standard Top-K initialization.Source: §6 · p. 10
  • BUTR relies on encoder-confidence scale and tested domains; TensorRT synchronization overhead can prevent fewer queries from reducing latency.The high-density subsets contain only 26 AI-TOD-v2 images and 7 VisDrone images, making their AP sensitive to test-sample composition.Source: §6 · p. 10

A Supplementary Comparison with DDQ-Style Initialization

At fixed query counts, separately trained DDQ-style and original Top-K initialization achieve the same overall AP on AI-TOD-v2 and VisDrone. The supplementary material also describes BUTR’s budget-to-Top-K procedure and illustrates how filtering without SPLIT can discard proposals aligned with ground-truth objects.

A Supplementary Comparison with DDQ-Style Initialization

  • DDQ-style initialization retains a fixed-size, spatially distinct subset; the comparison uses only this mechanism, not DDQ’s inter-layer selection or loss designs.D-FINE is used as a base close to Dome-DETR after removing PAQI and associated components, with query count held fixed.Source: A Supplementary Comparison with… · p. 13
  • 35.2 AP on AI-TOD-v2 and 29.6 AP on VisDrone were achieved by both separately trained initializers, using fixed budgets of 600 and 400 queries, respectively.The comparison uses DDQ-style versus original Top-K initialization, with each initializer used during training and inference.Source: Table 7 · p. 13, A Supplementary Comparison with… · p. 13

B Algorithmic Description of BUTR

  • BUTR aggregates encoder-proposal ranking scores into a density proxy, maps it through a safety factor to a feasible query budget, then initializes queries with standard Top-K(x).It modifies only the decoder query budget using existing encoder outputs and confidences, without adding learned modules or changing the host detector’s other components.Source: B Algorithmic Description of BUTR · pp. 13–14

C Qualitative Examples of Erroneous Filtering

D Why Budget-Matched Top-K Reproduces PAQI

Budget-matched Top-K reproduces PAQI’s overall AP in part because their selected queries overlap substantially and encoder proposals are redundant around objects. Increasing the budget mainly adds proposal density rather than new object coverage, supporting a budgeting interpretation without making query identities interchangeable in every setting.

Overlap

  • PAQI–Top-|Q| mean overlap was 71.5% ± 8.1% on AI-TOD-v2 and 84.6% ± 1.9% on VisDrone, versus random baselines of 42.0% ± 15.3% and 77.7% ± 4.2%, respectively.The overlap analysis matches query count per image; the AP comparison uses fixed budgets close to PAQI’s average count.Source: D Why Budget-Matched Top-K Reproduces… · p. 14

Redundancy

  • Even at Top-300, most ground-truth objects are covered by nearby proposals; Top-600 mainly increases proposal density and redundancy rather than adding object coverage.Encoder proposals often provide multiple candidate locations for the same object, especially around foreground regions.Source: D Why Budget-Matched Top-K Reproduces… · p. 14, Figure 7 · p. 15

Interpretation

  • Together, overlap and proposal redundancy help explain why sufficient-capacity Top-K can closely reproduce PAQI’s overall AP, without showing that query identities are interchangeable in all settings.These observations support a budgeting interpretation, not a claim that selection is universally unnecessary.Source: D Why Budget-Matched Top-K Reproduces… · pp. 14–15

E Full Density-Stratified Results

The full density-stratified results show BUTR gains in multiple lower-density subsets as well as the high-density tail, but not in every model-scale–subset comparison.

E Full Density-Stratified Results

  • BUTR’s gains appear in multiple lower-density subsets as well as the high-density tail, though it does not outperform every baseline in every model-scale–subset comparison.These density-stratified results describe performance across the density distribution rather than decompose the overall AP gain.Source: E Full Density-Stratified Results · p. 15

F Stability of Tail Performance Across Training Runs

Across repeated runs on AI-TOD-v2, BUTR-L shows lower variation in extreme-subset AP than Dome-DETR-L under both the fixed-seed and multi-seed evaluations. The reported comparisons use matched training and evaluation settings and select checkpoints by overall AP.

Evaluation protocol

  • Both evaluations report mean and standard deviation over three runs, selecting each run’s checkpoint by overall AP before evaluating the same extreme subset.The fixed-seed protocol repeats seed 0; the multi-seed protocol matches seeds 0, 1, and 2.Source: F Stability of Tail Performance Across… · p. 15

Fixed-seed reproducibility

Multi-seed robustness

Interpretation

  • The authors attribute the observed stability advantage as possibly reflecting avoidance of errors from Dome-DETR’s learned spatial mask, while BUTR adjusts query count and retains confidence-ranked initialization.The text presents this explanation as a possibility, not a demonstrated cause.Source: F Stability of Tail Performance Across… · p. 17

G Supplementary Experiment on Dense Agricultural Imagery

On cropped Sorghum-Grain-Count imagery, BUTR-L provides supplementary evidence for confidence-based budgeting in a dense agricultural setting. Evaluation is limited to retained nonempty patches, rather than empty-background images or full-image counting after merging overlapping predictions.

Data preparation

  • The supplementary evaluation uses visually similar sorghum-grain instances outside the aerial domain, cropped from source images into overlapping 800 × 800 patches with stride 640.Only patches containing at least one annotated grain are retained; reported patch statistics are not counts of unique grains in source images.Source: G Supplementary Experiment on Dense… · p. 17, Table 10 · p. 17
  • The protocol does not establish performance on empty-background images or full-image counting after overlapping predictions are merged.The test split is reserved for evaluation, and evaluation is on retained nonempty crops.Source: G Supplementary Experiment on Dense… · p. 17

Comparison protocol and results

  • BUTR-L improves over Dome-DETR-L by 0.4 AP while using approximately one-third as many queries and training 2.2× faster.All models use matched training and evaluation settings with 800 × 800 inputs.Source: G Supplementary Experiment on Dense… · p. 17
  • 58.0 AP with an average of 378 queries: BUTR-L exceeds D-FINE-L and matches DEIM-L at the larger fixed budget, while using less training time.The fixed-budget comparisons use 400 and 1100 queries; the table reports no T4 latency for this experiment.Source: Table 11 · p. 17, G Supplementary Experiment on Dense… · p. 17

H T4 Deployment Latency Protocol

The latency protocol measures paired inference on a single NVIDIA T4 with batch size 1 and includes dynamic-budget computation and synchronization within the timing boundary. The paper also frames its broader impact as methodological and notes potential surveillance misuse of stronger dense-scene detection.

H T4 Deployment Latency Protocol

  • Latency measurements use a single NVIDIA Tesla T4, batch size 1, and three rounds of 1,000 warmups plus 1,000 ordered test images, with preprocessing and input transfer excluded.The timer includes model stages, post-processing, dynamic-budget computation, readback, synchronization, and shape handling.Source: H T4 Deployment Latency Protocol · p. 17
  • The layout rewrite’s benefit depends on the surrounding graph and engine boundaries, rather than establishing a universal speedup.Applying the rewrite to complete AI-TOD-v2 D-FINE-S/DEIM-S graphs raises latency from 12.14/12.23 ms to 15.17/15.32 ms.Source: H T4 Deployment Latency Protocol · p. 18
  • Official D-FINE-S/M/L/X COCO checkpoint measurements deviate from published values by no more than 5%, with a predeclared mean absolute relative deviation of 2.68%.The same physical T4 is used for paired COCO plug-in measurements.Source: H T4 Deployment Latency Protocol · p. 18
  • Reported AP comes from detection experiments, not a full-test FP16 deployment evaluation; AI-TOD-v2 DEIM-S timing remains provisional because of an unresolved deployment-accuracy mismatch.BUTR’s paired layout checks preserve per-image budgets and change subset AP by less than 0.04 points across both tiny-object datasets.Source: H T4 Deployment Latency Protocol · p. 18

I Broader Impact

  • The broader impact is primarily methodological: the authors propose density-stratified evaluation as complementary when overall AP obscures rare high-density cases and report COCO plug-in applicability.They also note that stronger detection in aerial or crowded scenes could be misused for intrusive surveillance or monitoring.Source: I Broader Impact · p. 18

NeurIPS Paper Checklist

The checklist reports that the paper states its main claims and supplies reproducibility details, while acknowledging limitations and potential societal impacts. It marks statistical significance reporting as absent because main detection results use standard single-run AP evaluations.

Claims and scope

  • The checklist says the abstract and introduction accurately reflect the mechanism analysis, budgeting view, BUTR framework, and evaluations on AI-TOD-v2, VisDrone, and COCO.The checklist justification says these claims are supported by diagnostic analyses, ablations, and experimental results.Source: NeurIPS Paper Checklist · p. 19

Limitations and theory

  • The checklist says the paper discusses limits including its two-stage DETR-style focus, encoder-confidence density proxy, and empirical evaluation across considered datasets and baselines.The checklist also notes that the work has no theoretical results, theorems, or formal proofs.Source: NeurIPS Paper Checklist · pp. 19–20

Reproducibility and experimental details

  • The paper reports datasets, splits, training schedules, query budgets, latency protocol, hardware, and implementation details, and provides code and reproduction instructions.The checklist marks experimental-detail and reproducibility disclosures positively.Source: NeurIPS Paper Checklist · pp. 20–21

Statistical significance

  • The checklist marks statistical-significance reporting as absent: main detection results are standard single-run COCO-style AP evaluations, while some diagnostics report means and standard deviations over images.The reported diagnostic examples include query counts, overlap ratios, and query-usage behavior.Source: NeurIPS Paper Checklist · pp. 21–22

Ethics and societal impact

  • The checklist reports use of public datasets and standard detection benchmarks, and identifies possible misuse of improved aerial or dense-scene detection for surveillance or monitoring.It states that the method is not deployed in a real-world system in this work.Source: NeurIPS Paper Checklist · pp. 22–23

Related papers

Loading 2610.33430v1…