PaperlayerView exact revision on arXiv

Source-linked AI summary

RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, Neehar Peri

arXiv:2511.09554v2cs.CV

TL;DR

Open-vocabulary detectors often struggle with out-of-distribution classes and domains, and fine-tuning heavy VLMs can reduce runtime efficiency and open-vocabulary generalization. RF-DETR uses a lightweight specialist detector with weight-sharing NAS to search accuracy-latency configurations after fine-tuning a pretrained base network on the target dataset. It improves over prior real-time methods on COCO and RF100-VL, including a first reported real-time result above 60 AP on COCO.

  • Problem

    Open-vocabulary detectors struggle with out-of-distribution classes, tasks, and imaging modalities, while fine-tuned heavy VLMs incur runtime and generalization tradeoffs.

    Source: §1 · p. 1

  • Method

    RF-DETR fine-tunes a pretrained base network on a target dataset, then uses weight-sharing NAS to evaluate thousands of accuracy-latency configurations without retraining.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    RF-DETR outperforms prior state-of-the-art real-time methods on COCO and RF100-VL, including 5% AP over D-FINE (nano) on COCO.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    RF-DETR transfers specialist detection to diverse target datasets and hardware platforms while providing accuracy-latency tradeoffs through NAS.

    Source: §1 · p. 2, §5 · p. 10

    Latency measurements vary by up to 0.1ms because TensorRT recompilation can produce different engine timings, so latencies are reported to one decimal place.

    Source: §4 · p. 10

Abstract

from arXiv · show

Open-vocabulary detectors achieve impressive performance on COCO, but often fail to generalize to real-world datasets with out-of-distribution classes not typically found in their pre-training. Rather than simply fine-tuning a heavy-weight vision-language model (VLM) for new domains, we introduce RF-DETR, a light-weight specialist detection transformer that discovers accuracy-latency Pareto curves for any target dataset with weight-sharing neural architecture search (NAS). Our approach fine-tunes a pre-trained base network on a target dataset and evaluates thousands of network configurations with different accuracy-latency tradeoffs without re-training. Further, we revisit the "tunable knobs" for NAS to improve the transferability of DETRs to diverse target domains. Notably, RF-DETR significantly improves over prior state-of-the-art real-time methods on COCO and Roboflow100-VL. RF-DETR (nano) achieves 48.0 AP on COCO, beating D-FINE (nano) by 5.3 AP at similar latency, and RF-DETR (2x-large) outperforms GroundingDINO (tiny) by 1.2 AP on Roboflow100-VL while running 20x as fast. To the best of our knowledge, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO. Our code is available at https://github.com/roboflow/rf-detr

1 INTRODUCTION

RF-DETR addresses poor out-of-distribution generalization and the efficiency gap between fine-tuned VLMs and specialist detectors. It combines internet-scale pre-training with weight-sharing NAS to specialize real-time detectors across target datasets and hardware.

  • Fine-tuned VLMs improve in-domain performance but incur runtime costs and lose open-vocabulary generalization, while specialist detectors remain faster but less accurate.Source: §1 · p. 1
  • RF-DETR uses end-to-end weight-sharing NAS to explore accuracy-latency tradeoffs after fully training a base model on the target dataset.Thousands of sub-networks can be evaluated without further fine-tuning.Source: §1 · p. 2
  • 5% AP improvement over D-FINE (nano) on COCO demonstrates RF-DETR’s real-time advantage over a prior specialist detector.Source: §1 · p. 2
  • RF-DETR introduces scheduler-free NAS-based detection and segmentation models, tunable NAS components, and standardized latency benchmarking.Source: §1 · p. 2

2 RELATED WORKS

Prior detection research spans accuracy-focused NAS, real-time detectors, and VLMs, but these approaches leave unresolved efficiency, latency reproducibility, or out-of-distribution generalization challenges. RF-DETR combines real-time DETR architectures with internet-scale priors while evaluating accuracy-latency frontiers across detection and segmentation tasks.

  • Hardware-aware NAS addresses the computational expense of early accuracy-maximizing searches by incorporating efficiency into architecture discovery.Source: §2 · p. 2
  • RF-DETR’s Pareto curves compare accuracy and latency on COCO detection, COCO segmentation, and RF100-VL, with RF100-VL results averaged across 100 datasets.Source: Figure 1 · p. 3
  • Modern DETRs remove NMS and anchor boxes, while RT-DETR and LW-DETR adapt DETR accuracy to real-time inference.Source: §2 · p. 3
  • VLMs provide internet-scale priors but can perform poorly on categories outside pre-training and are often too slow for real-time tasks.Source: §2 · p. 3
  • The architecture uses a pre-trained ViT backbone, interleaved windowed and non-windowed attention, and decoder-layer losses supporting decoder dropout at inference.Source: Figure 2 · p. 4

3 RF-DETR: WEIGHT-SHARING NAS WITH FOUNDATION MODELS

RF-DETR combines foundation-model initialization with scheduler-free, end-to-end weight-sharing NAS. Training samples many configurations so inference can select operating points that trade accuracy, latency, resolution, and detection capacity.

Incorporating Internet-Scale Priors

  • RF-DETR replaces CAEv2 with DINOv2 initialization, improving detection accuracy on small datasets while NAS offsets the backbone’s added latency.Source: §3 · p. 4

NAS Search Space

  • The search varies patch size, decoder layers, query tokens, image resolution, and attention windows to produce different accuracy-latency operating points.Source: Figure 3 · p. 5

End-to-End Neural Architecture Search

  • Weight-sharing NAS samples configurations during training, enabling thousands of detector sub-networks to share weights and acting as architecture augmentation.Source: §3 · p. 5
  • Dropping decoder layers and query tokens at inference reduces latency, while query selection adapts the maximum detections to target-dataset object statistics.Source: §3 · p. 5
  • Higher image resolution improves small-object detection, whereas lower resolution improves runtime; window counts trade accuracy, global mixing, and efficiency.Source: §3 · p. 6
  • NAS configurations are evaluated after base-model training, generally avoiding retraining, with modest additional fine-tuning benefits reported on RF100-VL.Source: §3 · p. 6

4 EXPERIMENTS

RF-DETR is evaluated on COCO and RF100-VL using standardized latency procedures, with experiments covering detection, segmentation, NAS, backbones, and pre-training. Across these evaluations, it achieves strong real-time performance while exposing benchmarking and measurement caveats.

Latency Evaluation

  • Latency comparisons require standardized procedures because power throttling, GPU overheating, NMS handling, and model-export choices can materially affect measurements.Buffering 200ms between forward passes improves reproducibility, but the procedure is not intended to measure sustained throughput; TensorRT measurements still vary by up to 0.1ms.Source: §4 · pp. 6, 7, 10, Table 1 · p. 7

COCO Detection

  • RF-DETR (nano) outperforms D-FINE (nano) and LW-DETR (tiny) by more than 5 AP on COCO detection.Its nano model also reaches parity with YOLOv8 and YOLOv11 medium models.Source: §4 · p. 7, Table 2 · p. 7

COCO Instance Segmentation

  • RF-DETR-Seg (nano) outperforms FastInst by 5.4% while running nearly ten times faster on COCO instance segmentation.The comparison includes practical mask-generation latency for YOLO models.Source: Table 3 · p. 8, §4 · p. 8

RF100-VL

  • RF-DETR (2x-large) outperforms GroundingDINO and LLMDet on RF100-VL while requiring only a fraction of their runtime.YOLOv8 and YOLOv11 consistently underperform DETR-based detectors on this benchmark, and scaling them does not improve performance.Source: §4 · p. 8, Table 4 · p. 9

Neural Architecture Search

  • Weight-sharing NAS improves over LW-DETR by 2% without increasing latency.The ablation also reports that DINOv2 improves performance over CAEv2 by 2%, while gentler hyperparameters initially reduce performance by 1%.Source: §4 · p. 8, Table 5 · p. 9

5 CONCLUSION

RF-DETR is a NAS-based specialist detector designed for diverse target datasets and hardware platforms, outperforming prior real-time methods while exposing COCO overfitting and latency-reporting concerns.

  • RF-DETR outperforms prior state-of-the-art real-time methods on COCO and RF100-VL.Source: §5 · p. 10
  • 5% AP improvement over D-FINE (nano) is reported on COCO.Source: §5 · p. 10
  • Current architectures, learning-rate schedulers, and augmentation schedulers are tailored to maximize COCO performance.Source: §5 · p. 10
  • Latency benchmarking varies substantially because of power throttling, motivating a standardized protocol for reproducibility.Source: §5 · p. 10

A IMPLEMENTATION DETAILS

RF-DETR uses weight-sharing NAS to train and search model configurations across accuracy-latency tradeoffs, including detection and segmentation variants. Its implementation evaluates Pareto-optimal configurations after training and standardizes latency measurement.

Training Hyperparamters

  • RF-DETR pre-trains detection and segmentation heads using Objects-365 images pseudo-labeled with SAM2.The training procedure uses a 1e-4 learning rate and batch size 128.Source: A IMPLEMENTATION DETAILS · p. 15

Latency Evaluation

  • Latency is measured on the same artifact used for accuracy evaluation, with CUDA graphs in TensorRT to standardize inference.CUDA graphs pre-queue kernels and benefit RT-DETR, LW-DETR, and RF-DETR.Source: A IMPLEMENTATION DETAILS · p. 15

Pareto-Optimal Model Configurations in COCO

Parameter Sampling Grid

  • RF-DETR samples configurations during training and performs grid search during inference to identify Pareto-optimal models.Decoder layers and queries are dropped only during inference; total training takes roughly two to four times longer than a non-NAS baseline.Source: A IMPLEMENTATION DETAILS · p. 15
  • The sampling grid includes 6 decoder layers and 300 object queries.Source: A IMPLEMENTATION DETAILS · p. 16

B ABLATION ON QUERY TOKENS AND DECODER LAYERS

The ablation studies test inference-time query dropping and decoder-layer reduction as ways to trade accuracy for latency without retraining. Query removal has little effect on performance, while removing decoder layers yields larger latency savings with a measurable accuracy cost.

Query Tokens

  • Dropping the 100 lowest-confidence queries does not significantly reduce performance and modestly improves latency across decoder-layer settings.Source: Figure 4 · p. 16
  • RF-DETR can drop queries at test time without retraining, addressing the computational cost of processing 300 queries when images contain fewer objects.Source: B ABLATION ON QUERY TOKENS AND DECODER… · p. 16

Decoder Layers

Benchmarking

  • FlopCounterMode closely matches custom FLOP benchmarking and is more reliable than CalFLOPs in the reported comparisons.LW-DETR’s FLOP count is roughly twice its originally reported result.Source: Table 9 · p. 17, C BENCHMARKING FLOPS · p. 17

Class Names

E BENCHMARKING LARGER MODEL VARIANTS

The larger-model benchmark compares two RF-DETR backbone families with D-FINE at matched latency. The DINOv2-B family scales more favorably at higher latencies, while RF-DETR (2x-large) exceeds 60 AP on COCO and beats D-FINE.

Model Families

COCO Larger Variants

  • 0.8 AP is the RF-DETR (2x-large) advantage over D-FINE, and it is the first real-time detector reported to surpass 60 AP on COCO.Source: Table 12 · p. 18

Scaling Comparison

F PER-KNOB SENSITIVITY ANALYSIS

RF-DETR generalizes across unseen resolution and patch-size configurations, while fine-tuning after NAS offers limited benefit on COCO, especially for larger models.

Per-knob sensitivity analysis

  • Unseen resolution and patch-size configurations closely follow the Pareto trends established by seen configurations during inference.This indicates graceful interpolation beyond configurations encountered during training.Source: F PER-KNOB SENSITIVITY ANALYSIS · p. 18, Figure 5 · p. 19

Fine-tuning after NAS

H IMPACT OF DATASET CHARACTERISTICS ON TUNABLE KNOBS

Dataset characteristics influence the architectures selected by NAS, while COCO-optimized fixed architectures transfer well to RF100-VL but improve further with dataset-specific search.

Impact of dataset characteristics

Fixed architecture transfer

Backbone transfer

K ANALYSIS ON BUFFERING

Buffering beyond 200 ms does not further stabilize latency measurements, although a 200 ms buffer after every forward pass substantially increases total inference time.

Analysis on buffering

  • Buffering beyond 200 ms does not provide additional stability to latency measurements.The analysis evaluates different buffering amounts between consecutive forward passes.Source: K ANALYSIS ON BUFFERING · p. 23, Table 20 · p. 23
  • Adding a 200 ms buffer after every forward pass considerably increases overall inference time.The authors suggest that future work should consider alternatives to buffering for power throttling.Source: K ANALYSIS ON BUFFERING · p. 23

L DISCUSSION ON NOTABLE DISCOVERED ARCHITECTURES

RF-DETR’s Pareto-optimal families use coordinated architectural scaling, with performance tied strongly to spatial-token count and dataset-dependent query choices.

Discovered architectures

  • All tunable knobs contribute to Pareto-optimal model families, supporting the chosen search space and motivating its possible expansion.The knobs are used jointly to define families with different accuracy-latency tradeoffs.Source: L DISCUSSION ON NOTABLE DISCOVERED… · p. 23
  • Patch size remains consistent within model families, while encoder and decoder compute are scaled jointly across patch size, windows, resolution, depth, and queries.Examples include patch sizes 16 for DINOv2-S RF-DETR, 20 for DINOv2-B RF-DETR, and 12 for DINOv2-S RF-DETR-Seg.Source: L DISCUSSION ON NOTABLE DISCOVERED… · p. 23
  • RF-DETR performance correlates more with total spatial locations than with resolution or patch size alone.Fixed-resolution families with unseen patch sizes produce results nearly identical to the Pareto-optimal family, unlike RF-DETR-Seg where resolution also changes segmentation feature resolution.Source: L DISCUSSION ON NOTABLE DISCOVERED… · p. 23
  • Most Pareto-optimal RF-DETR models use 2 windows, unlike LW-DETR’s best-performing 4-window configuration, partly because RF-DETR duplicates class tokens across windows.Additional windows reduce runtime efficiency for RF-DETR’s DINOv2 backbone.Source: L DISCUSSION ON NOTABLE DISCOVERED… · pp. 23–24
  • Low-latency RF100-VL models tend to use fewer queries than COCO models at equivalent latency, consistent with fewer objects per image in RF100-VL.This connects dataset object density with the discovered decoder configuration.Source: L DISCUSSION ON NOTABLE DISCOVERED… · p. 24

Visualized predictions

  • RF-DETR (nano) produces fewer false positives, while RF-DETR-Seg (nano) produces more precise object boundaries than the compared baselines.The comparisons are visualized against LW-DETR (tiny) and YOLOv11 (nano), respectively.Source: M VISUALIZING MODEL PREDICTIONS · p. 24, Figure 8 · p. 24

Related papers

Loading 2511.09554v2…