PaperlayerView exact revision on arXiv

Source-linked AI summary

RF-DETR: Neural Architecture Search for Real-Time Detection Transformers

Isaac Robinson, Peter Robicheaux, Matvei Popov, Deva Ramanan, Neehar Peri

arXiv:2511.09554v2cs.CV

TL;DR

Open-vocabulary detectors often struggle with out-of-distribution classes and domains, and fine-tuning heavy VLMs can reduce runtime efficiency and open-vocabulary generalization. RF-DETR uses a lightweight specialist detector with weight-sharing NAS to search accuracy-latency configurations after fine-tuning a pretrained base network on the target dataset. It improves over prior real-time methods on COCO and RF100-VL, including a first reported real-time result above 60 AP on COCO.

  • Problem

    Open-vocabulary detectors struggle with out-of-distribution classes, tasks, and imaging modalities, while fine-tuned heavy VLMs incur runtime and generalization tradeoffs.

  • Method

    RF-DETR fine-tunes a pretrained base network on a target dataset, then uses weight-sharing NAS to evaluate thousands of accuracy-latency configurations without retraining.

  • Results

    RF-DETR outperforms prior state-of-the-art real-time methods on COCO and RF100-VL, including 5% AP over D-FINE (nano) on COCO.

  • Takeaways & Limitations

    RF-DETR transfers specialist detection to diverse target datasets and hardware platforms while providing accuracy-latency tradeoffs through NAS.

  • Takeaways & Limitations

    Latency measurements vary by up to 0.1ms because TensorRT recompilation can produce different engine timings, so latencies are reported to one decimal place.

Abstract

from arXiv · show

Open-vocabulary detectors achieve impressive performance on COCO, but often fail to generalize to real-world datasets with out-of-distribution classes not typically found in their pre-training. Rather than simply fine-tuning a heavy-weight vision-language model (VLM) for new domains, we introduce RF-DETR, a light-weight specialist detection transformer that discovers accuracy-latency Pareto curves for any target dataset with weight-sharing neural architecture search (NAS). Our approach fine-tunes a pre-trained base network on a target dataset and evaluates thousands of network configurations with different accuracy-latency tradeoffs without re-training. Further, we revisit the "tunable knobs" for NAS to improve the transferability of DETRs to diverse target domains. Notably, RF-DETR significantly improves over prior state-of-the-art real-time methods on COCO and Roboflow100-VL. RF-DETR (nano) achieves 48.0 AP on COCO, beating D-FINE (nano) by 5.3 AP at similar latency, and RF-DETR (2x-large) outperforms GroundingDINO (tiny) by 1.2 AP on Roboflow100-VL while running 20x as fast. To the best of our knowledge, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO. Our code is available at https://github.com/roboflow/rf-detr

1 INTRODUCTION

RF-DETR addresses poor out-of-distribution generalization and the efficiency gap between fine-tuned VLMs and specialist detectors. It combines internet-scale pre-training with weight-sharing NAS to specialize real-time detectors across target datasets and hardware.

  • Fine-tuned VLMs improve in-domain performance but incur runtime costs and lose open-vocabulary generalization, while specialist detectors remain faster but less accurate.
  • RF-DETR uses end-to-end weight-sharing NAS to explore accuracy-latency tradeoffs after fully training a base model on the target dataset.Thousands of sub-networks can be evaluated without further fine-tuning.
  • 5% AP improvement over D-FINE (nano) on COCO demonstrates RF-DETR’s real-time advantage over a prior specialist detector.
  • RF-DETR introduces scheduler-free NAS-based detection and segmentation models, tunable NAS components, and standardized latency benchmarking.

2 RELATED WORKS

Prior detection research spans accuracy-focused NAS, real-time detectors, and VLMs, but these approaches leave unresolved efficiency, latency reproducibility, or out-of-distribution generalization challenges. RF-DETR combines real-time DETR architectures with internet-scale priors while evaluating accuracy-latency frontiers across detection and segmentation tasks.

  • Hardware-aware NAS addresses the computational expense of early accuracy-maximizing searches by incorporating efficiency into architecture discovery.
  • RF-DETR’s Pareto curves compare accuracy and latency on COCO detection, COCO segmentation, and RF100-VL, with RF100-VL results averaged across 100 datasets.
  • Modern DETRs remove NMS and anchor boxes, while RT-DETR and LW-DETR adapt DETR accuracy to real-time inference.
  • VLMs provide internet-scale priors but can perform poorly on categories outside pre-training and are often too slow for real-time tasks.
  • The architecture uses a pre-trained ViT backbone, interleaved windowed and non-windowed attention, and decoder-layer losses supporting decoder dropout at inference.

3 RF-DETR: WEIGHT-SHARING NAS WITH FOUNDATION MODELS

RF-DETR combines foundation-model initialization with scheduler-free, end-to-end weight-sharing NAS. Training samples many configurations so inference can select operating points that trade accuracy, latency, resolution, and detection capacity.

  • Incorporating Internet-Scale Priors: RF-DETR replaces CAEv2 with DINOv2 initialization, improving detection accuracy on small datasets while NAS offsets the backbone’s added latency.
  • NAS Search Space: The search varies patch size, decoder layers, query tokens, image resolution, and attention windows to produce different accuracy-latency operating points.
  • End-to-End Neural Architecture Search: Weight-sharing NAS samples configurations during training, enabling thousands of detector sub-networks to share weights and acting as architecture augmentation.
  • Dropping decoder layers and query tokens at inference reduces latency, while query selection adapts the maximum detections to target-dataset object statistics.
  • Higher image resolution improves small-object detection, whereas lower resolution improves runtime; window counts trade accuracy, global mixing, and efficiency.
  • NAS configurations are evaluated after base-model training, generally avoiding retraining, with modest additional fine-tuning benefits reported on RF100-VL.

4 EXPERIMENTS

RF-DETR is evaluated on COCO and RF100-VL using standardized latency procedures, with experiments covering detection, segmentation, NAS, backbones, and pre-training. Across these evaluations, it achieves strong real-time performance while exposing benchmarking and measurement caveats.

  • Latency Evaluation: Latency comparisons require standardized procedures because power throttling, GPU overheating, NMS handling, and model-export choices can materially affect measurements.Buffering 200ms between forward passes improves reproducibility, but the procedure is not intended to measure sustained throughput; TensorRT measurements still vary by up to 0.1ms.
  • COCO Detection: RF-DETR (nano) outperforms D-FINE (nano) and LW-DETR (tiny) by more than 5 AP on COCO detection.Its nano model also reaches parity with YOLOv8 and YOLOv11 medium models.
  • COCO Instance Segmentation: RF-DETR-Seg (nano) outperforms FastInst by 5.4% while running nearly ten times faster on COCO instance segmentation.The comparison includes practical mask-generation latency for YOLO models.
  • RF100-VL: RF-DETR (2x-large) outperforms GroundingDINO and LLMDet on RF100-VL while requiring only a fraction of their runtime.YOLOv8 and YOLOv11 consistently underperform DETR-based detectors on this benchmark, and scaling them does not improve performance.
  • Neural Architecture Search: Weight-sharing NAS improves over LW-DETR by 2% without increasing latency.The ablation also reports that DINOv2 improves performance over CAEv2 by 2%, while gentler hyperparameters initially reduce performance by 1%.

5 CONCLUSION

RF-DETR is a NAS-based specialist detector designed for diverse target datasets and hardware platforms, outperforming prior real-time methods while exposing COCO overfitting and latency-reporting concerns.

  • RF-DETR outperforms prior state-of-the-art real-time methods on COCO and RF100-VL.
  • 5% AP improvement over D-FINE (nano) is reported on COCO.
  • Current architectures, learning-rate schedulers, and augmentation schedulers are tailored to maximize COCO performance.
  • Latency benchmarking varies substantially because of power throttling, motivating a standardized protocol for reproducibility.

A IMPLEMENTATION DETAILS

RF-DETR uses weight-sharing NAS to train and search model configurations across accuracy-latency tradeoffs, including detection and segmentation variants. Its implementation evaluates Pareto-optimal configurations after training and standardizes latency measurement.

  • Training Hyperparamters: RF-DETR pre-trains detection and segmentation heads using Objects-365 images pseudo-labeled with SAM2.The training procedure uses a 1e-4 learning rate and batch size 128.
  • Latency Evaluation: Latency is measured on the same artifact used for accuracy evaluation, with CUDA graphs in TensorRT to standardize inference.CUDA graphs pre-queue kernels and benefit RT-DETR, LW-DETR, and RF-DETR.
  • Pareto-Optimal Model Configurations in COCO: Pareto-optimal RF-DETR and RF-DETR-Seg configurations are reported for COCO detection and segmentation.
  • Parameter Sampling Grid: RF-DETR samples configurations during training and performs grid search during inference to identify Pareto-optimal models.Decoder layers and queries are dropped only during inference; total training takes roughly two to four times longer than a non-NAS baseline.
  • Parameter Sampling Grid: The sampling grid includes 6 decoder layers and 300 object queries.

B ABLATION ON QUERY TOKENS AND DECODER LAYERS

The ablation studies test inference-time query dropping and decoder-layer reduction as ways to trade accuracy for latency without retraining. Query removal has little effect on performance, while removing decoder layers yields larger latency savings with a measurable accuracy cost.

  • Query Tokens: Dropping the 100 lowest-confidence queries does not significantly reduce performance and modestly improves latency across decoder-layer settings.
  • Query Tokens: RF-DETR can drop queries at test time without retraining, addressing the computational cost of processing 300 queries when images contain fewer objects.
  • Decoder Layers: 10% lower latency accompanies elimination of the final decoder layer, with only a 2 mAP performance drop.
  • Decoder Layers: Removing all decoder layers eliminates cross-attention and query self-attention, producing a substantially faster model resembling a single-stage YOLO-style detector without NMS.
  • Benchmarking: FlopCounterMode closely matches custom FLOP benchmarking and is more reliable than CalFLOPs in the reported comparisons.LW-DETR’s FLOP count is roughly twice its originally reported result.
  • Class Names: Fine-tuning GroundingDINO with class names provides no significant benefit over class indices, suggesting diminished impact from internet-scale pre-training after fine-tuning.

E BENCHMARKING LARGER MODEL VARIANTS

The larger-model benchmark compares two RF-DETR backbone families with D-FINE at matched latency. The DINOv2-B family scales more favorably at higher latencies, while RF-DETR (2x-large) exceeds 60 AP on COCO and beats D-FINE.

  • Model Families: RF-DETR model families are built from DINOv2-S and DINOv2-B backbones, with scaling strategies discovered through grid-based NAS rather than hand-designed variants.
  • COCO Larger Variants: 0.8 AP is the RF-DETR (2x-large) advantage over D-FINE, and it is the first real-time detector reported to surpass 60 AP on COCO.
  • Scaling Comparison: DINOv2-B RF-DETR models show a narrowing performance gap against D-FINE as latency increases, indicating better scalability than the DINOv2-S family.
  • Scaling Comparison: The DINOv2-S family initially surpasses D-FINE in mAP@50:95 but loses this advantage at larger model sizes.This suggests its scaling strategy is less effective than D-FINE’s manual design.

F PER-KNOB SENSITIVITY ANALYSIS

RF-DETR generalizes across unseen resolution and patch-size configurations, while fine-tuning after NAS offers limited benefit on COCO, especially for larger models.

  • Per-knob sensitivity analysis: Unseen resolution and patch-size configurations closely follow the Pareto trends established by seen configurations during inference.This indicates graceful interpolation beyond configurations encountered during training.
  • Fine-tuning after NAS: Fine-tuning after NAS provides limited benefit for COCO detection and segmentation, particularly for larger models.The authors attribute this to NAS architecture augmentation acting as a strong regularizer that is removed during fine-tuning.
  • Fine-tuning after NAS: Reducing the number of NAS configurations or training beyond 100 epochs may improve performance when weight-sharing NAS models fail to converge.This is presented as a proposed remedy for non-convergent training cases.

H IMPACT OF DATASET CHARACTERISTICS ON TUNABLE KNOBS

Dataset characteristics influence the architectures selected by NAS, while COCO-optimized fixed architectures transfer well to RF100-VL but improve further with dataset-specific search.

  • Impact of dataset characteristics: Dataset characteristics and model parameters exhibit non-linear relationships, with regression trends used to explain general correlations.The analysis considers object size, spatial locations, decoder layers, windows, classes, annotations, objects per image, and queries.
  • Fixed architecture transfer: COCO-optimized fixed architectures perform well on RF100-VL without additional dataset-specific NAS, but dataset-specific NAS provides significant further gains.The fixed RF-DETR architecture also achieves the best performance among prior real-time models in the reported comparison.
  • Backbone transfer: The backbone ablation on RF20-VL preserves the trends reported in the main paper.This reproduces the main backbone findings on another dataset.

K ANALYSIS ON BUFFERING

Buffering beyond 200 ms does not further stabilize latency measurements, although a 200 ms buffer after every forward pass substantially increases total inference time.

  • Analysis on buffering: Buffering beyond 200 ms does not provide additional stability to latency measurements.The analysis evaluates different buffering amounts between consecutive forward passes.
  • Analysis on buffering: Adding a 200 ms buffer after every forward pass considerably increases overall inference time.The authors suggest that future work should consider alternatives to buffering for power throttling.

L DISCUSSION ON NOTABLE DISCOVERED ARCHITECTURES

RF-DETR’s Pareto-optimal families use coordinated architectural scaling, with performance tied strongly to spatial-token count and dataset-dependent query choices.

  • Discovered architectures: All tunable knobs contribute to Pareto-optimal model families, supporting the chosen search space and motivating its possible expansion.The knobs are used jointly to define families with different accuracy-latency tradeoffs.
  • Discovered architectures: Patch size remains consistent within model families, while encoder and decoder compute are scaled jointly across patch size, windows, resolution, depth, and queries.Examples include patch sizes 16 for DINOv2-S RF-DETR, 20 for DINOv2-B RF-DETR, and 12 for DINOv2-S RF-DETR-Seg.
  • Discovered architectures: RF-DETR performance correlates more with total spatial locations than with resolution or patch size alone.Fixed-resolution families with unseen patch sizes produce results nearly identical to the Pareto-optimal family, unlike RF-DETR-Seg where resolution also changes segmentation feature resolution.
  • Discovered architectures: Most Pareto-optimal RF-DETR models use 2 windows, unlike LW-DETR’s best-performing 4-window configuration, partly because RF-DETR duplicates class tokens across windows.Additional windows reduce runtime efficiency for RF-DETR’s DINOv2 backbone.
  • Discovered architectures: Low-latency RF100-VL models tend to use fewer queries than COCO models at equivalent latency, consistent with fewer objects per image in RF100-VL.This connects dataset object density with the discovered decoder configuration.
  • Visualized predictions: RF-DETR (nano) produces fewer false positives, while RF-DETR-Seg (nano) produces more precise object boundaries than the compared baselines.The comparisons are visualized against LW-DETR (tiny) and YOLOv11 (nano), respectively.
Loading 2511.09554v2…