PaperlayerView exact revision on arXiv

Source-linked AI summary

If It Moves, Radar Knows: A Physics-Aware Radar Transformer for Class-Agnostic Moving-Object Detection

Yinghao Sun, Shuguang Li, Jinliang Shao, Tieshan Li

arXiv:2609.02289v1cs.CV

TL;DR

Closed-set detectors can miss rare moving objects, while sparse radar returns make complete 3D box recovery difficult despite radar’s category-independent motion cues. PART predicts object existence, surface points, and velocity from sparse radar using observation-driven queries, physics-guided association, and uncertainty-aware supervision. On nuScenes it achieves CA-AP 0.8827, mASTE 0.3188 m, mAVE 0.8084 m/s, and 0.9203 recall on rare and safety-relevant categories.

  • Problem

    Closed-set annotations can miss rare moving objects, while sparse, noisy radar returns hinder complete 3D box detection despite useful Doppler motion cues.

    Source: §I · p. 1

  • Method

    PART is a fully sparse radar-only detector that predicts existence, surface point, and velocity, using position–velocity cluster queries, Doppler/RCS-guided attention, and uncertainty-aware supervision.

    Source: §I · p. 1

  • Results

    CA-AP 0.8827, mASTE 0.3188 m, and mAVE 0.8084 m/s on nuScenes; rare and safety-relevant-category recall reaches 0.9203.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    A representative surface location and velocity remain useful for motion reasoning and collision avoidance when complete box geometry is difficult to recover.

    Source: §I · p. 1

    Tangential-direction motion remains outside the present scope, as do semantic classification and complete 3D box estimation.

    Source: §V · p. 8

Abstract

from arXiv · show

Detectors trained on closed-set annotations can miss rare moving objects outside the training taxonomy. Automotive radar provides category-independent Doppler motion cues and is less affected by adverse illumination and weather, but sparse, noisy returns hinder class-aware 3D box detection. Surface location and velocity remain useful for motion reasoning and collision avoidance when full box geometry is difficult to recover. We present the Physics-Aware Radar Transformer (PART), a fully sparse radar-only detector that predicts existence confidence, a representative surface point, and 2D ground-plane velocity for each moving-object hypothesis. Doppler-Aware Query Initialization (DAQI) replaces scene-independent learned queries with input-dependent proposals by clustering radar returns in position and velocity, easing query-object assignment in sparse scenes. Physics-Guided Cross-Attention (PGCA) incorporates radial-Doppler consistency and radar cross section (RCS) into query-point association. Uncertainty-aware supervision randomly masks ground-truth objects and assigns soft existence targets to ambiguous radar-supported queries, reducing reliance on exhaustive annotations. With only 1.1 million parameters, PART achieves a class-agnostic average precision (CA-AP) of 0.8827, a mean average surface translation error (mASTE) of 0.3188 m, and a mean average velocity error (mAVE) of 0.8084 m/s on nuScenes. It attains 0.9203 recall on rare and safety-relevant categories excluded from the standard evaluation and remains effective at night, in rain, and under severe occlusion. Inspection of apparent false positives shows that some predictions correspond to moving objects absent from the nuScenes annotations. Code and pretrained model weights will be publicly available at https://github.com/sunyinghao-uestc/PART.

I. INTRODUCTION

Closed-set detectors can miss rare moving objects, while radar’s motion cues remain difficult to use for full 3D boxes because returns are sparse and noisy. PART uses sparse radar evidence to predict object existence, surface location, and velocity, with query initialization, physics-guided association, and uncertainty-aware supervision.

I. INTRODUCTION

  • Closed-set detectors depend on training coverage, while radar offers category-independent Doppler cues but sparse, noisy returns hinder complete 3D box recovery.Surface location and velocity remain useful for motion reasoning and collision avoidance when full geometry is difficult to recover.Source: §I · p. 1
  • PART predicts existence, a representative surface point, and 2D ground-plane velocity; masked objects and graded existence targets reduce hard-negative treatment of ambiguous radar-supported queries.This supervision reduces reliance on exhaustive moving-object annotations.Source: §I · pp. 1–2
  • With 1.1 million parameters, PART is a fully sparse transformer that operates on radar points without dense bird’s-eye-view features.The paper frames it as a compact alternative to camera- and lidar-based detectors.Source: §I · p. 1
  • DAQI forms input-dependent object queries from radar clusters coherent in position and velocity, grounding proposals in current observations rather than scene-independent learned queries.This reduces the burden of learning query–object correspondence from scratch.Source: §I · p. 1
  • PGCA constrains proposed 2D velocity with measured radial Doppler speeds and incorporates point- and cluster-level RCS into attention to distinguish object returns from noise and multipath.Source: §I · pp. 1–2
  • CA-AP 0.8827, mASTE 0.3188 m, and mAVE 0.8084 m/s on nuScenes; rare and safety-relevant-category recall reaches 0.9203.PART also remains effective under adverse weather, low illumination, and severe occlusion.Source: §I · p. 2

II. RELATED WORK

Prior 3D detection primarily relies on camera or lidar geometry, while radar is commonly auxiliary and radar-only methods often retain closed-set box formulations. Existing transformer designs also leave sparse-scene query guidance and radar-specific physical cues underused.

A. Radar-Based 3D Object Detection

  • Most 3D detectors use lidar, cameras, or their fusion, while radar typically supplies auxiliary range and Doppler cues.Detection can therefore remain dependent on camera or lidar when adverse weather or poor illumination degrades the primary input.Source: A. Radar-Based 3D Object Detection · p. 2
  • Radar-only methods adapt lidar-oriented graph or hybrid-grid designs, aggregate temporal features, or distill lidar features into radar-only students.Radar returns are sparse, noisy, and susceptible to multipath reflections.Source: A. Radar-Based 3D Object Detection · p. 2
  • These approaches largely retain closed-set 3D boxes or lidar supervision, leaving Doppler geometry and RCS underused for target association.Source: A. Radar-Based 3D Object Detection · p. 2
  • Long-tail targets can form coherent radar clusters, but noise and multipath can produce similar patterns that closed-set supervision does not explicitly resolve.Source: A. Radar-Based 3D Object Detection · pp. 2–3

B. Transformer-Based 3D Detectors

  • DETR uses learned queries with bipartite matching, but scene-independent queries provide limited spatial guidance in sparse 3D scenes.Source: B. Transformer-Based 3D Detectors · p. 3
  • Lidar transformers adapt attention to sparse representations, yet global point attention can be costly and their designs omit radar-specific Doppler constraints; voxelization may fragment sparse object returns.Source: B. Transformer-Based 3D Detectors · p. 3

III. PART

PART encodes sparse radar points and initializes queries from spatially and kinematically coherent clusters. Decoder layers refine those queries through self-attention and physics-guided cross-attention, producing class-agnostic object predictions and auxiliary geometric supervision.

III. PART

  • PART uses a fully sparse encoder, DAQI cluster queries, and decoder layers with query self-attention and PGCA; its outputs include existence, surface point, velocity, and auxiliary center predictions.Training combines point-wise surface supervision with uncertainty-aware existence targets.Source: §III · p. 3

A. Point Embedding and Encoder

  • The input aggregates six radar sweeps, compensates for ego motion, and removes returns below 0.5 m/s or outside the ±54 m sensing region.Source: A. Point Embedding and Encoder · p. 3
  • After filtering, most nuScenes samples contain fewer than 300 radar-point tokens and none exceed 800, keeping global point-wise attention computationally practical.Source: A. Point Embedding and Encoder · p. 3
  • Each point has 16 features, mapped by a three-layer MLP to a 128-dimensional embedding; Doppler and RCS supply physical inputs to PGCA.Source: A. Point Embedding and Encoder · p. 3
  • The encoder stacks three blocks of SECA channel attention, 2D RoPE self-attention, and feed-forward processing; encoded point features become decoder keys and values.Source: A. Point Embedding and Encoder · p. 3

B. Doppler-Aware Query Initialization (DAQI)

  • DAQI applies DBSCAN in [x, y, vx, vy] space to clusters of at least three radar points with similar positions and ego-compensated velocities.Cluster centroids and pooled encoder features initialize query positions and features.Source: B. Doppler-Aware Query Initialization… · p. 3
  • Zero-initialized offset-head final layers make initial position predictions coincide with each cluster centroid.Source: B. Doppler-Aware Query Initialization… · pp. 3–4
  • DAQI uses 100 queries; training adds balanced random queries away from proposals and matching regions, while inference retains cluster-derived queries that remain proposals rather than confirmed objects.Radar noise and multipath can also form coherent groups, so PGCA provides additional Doppler and RCS evidence.Source: B. Doppler-Aware Query Initialization… · p. 4

C. Physics-Guided Cross-Attention (PGCA)

PGCA associates object queries with radar points using feature, Doppler, and RCS evidence. It checks whether proposed ground-plane motion agrees with measured radial motion and uses RCS in attention projections.

C. Physics-Guided Cross-Attention (PGCA)

  • PGCA associates each query with encoded radar points using feature, Doppler, and RCS evidence.Each decoder layer first applies 2D RoPE multi-head self-attention to active queries.Source: C. Physics-Guided Cross-Attention (PGCA) · p. 4
  • Because radar measures motion only along the line of sight, PGCA compares the signed radial components of a query’s proposed velocity and a point’s ego-compensated velocity.A large residual indicates Doppler-inconsistent motion and reduces the query-point association.Source: C. Physics-Guided Cross-Attention (PGCA) · p. 4
  • Cluster-level RCS statistics condition the query projection, while each point’s RCS conditions its key and value projections.This lets PGCA learn RCS compatibility from annotated radar returns without imposing a fixed scattering rule.Source: C. Physics-Guided Cross-Attention (PGCA) · p. 4
  • The attended radar message updates each query through a residual connection, layer normalization, and a feed-forward network before final-layer queries reach the prediction heads.The message combines outputs across radar points and attention heads.Source: C. Physics-Guided Cross-Attention (PGCA) · p. 4

D. Prediction Heads and Loss

After three decoder layers, PART predicts class-agnostic existence, a surface point, a center, and ground-plane velocity. Query and point surface supervision use nearest locations on box perimeters, alongside query-wise regression and an auxiliary point-wise surface head.

D. Prediction Heads and Loss

  • After three decoder layers, four query-wise heads predict existence, a surface point, a center, and ground-plane velocity.Surface location is the primary localization output because radar returns commonly originate from object surfaces; the center head supports center-based protocols, and the velocity head refines PGCA’s proposal.Source: D. Prediction Heads and Loss · pp. 4–5
  • Query-wise surface targets project references to the nearest ground-truth box perimeter location, and only DBSCAN-initialized queries can become hard positives.Hungarian matching and a per-box top-K rule select positive query–box pairs; the same perimeter construction defines query-wise and point-wise surface targets.Source: D. Prediction Heads and Loss · p. 5, Figure 5 · p. 5
  • An auxiliary point-wise surface head predicts offsets from radar-point positions, embedding them into encoder features to provide local surface geometry for query decoding.The encoder associates each radar point with a compatible ground-truth box using angular, surface, velocity-direction, and radial-speed consistency.Source: D. Prediction Heads and Loss · p. 5
  • The complete objective combines existence supervision with three query-wise regression terms, while only queries in Qp contribute to surface, center, and velocity losses.The regression terms share a common form and use scalar loss weights.Source: D. Prediction Heads and Loss · p. 5

E. Uncertainty-Aware Supervision

Uncertainty-aware supervision reduces dependence on exhaustive moving-object annotations by randomly masking ground-truth boxes and softening existence targets for ambiguous radar-supported queries. Only hard-positive queries receive query-wise regression losses.

E. Uncertainty-Aware Supervision

  • Randomly masking each ground-truth box with probability 0.5 exposes PART to radar-supported patterns without hard positive labels, encouraging existence predictions to rely on physical radar evidence.The method addresses unmatched radar-supported queries that may correspond to rare or previously unseen objects rather than clutter.Source: E. Uncertainty-Aware Supervision · p. 5
  • Existence targets are 1 for queries matched to unmasked boxes, 0.75 for masked-box matches, and 0.5 for qualifying unmatched DBSCAN queries.The 0.75 target preserves evidence for a masked object without enabling regression; 0.5 retains ambiguity between an unannotated object and radar clutter.Source: E. Uncertainty-Aware Supervision · p. 5
  • The existence head uses sigmoid focal loss, but only queries in Qp contribute to query-wise surface, center, and velocity losses.The focal-loss settings are γ = 2.0 and α = 0.25.Source: E. Uncertainty-Aware Supervision · p. 5

A. Experimental Protocol and Setup

Evaluation uses the official nuScenes validation split under a class-agnostic moving-object protocol, with static targets excluded. The section also specifies training settings and radar-point perturbations.

A. Experimental Protocol and Setup

  • Evaluation uses the official nuScenes validation split within ±54 m in the RADAR FRONT frame and includes ground-truth objects moving faster than 0.5 m/s.Static targets are excluded, and all methods follow the same class-agnostic protocol.Source: A. Experimental Protocol and Setup · p. 5
  • CA-AP is computed from the precision–recall curve, while mASTE measures matched surface localization and mATE and mAVE measure center and ground-plane velocity errors.For mASTE, PART uses distance to the box perimeter; box-based baselines use the nearest predicted and ground-truth keypoints among four BEV corners and four edge centers.Source: A. Experimental Protocol and Setup · p. 5
  • PART is trained for 30 epochs with Adam and a one-cycle schedule, using batch size eight per GPU and peak learning rate 1.2 × 10^-3.Training includes GT sampling, flipping, rotation, and scaling; sampled objects are radially perturbed, up to 50% of returns are dropped, and points receive radial Gaussian noise with σ = 0.5 m.Source: A. Experimental Protocol and Setup · p. 6

B. Comparison with Existing 3D Detectors

Overall, PART is near the best CA-AP and has the lowest mASTE; it also leads CA-AP across night and rain subsets and reports high recall on several excluded categories.

B. Comparison with Existing 3D Detectors

  • 0.8827 CA-AP is within 0.0050 of the best, while 0.3188 m mASTE is lowest; PART uses 1.1M parameters.Against RadarDistill, PART improves CA-AP by 0.1143 and reduces mASTE by 58.9%, using approximately 97% fewer parameters and no lidar supervision. Its mAVE is 0.8084 m/s, compared with 0.7499 m/s for RadarDistill.Source: Table I · p. 6, B. Comparison with Existing 3D Detectors · p. 6
  • 1.14, 0.52, and 0.29 percentage-point CA-AP leads over the strongest competitor in Night, Rainy, and Night+Rainy, respectively.PART also has the lowest mASTE in Rainy and Night+Rainy, and the lowest mAVE in Night and Night+Rainy; in Night+Rainy it reduces mASTE by 32.4% and mAVE by 55.1% against the stated competitors.Source: B. Comparison with Existing 3D Detectors · p. 6
  • 95.71%, 95.77%, 80.39%, and 92.31% recall for police vehicles, wheelchair users, personal-mobility users, and animals exceed strongest alternatives by 10.00, 7.69, 33.33, and 30.77 points.The diagnostic pooled training and validation splits; conditional ASTE and AVE are unranked because recall gaps yield different detected subsets.Source: B. Comparison with Existing 3D Detectors · p. 6

C. Ablation and Additional Analyses

Ablations show DAQI has the largest impact and that uncertainty-aware supervision supports generalization beyond the training taxonomy. Additional analysis identifies unannotated moving objects among apparent false positives.

C. Ablation and Additional Analyses

  • Without uncertainty-aware supervision, CA-AP falls to 0.7948 and special-category recall to 0.3899; PGCA removal degrades all metrics, while PWSH removal worsens errors and special recall.Removing PWSH leaves CA-AP unchanged.Source: C. Ablation and Additional Analyses · pp. 6–7
  • Table III covers rare and safety-relevant categories; gray cells mean zero recall despite eligible ground-truth instances.N_GT denotes the number of eligible ground-truth boxes in each category.Source: Table III · p. 7, C. Ablation and Additional Analyses · p. 7
  • Removing DAQI causes the largest degradation: CA-AP drops 0.2629, mASTE rises 0.3589 m, and mAVE rises 0.4964 m/s.The DAQI-free variant is reported at score threshold 0.2 because it yields no valid detections at 0.5.Source: C. Ablation and Additional Analyses · pp. 6–7, Table IV · p. 7
  • Some apparent false positives are visually identifiable unannotated moving objects; the authors say this suggests response to motion-consistent evidence beyond the training-label distribution.Source: C. Ablation and Additional Analyses · p. 7

V. CONCLUSION

PART is a lightweight radar-only detector that estimates moving-object existence, a representative surface point, and ground-plane velocity rather than semantic classes or complete 3D boxes. The conclusion reports strong nuScenes results while noting that the method requires sufficient radar support and measurable radial motion.

V. CONCLUSION

  • On nuScenes, PART reaches CA-AP 0.8827, mASTE 0.3188 m, and mAVE 0.8084 m/s, with strong rare-object recall and stable performance under night, rain, and severe occlusion.The model has 1.1 million parameters.Source: §V · p. 7
  • PART predicts existence confidence, a representative surface point, and ground-plane velocity; DAQI uses spatially and kinematically coherent clusters, while PGCA incorporates Doppler consistency and RCS.Uncertainty-aware supervision reduces dependence on exhaustive moving-object annotations.Source: §V · p. 7
  • PART requires sufficient radar support and measurable radial motion; semantic classification and complete 3D box estimation remain outside its scope, with temporal tracking and broader perception integration left for future work.The planned extensions retain radar-only operation.Source: §V · pp. 7–8
Loading 2609.02289v1…