Source-linked AI summary
Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking
Jue Wang, Xuan Wang, Hao Zhou, Ruixiang Zhou, Yixuan Zhou, Tianshuo Yuan, Jieming Ma, Jie Zhang, Fei Luo
TL;DR
Fixed-camera surveillance lacks benchmarks that jointly provide radar, dense moving-instance masks, and persistent identities, while visual motion cues degrade under adverse conditions. This paper introduces RGBTR-Motion and SAM-Radar, which ground radar in multimodal segmentation and tracking to improve instance recovery and identity continuity. SAM-Radar achieves 0.7027 IoU and 0.8090 F1-50 while improving MOTA, HOTA, and IDF1 over competing values.
Problem
Existing surveillance benchmarks do not jointly provide radar measurements, dense moving-instance masks, and temporally consistent identities, while visual motion evidence degrades under weak appearance and occlusion.
Method
SAM-Radar grounds sparse radar measurements in calibrated RGB-thermal features and uses radar association with controlled visual memory to preserve identities during temporary appearance failures.
Results
MOTA, HOTA, and IDF1 improve by 0.2977, 0.1603, and 0.2857 over their respective strongest competing values, alongside 0.7027 IoU and 0.8090 F1-50.
Takeaways & Limitations
Radar improves instance recovery and identity continuity when its sparse measurements are explicitly grounded, assigned to tracks, and coupled to controlled visual memory.
Takeaways & Limitations
Visual-correspondence motion evidence remains vulnerable to illumination changes, thermal crossover, occlusion, and fast or subtle motion.
Abstract
from arXiv · showhide
Moving-object perception must decide which image regions correspond to real motion and keep every instance identified over time. Methods that read motion from appearance, optical flow, or estimated trajectories lose that evidence under poor illumination, adverse weather, reflections, and occlusion. Radar is a natural remedy because it measures radial velocity directly instead of inferring it from photometric correspondence. However, existing benchmarks do not jointly provide radar measurements, dense moving-instance masks, and temporally consistent identities for surveillance. We therefore introduce RGBTR-Motion, a synchronized and calibrated fixed-camera benchmark that pairs RGB, thermal, and radar streams with dense instance masks and temporally consistent identities across diverse surveillance scenes. We also develop SAM-Radar, an RGB, thermal, and radar-based segmentation and tracking framework built on SAM 3. SAM-Radar's radar-aware detector fuses calibrated RGBT features with radar returns that are grounded at their projected image locations, and motion supervision, implemented as foreground classification of those projected returns, teaches the detector to reject clutter without any text prompt. The tracker associates accepted radar returns with individual trajectories and uses them as physical evidence that a visually degraded target remains present. This allows it to bridge short periods of low visibility or occlusion and reconnect a reappearing target to its existing identity instead of starting a new track. SAM-Radar attains 0.7027 IoU and 0.8090 F1-50, and raises MOTA, HOTA, and IDF1 by 0.2977, 0.1603, and 0.2857 over the strongest competing values.
1 Introduction
The section frames moving-object perception as recovering accurate masks and persistent identities for physical movers in fixed-camera surveillance, where visual motion evidence fails under degradation and occlusion. It introduces RGBTR-Motion and SAM-Radar, combining calibrated RGBT-radar sensing, motion supervision, radar association, and staged training to segment and track movers through temporary appearance failures.
- Problem setting: Fixed-camera surveillance must recover accurate masks and preserve physical movers’ identities over time despite weak appearance, crossings, and occlusions.The task targets pedestrians, cyclists, and vehicles rather than photometric or background changes.
- Motivation: Visual-correspondence methods lose motion evidence under illumination changes, thermal crossover, occlusion, and fast or subtle motion.Radar provides radial velocity directly, but remains sparse and noisy and can fail for stopped or tangentially moving targets.
- SAM-Radar: SAM-Radar’s detector fuses calibrated RGBT features with spatially anchored radar tokens and uses motion supervision to reject multipath reflections and background clutter.The radar returns are anchored to projected image neighborhoods so visual structure and measured motion jointly locate movers.
- SAM-Radar: Radar association, confidence-gated residual injection, and Hold-Lost memory preserve identities when visual evidence weakens or short occlusions occur.The identity is released only after both visual and radar evidence remain absent.
- Benchmark: 107 sequences and 8,537 annotated frames comprise RGBTR-Motion, a synchronized, calibrated fixed-camera benchmark spanning daytime, nighttime, rainy, and indoor conditions.The benchmark provides frame-level masks, persistent identities, and an acquisition-grouped split that prevents temporal leakage.
- Training: The two-stage strategy trains the detector first on aligned RGBT-radar inputs, then freezes it while training the tracker to preserve identities over time.Stage 1 discovers and segments moving objects; Stage 2 focuses on identity preservation during visual degradation.
2 Related Work
Prior work spans geometric, appearance-based, promptable, and multimodal approaches to motion segmentation, video identities, and radar perception. Existing systems and benchmarks do not jointly provide motion-centric dense instance masks with sequence-level identities, motivating RGBTR-Motion’s aligned thermal, radar, and visible-image representation.
- Motion segmentation: Classical methods group pixels or trajectories geometrically, while fixed-camera background subtraction and optical-flow systems remain vulnerable to changing appearance.These limitations are noted for nuisance appearance changes in surveillance settings.
- Instance segmentation and tracking: MOTS and video instance segmentation couple masks with identities, while SAM, SAM 2, SAM 3, and SegAnyMo extend promptable, streaming, open-vocabulary, or point-track-based segmentation.These approaches assume or build on different forms of vocabulary, memory, detection, tracking, and prompting.
- Multimodal perception: RGB and thermal provide complementary appearance cues, whereas radar contributes independent range and radial velocity for multimodal perception under adverse conditions.Condition-aware fusion improves robustness when either RGB or thermal degrades, but both remain appearance based.
- Multimodal benchmarks: ASY-VRNet and RADCI demonstrate visual-radar fusion, but neither provides motion-centric dense instance masks with sequence-level identity annotations.RADCI is a fixed-tripod RGBT-radar benchmark with 2D boxes and target IDs; the proposed direction instead aligns thermal features and sparse returns to the visible image.
3 The RGBTR-Motion Benchmark
RGBTR-Motion is a fixed-camera, synchronized RGB–thermal–radar benchmark for motion perception, pairing calibrated multimodal streams with dense moving-instance masks and persistent sequence-local identities. It supports pixel-level segmentation and tracking across diverse surveillance conditions and events, including entry, crossing, occlusion, and exit.
- Acquisition setup: RGBTR-Motion uses a rigid tri-modal rig with 1280×720 visible, 640×512 long-wave infrared, and 4D millimeter-wave radar sensors, all recording at 10 fps.The sensors share a common plate and remain fixed throughout each acquisition, providing consistent geometry across daytime, nighttime, and rainy scenes.
- Benchmark scale: The audited benchmark contains 107 sequences, 8,537 annotated frames, 15,475 instance masks, and 274 sequence-local identities.It targets sustained pedestrians, cyclists, and vehicles traversing surveillance scenes while preserving identity through entry, crossing, occlusion, and exit.
- Annotations and coverage: Beyond category-level boxes, RGBTR-Motion links motion-centric instance masks to sequence-local identities for pixel-level segmentation and tracking through entry, crossing, occlusion, and exit.The benchmark spans indoor surveillance across daytime, nighttime, and rain, with its composition illustrated through calibrated RGBT-radar examples.
- Calibration and synchronization: All streams are aligned to the visible-camera timeline, with infrared frames warped into visible-image coordinates and radar returns calibrated, projected, and filtered by image-plane validity.Radar points are grouped around each video frame’s timestamp, retaining reliable moving points alongside high-SNR static or slow points.
- Task definitions: Each visible frame provides COCO-compatible masks, boxes, categories, and persistent identities supporting frame-level moving-object segmentation and video-level segmentation and tracking.The annotation policy covers temporary full occlusion, field-of-view exits, and stationary intervals.
4 Method
SAM-Radar predicts moving-object masks and persistent identities from synchronized RGB, thermal, and projected radar streams through two-stage detector–tracker training. Its method grounds radar in visual features for motion-aware detection and uses radar association, feature updates, and Hold-Lost memory to maintain tracks through degraded visibility.
- 4 Method: The model trains in two stages: first frame-level discovery and segmentation across RGB, thermal, and radar, then radar association, feature updates, and Hold-Lost memory for tracking.The detector and radar encoder are frozen during tracker training to separate spatial detection from temporal learning.
- Calibrated RGBT fusion: Separate RGB and thermal trunks preserve the pretrained RGB path while learned fusion incorporates thermal evidence at the detector and tracker scales.Thermal features are fused throughout tracking while the spatial encoder, detector, and memory writer remain frozen in stage two.
- Spatially anchored radar tokens: Each radar token combines projected position, normalized range, radial velocity, signal strength, local RGBT context, modality encoding, and a predicted foreground embedding.This construction links sparse physical measurements to image structure at their projected locations.
- Motion supervision: Projected-return foreground classification adds motion supervision that teaches the detector which radar measurements correspond to targets rather than clutter.The supervision uses binary return labels and a focal objective alongside SAM 3 classification, box, generalized-IoU, and mask losses.
- Hold-Lost memory: Hold-Lost memory preserves the latest reliable visual state when visual confidence weakens but an associated radar measurement remains, preventing short occlusions from discarding tracks.The tracker distinguishes reliable TRACKED predictions from radar-supported HOLD states rather than writing every low-confidence prediction into memory.
- Radar association: Radar association multiplies spatial, range-velocity, and descriptor terms, assigns each return to at most one track, and rejects it through a null option or confidence test.The fixed null evidence is 0.25 and the ownership threshold is 0.35.
5 Experiments
SAM-Radar outperforms competing methods on moving-object segmentation and tracking, with especially large gains in instance recovery and identity preservation. Ablations and qualitative sequences attribute these improvements to motion-supervised radar detection, radar association, Hold-Lost memory, and robustness under degraded visibility.
- Main comparison: 0.7027 IoU and 0.8090 F150 exceed strongest competing scores of 0.6574 and 0.6243, respectively.The gains are 0.0453 for IoU and 0.1847 for F150, indicating a larger advantage in recovering matchable moving instances.
- Main comparison: 0.2977, 0.1603, and 0.2857 are SAM-Radar’s gains in MOTA, HOTA, and IDF1 over their respective strongest competing values.Identity-sensitive tracking benefits especially from preserving trajectories through occlusion and avoiding fragmentation or reassignment.
- Detector components: 0.5850 Pixel IoU, 0.7382 Pixel Dice, and 0.7927 F150 result when motion supervision is added to radar attributes, up from 0.4677, 0.6373, and 0.5599.The largest absolute change is in F150, while simultaneous pixel-metric gains indicate improved foreground estimation.
- Tracker components: 0.1291/0.0530/0.0903 and 0.1190/0.0444/0.0755 are the MOTA/HOTA/IDF1 gains from radar association and Hold-Lost memory, respectively.Combining both mechanisms gives the largest gains, 0.2229/0.1053/0.1380, and the best value for every reported metric.
- Qualitative results: SAM-Radar produces compact masks across distances, coherent silhouettes under low illumination, and the same track identity after occlusion.It correctly produces no mask in Frame 62 and recovers the obstructed track in Frames 10 and 25.
6 Conclusion
The paper introduces RGBTR-Motion and SAM-Radar for radar-supported motion segmentation and tracking in varied surveillance conditions. Its findings show that explicitly grounded radar measurements, track assignment, and controlled visual memory improve instance recovery and identity continuity.
- Contributions: RGBTR-Motion is an RGBT-radar benchmark for motion segmentation and tracking under varied surveillance conditions, paired with SAM-Radar, a radar-enhanced framework.SAM-Radar uses radar measurements to strengthen object discovery and temporal association.
- Radar-aware detection: The detector grounds each radar return at its projected image location and uses range, radial velocity, and SNR as physical motion and reliability cues.These radar cues complement local RGBT appearance during object discovery.
- Findings: The designs improve instance recovery and identity continuity by grounding sparse radar measurements, assigning them to individual tracks, and coupling them to controlled visual memory.The findings identify this combination as the most effective use of radar for robust tracking.