Source-linked AI summary
CARE: Camera-Residual Reserves for First Sightings in Adaptive LiDAR Sensing
Jiachen Gong, Yun Li, Ehsan Javanmardi, Wencan Mao, Manabu Tsukada
TL;DR
History-driven adaptive LiDAR scanning can miss objects without prior tracks, while camera-guided allocation may resample already covered objects. CARE reserves rays for camera detections unexplained by track forecasts, improving first-sighting recall across budgets while preserving overall recall closely; safety-bounded forgetting further releases budget under guarded conditions.
Problem
History-driven scanning depends on prior tracks, while random and unrestricted camera-guided alternatives lack targeted awareness of where unseen objects appear or waste budget on covered objects.
Method
CARE is a training-free allocation rule that reserves part of a fixed ray budget for depth-free angular wedges from camera detections unexplained by track forecasts, with unused reserve flowing to a shared random floor.
Results
CARE improves first-sighting recall over history at every tested budget, with gains of 5.2, 5.2, and 4.3 points at 10%, 20%, and 35% budgets; residual filtering changes overall recall by only −0.1, +0.5, and +0.7 points.
Takeaways & Limitations
The camera cue drives CARE’s first-sighting gain, while residual filtering is most valuable in crowded scenes and the first-sighting versus overall trade-off depends on budget.
Takeaways & Limitations
Vehicle results are descriptive, and the CARE configuration uses fixed off-the-shelf detector and margin settings that were not tuned on the evaluation split.
Abstract
from arXiv · showhide
Adaptive LiDAR scanning concentrates a limited sensing budget on regions of interest predicted from past object tracks, lowering data volume in autonomous driving while maintaining detection accuracy. However, existing scanning policies face three challenges. First, history-driven approaches depend on past tracks, so unseen objects are detected late or missed. Second, random or uniform sampling outside the predicted regions has no awareness of where new objects appear. Third, camera-guided alternatives spend budget on all camera detections, resampling objects already covered, costing recall in crowded scenes and range when budgets are scarce. This paper introduces the CAmera-REsidual reserve (CARE), a training-free allocation rule that reserves part of a fixed ray budget for the directions of current camera detections that the track forecasts cannot explain; the rest follows the base history policy, and unused reserve returns to a random floor. The paper makes three contributions. First, a leakage-free ray-budget evaluation on nuScenes (150 scenes, 4,148 events) measuring the first-sighting loss of history-driven scanning, with a strict-causal variant using the preceding keyframe. Second, CARE raises first-sighting recall by 5.2, 5.2, and 4.3 points at 10%, 20%, and 35% budgets over the history policy, with paired intervals excluding zero; the camera cue drives this gain, and the first-sighting versus overall trade-off is a budget-dependent Pareto choice. Third, a safety-bounded forgetting module that releases budget from receding or static tracks beyond a speed-dependent guard distance; at tight budgets, forgetting without the guard significantly harms near-field recall, so the guard is what keeps it safe. The pipeline runs end to end on a real vehicle and, in closed-loop simulation, detects an occluded pedestrian earlier and brakes more reliably than history-driven scanning.
1 Introduction
CARE addresses first-sighting under-allocation by reserving rays for camera detections that track forecasts cannot explain, while retaining the history policy for tracked regions. The paper evaluates this allocation with a leakage-free protocol and reports improved first-sighting recall across budgets.
- History-driven scanning under-allocates newly entering objects because they lack forecast regions, while random fallback is not directed toward where new objects appear.
- CARE is a training-free rule that reserves part of the ray budget for residual camera detections after removing detections explained by projected track forecasts.The remaining budget follows the base history policy, and unused reserve returns to the random floor.
- The evaluation uses a leakage-free fixed angular ray-cell budget with shared detector, budget, and random floor, plus a strict-causal preceding-keyframe camera variant.
- CARE raises first-sighting recall by 5.2, 5.2, and 4.3 points at 10%, 20%, and 35% budgets over history, with paired confidence intervals excluding zero.The evaluation covers 150 nuScenes scenes and 4,148 events.
- The paper also introduces safety-bounded forgetting that releases budget from receding or static tracks beyond a speed-dependent guard distance.Forgetting without the guard loses 1.8 points of near-field recall at the tightest budget, while the guarded default shows no significant loss.
- CARE is compared with policy-level re-implementations of history-driven scanning, uncertainty-driven reserve, and beam reduction under the same allocator, budget, and detector.
2 Related Work
Prior adaptive LiDAR methods use image cues, scene priors, attention, track forecasts, uncertainty, or post-acquisition sampling. CARE instead uses current camera detections before acquisition to direct rays toward newly entering objects.
- History-aware scanning concentrates dense sensing on forecast object regions, but newly appearing objects have no track and therefore receive no priority.
- CARE retains history-driven scanning as its base and adds current-frame camera detections that the history state cannot contain to cover new objects.
- Uncertainty-driven reserves can achieve high overall recall but trail CARE on first sightings because uncertainty requires previously returned sensor points.
- Post-acquisition samplers and beam-reduction methods subsample already acquired clouds, saving downstream computation rather than changing where LiDAR looks.
- Camera-LiDAR fusion improves processing of returned points after the scan pattern is fixed, whereas CARE uses camera boxes earlier to decide the next ray directions.
3 Problem Formulation
The paper formulates adaptive LiDAR sensing as selecting angular ray cells before observing the current returns, under a fixed budget and causally eligible history or camera information. It measures first-sighting, timing, and overall detection performance.
- The LiDAR field of view is discretized into A azimuth columns and E elevation rows, producing N = A · E firing cells.
- A policy selects a mask S_t ⊆ G with |S_t| ≤ B, where B = round(βN), and retains only points whose viewing directions fall in selected cells.
- Selection is leakage-free because the mask is fixed before frame-t returns are observed, allowing past information and camera cues but not the cloud being acquired.
- The main evaluation uses synchronized keyframe camera detections, while a strict-causal variant uses detections from the preceding keyframe.The preceding-keyframe detections are half a second stale but complete before the current decision.
- First-sighting recall measures detections at an annotated object's first in-range keyframe; the study also reports time to first detection and overall recall.Contrasts use paired per-scene bootstrap intervals and are stratified by range and azimuth.
- The diagnostic uses A=512, E=32, budgets β ∈ {0.10, 0.20, 0.35}, range R=50 m, and a class-consistent 2 m center-distance match.
4 Method
CARE combines a shared history-based allocator with a prioritized residual camera reserve and safety-bounded forgetting. It protects exploitation cells, directs unused capacity through deterministic fallback, and guards near-field tracks while releasing budget from low-criticality tracks.
- CARE pipeline: The pipeline projects track forecasts into the image, retains camera detections they cannot explain as residuals, allocates their angular wedges, and holds the resulting mask over intervening sweeps.
- Shared budget allocator: The shared allocator fills an exploitation share from forecast hulls, then reserve layers in priority order, and finally a seeded random floor without exceeding B.
- History policy: The base exploitation set uses constant-velocity forecasts whose angular hulls are dilated to absorb forecast error.
- Camera-residual reserve: CARE defines residual detections as same-class camera boxes lacking an explained projected forecast, then maps them to depth-free angular wedges for the reserve.
- Camera-residual reserve: Matching uses full forecasts rather than SBF-shrunk hulls, while a cap protects exploitation cells from false camera detections.
- Safety-bounded forgetting: SBF shrinks hulls only for receding or quasi-static tracks beyond a speed-dependent guard distance, leaving other tracks untouched.
- Safety-bounded forgetting: Guarded forgetting releases cells to the residual reserve, whereas forgetting without the guard harms near-field recall and the guarded default shows no significant loss at the tightest budget.
- Safety-bounded forgetting: The guard is fixed from published traffic rules and models, with T_h=2 s and s_0=2 m; the safety factor is fixed at 2.0 before evaluation.
5 Experiments
On nuScenes and vehicle studies, CARE improves first-sighting allocation by reserving rays for camera detections unexplained by track forecasts, while its forgetting guard protects near-field recall.
- Main results: 5.2, 5.2, and 4.3 points: CARE's first-sighting recall gain over history-driven scanning at 10%, 20%, and 35% budgets, respectively.All paired intervals exclude zero; equal scene weighting gives 5.3, 5.2, and 4.2 points.
- Main results: CARE occupies the knee of the first-sighting-versus-overall-recall Pareto frontier at 10%, 20%, and 35% budgets.Overall recall versus history changes by −0.8, −0.5, and +0.6 points, so the trade-off is budget-dependent rather than uniformly favorable.
- Baseline comparison: The uncertainty reserve leads overall recall but trails CARE on first sightings by 3.5, 2.7, and 2.1 points across the three budgets.Beam reduction gives up 17.4, 11.5, and 6.4 overall-recall points, while CARE retains the better joint trade-off.
- Vehicle studies: In closed-loop simulation, camera policies detect an occluded pedestrian at about 15 m and brake in every episode, versus history detection at 11.9 m and braking in three of five episodes.In the dense 3% CARLA scene, CARE detects the pedestrian at 15.1 m, matching the full-scan bound, while the all-camera reserve detects at 10.3 m.
6 Conclusion
CARE addresses the first-sighting blind spot of history-driven adaptive LiDAR with a training-free reserve for camera detections that track memory cannot explain. It improves first-sighting recall across budgets, while real-vehicle and simulation results provide evidence of earlier detection and braking benefits within stated evaluation limits.
- CARE repairs history-driven scanning’s first-sighting failure by reserving rays for camera detections that memory cannot explain.
- The mechanism pays off in crowded scenes, where residual camera detections supplement the history policy’s coverage of newly entering objects.
- CARE runs end to end on a real vehicle and, in simulation, detects an occluded pedestrian earlier and brakes more reliably than history-driven scanning.
- The evidence is limited to one frozen detector pair, ray-cell emulation, and new-entry events; steerable hardware remains the next step.