PaperlayerView exact revision on arXiv

Source-linked AI summary

AevaScenes: An FMCW LiDAR Dataset and Benchmark for Long-Range Perception

Gautham Narayan Narasimhan, Heethesh Vhavle, Kumar Bhargav Viswanatha, James Reuther, Deva Ramanan

arXiv:2609.33230v1cs.CVcs.RO

TL;DR

Long-range perception remains difficult because existing LiDAR benchmarks cover shorter distances, where sparse returns complicate annotation and detection. This paper presents an FMCW LiDAR dataset and three-task benchmark, and reports Doppler-related gains in long-range detection and scene flow.

  • Problem

    Existing LiDAR benchmarks annotate only to 80–214 m, and objects beyond about 150 m often have single-digit point counts.

    Source: §I · p. 1

  • Method

    The paper presents a 575-sequence FMCW LiDAR dataset with 3D boxes and semantic labels, and benchmarks detection, scene flow, and semantic segmentation to 400 m.

    Source: §I · p. 2, B. Benchmark Tasks and Metrics · p. 5

  • Results

    Doppler improves long-range detection and consistently reduces scene-flow error; at 200–400 m, TransFusion’s one-frame +vel AP is 0.72 versus 0.60 for five-frame XYZ-only.

    Source: §I · p. 2, B. Learned Object Detection · p. 6, C. Scene Flow · p. 8

  • Takeaways & Limitations

    One-frame velocity-aware models match or exceed five-frame geometry-only vehicle detection at long range, while Doppler also reduces scene-flow error.

    Source: §VI · p. 8

    Tangential motion must still be inferred for scene flow.

    Source: §VI · p. 8

Abstract

from arXiv · show

FMCW LiDAR measures per-point radial Doppler velocity alongside range, providing a motion cue unavailable in conventional time-of-flight sensors. Exploiting this signal at long range remains understudied. We present an FMCW LiDAR dataset of 575 sequences (57.5K frames) with over 8 million annotated 3D boxes across 16 detection classes and per-point labels across 24 semantic classes, captured by six commercial FMCW LiDAR sensors and six paired 4K cameras across eight Bay Area cities, including 237 nighttime sequences, with annotations extending to 400m. We define a benchmark with three tasks: 3D object detection, scene flow estimation, and semantic segmentation. Detection and scene flow are evaluated across three range bins to 400m, with a public evaluation server. We explore the impact of Doppler measurements on flagship recognition tasks, and find significant improvements up to 2X in detection AP of far-away vehicles and pedestrians, particularly in low-latency single-frame settings. We similarly find scene flow accuracy is significantly improved with Doppler measurements across all ranges. Our dataset and benchmark have been publicly released at https://scenes.aeva.com.

I. INTRODUCTION

The paper introduces a large-scale FMCW LiDAR dataset and benchmark for long-range perception, where conventional datasets stop short of highway-relevant distances and per-object geometry becomes sparse. It evaluates Doppler’s role across detection, scene flow, and segmentation, finding gains in detection and scene flow, especially for single-frame long-range perception.

I. INTRODUCTION

  • Existing benchmarks annotate only to 80–214 m, while objects beyond about 150 m often have single-digit point counts and multi-frame aggregation adds blur, registration noise, and latency.At 60 mph, five frames (500 ms) cost about 13 m of travel.Source: §I · p. 1
  • FMCW LiDAR measures per-point range and radial velocity in one sweep, offering a motion cue when long-range geometry is sparse.Its coherent processing is also described as robust to sunlight, other LiDAR interference, fog, and retro-reflector blooming.Source: §I · pp. 1–2
  • The dataset provides 575 sequences, 57.5K frames, over 8 million 3D boxes across 16 classes, 24 semantic classes, and annotations to 400 m.It includes 237 nighttime sequences and spans urban and highway settings across eight Bay Area cities.Source: §I · p. 2
  • Doppler improves long-range detection and consistently reduces scene-flow error, while preliminary segmentation baselines show no consistent dynamic-class gains.For far-away vehicles and pedestrians, the reported detection gains are strongest in single-frame settings.Source: §I · p. 2

II. RELATED WORK

  • Prior FMCW datasets and benchmarks are limited by missing 3D annotations, narrow settings, shorter range, or limited exploration of Doppler’s detection and velocity-estimation impact.TruckDrive reaches 400 m but focuses on heavy-truck highway driving; 4DLidarOpen reaches 150 m and evaluates one detector.Source: §I · p. 2, §II · p. 3

III. AEVASCENES DATASET

AevaScenes pairs long-range FMCW LiDAR sensing with extensive annotations across varied environments. Its supervision extends to 400 m, with point-wise velocity and semantic labels supporting long-range perception.

A. FMCW LiDAR

  • Annotations extend to 400 m, with about 13–15 returns per object at 200–400 m and roughly 40% of objects beyond 80 m.About 18% lie beyond 141 m, and the annotation distribution tails continuously to 400 m.Source: Figure 5 · p. 3, A. FMCW LiDAR · p. 5
  • FMCW measurements remain observable in direct sunlight, dense multi-LiDAR traffic, fog, and scenes with retro-reflective infrastructure.Under fog, scene structure and motion cues remain observable despite significant visual degradation.Source: A. FMCW LiDAR · pp. 3–4
  • The collection vehicle uses six Aeva Aeries II FMCW LiDAR sensors and six aligned RGB cameras, with wide- and narrow-field-of-view LiDAR coverage extending to 400 m.LiDARs run at 10 Hz; each point records position, reflectivity, radial velocity, line index, and acquisition timestamp.Source: A. FMCW LiDAR · p. 4
  • The dataset contains 575 ten-second sequences (57,500 frames), over 8 million 3D boxes across 16 classes, and 237 dedicated nighttime sequences.Collection spans eight Bay Area cities, with 55.8% urban and 44.2% highway frames.Source: A. FMCW LiDAR · p. 4
  • Annotations include 3D boxes, tracking IDs, scene-flow labels, and per-point semantic labels; long-range radial velocity helps annotators distinguish moving objects from static background.The dataset also includes nighttime sequences and a long-tailed distribution across four detection groups.Source: A. FMCW LiDAR · p. 5

B. Benchmark Tasks and Metrics

The benchmark evaluates 3D object detection, scene-flow estimation, and semantic segmentation. Detection and scene flow are measured across range bins through 400 m, while segmentation is evaluated on labeled points.

B. Benchmark Tasks and Metrics

  • Detection uses 101-point-interpolated AP, a minimum of five ground-truth points per box, and three range bins from 0–100 m through 200–400 m.Predictions match ground truth at center-distance thresholds of 2 m for vehicles and 1 m for pedestrians; evaluation uses a 10K-frame test split.Source: B. Benchmark Tasks and Metrics · p. 5
  • Scene-flow estimation uses ground-truth object tracks and reports absolute end-point error for all, static, and dynamic points across the same three range bins.The benchmark permits consecutive or multi-frame inputs; per-point radial velocity can serve as a prior for static-point gating and dynamic-point flow.Source: B. Benchmark Tasks and Metrics · p. 5
  • Semantic segmentation evaluates point-wise labels across 24 classes using mIoU and per-class IoU, restricted to points with ground-truth labels.The benchmark compares settings with and without compensated radial velocity.Source: B. Benchmark Tasks and Metrics · p. 5

IV. BASELINE METHODS

The classical baseline detects dynamic-object seeds from ego-motion-compensated Doppler, clusters and estimates their motion, then tracks them with a UKF. Its radial-velocity cue has a zero-Doppler blind spot, and sparse subthreshold pedestrian returns limit recall.

A. Classical Dynamic Object Detection and Tracking

  • After ego-motion compensation, nonzero radial velocity distinguishes dynamic returns from static ones; thresholding at |d′i| > 0.7 m/s selects seeds without multi-frame differencing.The zero-Doppler blind spot leaves motion perpendicular to all lines of sight indistinguishable from background.Source: A. Classical Dynamic Object Detection… · p. 5
  • DBSCAN clusters seeds in BEV, then Doppler-based splitting and re-merging separates overlapping objects moving at different speeds.Cluster velocity is estimated from radial measurements; a radial-only fallback preserves speed when tangential velocity is underdetermined.Source: A. Classical Dynamic Object Detection… · pp. 5–6
  • A UKF with a CTRV model tracks position, speed, heading, and yaw rate, using cluster velocity to initialize and update speed and heading.For radial-only detections, inflated yaw noise lets the motion model carry heading through Doppler-blind phases.Source: A. Classical Dynamic Object Detection… · p. 6

B. Learning-Based Detection

The experiments compare Doppler-aware and geometry-only detection and scene-flow baselines, alongside semantic-segmentation methods. Doppler particularly benefits single-frame, long-range detection and reduces scene-flow error across ranges.

C. Scene Flow

  • DeFlow, Flow4D, and SSF vary their spatial representation, temporal context, and fusion or decoding strategy; each +vel variant adds radial velocity with other settings unchanged.DeFlow uses two scans, Flow4D five, and SSF aligns unequal occupied-voxel sets before feature fusion.Source: C. Scene Flow · p. 6

D. Semantic Segmentation

  • Cylinder3D and Point Transformer V3 are evaluated for semantic segmentation, with +vel appending compensated radial velocity to geometry and reflectivity features.Source: D. Semantic Segmentation · p. 6

V. EXPERIMENTS

  • Baselines use XYZ and reflectivity, while +vel adds ego-motion-compensated radial velocity; evaluations compare one frame with five ego-motion-compensated frames under unchanged within-architecture settings.Source: §V · p. 6

A. Classical Object Detection

  • The learning-free detector obtains 0.41 dynamic-vehicle AP at 200–400 m, above one-frame XYZ-only CenterPoint, DSVT, and VoxelNeXt estimates but below TransFusion.Its candidate generation requires |di| > 0.7 m/s; learned detectors additionally cover stationary and low-radial-velocity objects.Source: A. Classical Object Detection · p. 6

B. Learned Object Detection

  • At 200–400 m, +vel dynamic-vehicle AP is 0.67/0.63/0.68/0.72 for CenterPoint/DSVT/VoxelNeXt/TransFusion, versus five-frame XYZ-only 0.52/0.62/0.56/0.60.The one-frame configuration uses a 100 ms rather than 500 ms observation window.Source: B. Learned Object Detection · p. 6, Figure 9 · p. 7
  • With five-frame input, +vel raises long-range all-motion-state pedestrian AP from 0.01 to 0.13 for CenterPoint, 0.23 to 0.29 for VoxelNeXt, and 0.13 to 0.33 for TransFusion.The results support complementary evidence from temporal stacking and per-point radial velocity.Source: B. Learned Object Detection · pp. 7–8

C. Scene Flow

  • At 200–400 m, all-point scene-flow EPE falls from 9.47 to 5.02 cm for DeFlow, 17.87 to 5.44 cm for Flow4D, and 10.34 to 5.73 cm for SSF with radial velocity.The plotted comparison reports EPE by input configuration and range.Source: Figure 10 · p. 8

B. Learned Object Detection

C. Scene Flow

Adding ego-motion-compensated radial velocity improves scene-flow estimation across methods, point types, and ranges, with larger gains at longer range. The conclusion notes that tangential motion must still be inferred.

C. Scene Flow

  • Ego-motion-compensated velocity reduces scene-flow EPE in all 36 method–point-type–range combinations, with average all-point reductions of 23.9%, 55.7%, and 57.0% across increasing range bins.At 200–400 m, all-point EPE reaches 5.0–5.7 cm across methods; static-point EPE reductions are 72.7–87.4%.Source: C. Scene Flow · p. 8

VI. CONCLUSION

  • The conclusion reports that Doppler consistently reduces scene-flow error, but tangential motion must still be inferred.Source: §VI · p. 8

A. Dataset

The dataset visualizes compensated radial velocity alongside dense 3D boxes and per-point semantic labels. The accompanying classical baseline uses Doppler and geometric clustering for detection and tracking.

A. Dataset

  • A representative bird’s-eye view shows points with direct radial-velocity measurements; ego-motion compensation leaves dynamic-object motion relative to static background, within an 800 m field of view.The dataset supplies dense 3D boxes and per-point semantic labels.Source: A. Dataset · p. 10, Figure 11 · p. 11

B. Classical Dynamic Object Detection and Tracking

C. Learned Baseline Implementation

Each scan is split into front and rear hemispheres, with the rear rotated to create two samples spanning 0–400 m. Baselines use all six LiDARs and ego-motion-compensated clouds unless noted.

C. Learned Baseline Implementation

1) 3D Object Detection:

Five-frame detection inputs combine the current scan with four earlier scans transformed into the current ego frame, without geometric or sampling augmentation.

1) 3D Object Detection:

2) Scene Flow:

The section specifies scene-flow inputs and preprocessing, segmentation settings, evaluation protocols, and detection metrics for the reported tasks.

2) Scene Flow:

  • Scene-flow methods use ego-motion-compensated coordinates with or without compensated radial velocity; retained DeFlow uses stride two and random hemisphere selection, while Flow4D uses two frames.All three methods remove ground points at z ≤−1.4 m.Source: 2) Scene Flow · p. 12

3) Semantic Segmentation:

  • Both segmentation methods crop to [0, −100, −2, 400, 100, 4] m and append radial velocity normalized as tanh(vr/60).Source: 3) Semantic Segmentation · p. 12
  • Cylinder3D uses no augmentation beyond sensor fusion, spatial cropping, and hemisphere transformation; PTv3 uses random rotation, scaling, flipping, and clipped Gaussian jitter, with Mixup disabled.PTv3 rotation is in [−1, 1] rad, scaling in [0.9, 1.1], and jitter has standard deviation 0.005 clipped to 0.02.Source: 3) Semantic Segmentation · p. 12
  • Task-specific metrics, matching thresholds, and range bins are summarized in Table V(e).Source: 3) Semantic Segmentation · p. 12

D. Object Detection Results

  • Detection reports vehicle and pedestrian AP with 101-point interpolation for dynamic and all objects on the 10K-frame test split, plus matched-detection 3D velocity L2 error (SpdE).Source: D. Object Detection Results · p. 12

E. Scene Flow Estimation

FMCW Doppler provides per-point radial velocity, leaving scene-flow methods to estimate the tangential component. The evaluation measures EPE across point types and range bins without fixing frame pairing.

E. Scene Flow Estimation

  • FMCW Doppler directly measures each point’s radial velocity, reducing scene-flow estimation to recovering the unobserved tangential component.The radial measurement can serve as a prior for gating static points and constraining dynamic-point flow.Source: E. Scene Flow Estimation · p. 12
  • Scene-flow labels derive from ground-truth human-annotated object tracks, and methods may use consecutive or multi-frame inputs.Source: E. Scene Flow Estimation · p. 12
  • Absolute end-point error is reported for all, static, and dynamic points across the same three range bins as detection.Source: E. Scene Flow Estimation · p. 12

F. Semantic Segmentation

The segmentation experiments test whether appending per-point radial velocity to geometry-only models improves semantic labeling. Both evaluated architectures show lower mIoU with velocity, despite isolated class improvements.

F. Semantic Segmentation

  • Cylinder3D and PTv3 are evaluated with geometry-only xyz input and with ego-motion-compensated radial velocity appended as an additional channel.Source: F. Semantic Segmentation · p. 12
  • Velocity produces mixed effects across classes, without consistent gains across vehicles and vulnerable road users.The authors suggest generic velocity features do not expose motion structure effectively.Source: F. Semantic Segmentation · pp. 12–13
  • Segmentation is measured using per-class IoU and mIoU over all 24 semantic classes.Source: F. Semantic Segmentation · p. 13
  • mIoU falls from 41.7 to 39.7 for Cylinder3D and from 49.1 to 44.3 for PTv3 when velocity is added.The reported test results cover all 24 semantic classes.Source: F. Semantic Segmentation · p. 12, Table XI · p. 13
Loading 2609.33230v1…