Source-linked AI summary
Under the Radar: Learning to Predict Robust Keypoints for Odometry Estimation and Metric Localisation in Radar
Dan Barnes, Ingmar Posner
TL;DR
Radar keypoint detectors may be poorly suited to the modality’s noise, and robust localization keypoints are not straightforward to specify. This paper learns keypoints from pose error using a differentiable point-based estimator. On the Oxford Radar RobotCar Dataset, it improves point-based radar odometry over the benchmark while also detecting metric loop closures for mapping and localisation.
Problem
Radar has ghost objects, phase noise, speckle and saturation, while robust keypoints for localization are not defined by a task-specific detector.
Source: §I · p. 1Method
A differentiable point-based pose estimator trains radar keypoint locations, scores and descriptors from pose information alone.
Source: §I · p. 1Results
45% lower translational error and 29% lower rotational error than RO Cen, with 28.5Hz runtime; the method also detects metric loop closures.
Source: §I · p. 1, A. Odometry Performance · p. 4Takeaways & Limitations
The odometry and metric loop-closure outputs together provide a framework for radar-based mapping and localisation.
Source: §VI · p. 6The method does not surpass the current state of the art in dense radar odometry.
Source: §VI · p. 6
Abstract
from arXiv · showhide
This paper presents a self-supervised framework for learning to detect robust keypoints for odometry estimation and metric localisation in radar. By embedding a differentiable point-based motion estimator inside our architecture, we learn keypoint locations, scores and descriptors from localisation error alone. This approach avoids imposing any assumption on what makes a robust keypoint and crucially allows them to be optimised for our application. Furthermore the architecture is sensor agnostic and can be applied to most modalities. We run experiments on 280km of real world driving from the Oxford Radar RobotCar Dataset and improve on the state-of-the-art in point-based radar odometry, reducing errors by up to 45% whilst running an order of magnitude faster, simultaneously solving metric loop closures. Combining these outputs, we provide a framework capable of full mapping and localisation with radar in urban environments.
I. INTRODUCTION
Radar keypoints must remain useful amid modality-specific noise, yet conventional detectors may not be optimized for localization. The paper learns keypoints from pose information and reports point-based odometry and metric loop-closure results.
I. INTRODUCTION
- Radar’s ghost objects, phase noise, speckle and saturation make off-the-shelf detectors designed for other modalities ill-advised.The paper presents radar as a challenging candidate for learning a more suitable detector.Source: §I · p. 1
- The method learns keypoint locations, scores and descriptors from pose supervision through a differentiable pose estimator, without assuming what makes a robust keypoint.The architecture is sensor agnostic when real-world keypoint locations can be inferred.Source: §I · p. 1
- The approach achieves state-of-the-art point-based radar odometry on the Oxford Radar RobotCar Dataset [1] and detects metric loop closures for radar mapping and localisation.The paper frames these outputs as a full mapping and localisation system.Source: §I · p. 1
II. RELATED WORK
- Unlike keypoint detectors trained for repeatability or with labels from other detectors, this approach specializes keypoints for localisation using pose information alone.The related-work discussion also notes that the current radar odometry state of the art [13] is limited to a maximum rotation, while the proposed approach predicts poses at any angle.Source: §II · p. 2
III. LEARNING POINT-BASED LOCALISATION
The framework predicts radar keypoints and learns them through errors from a differentiable point-based pose estimator. Its design is not tailored to radar and can extend to other modalities.
III. LEARNING POINT-BASED LOCALISATION
- The framework learns robust keypoints from ground-truth pose information without tailoring the model design to radar.The approach is presented as applicable to other modalities such as vision or LIDAR.Source: §III · p. 2
1) Keypoint Prediction:
- The method predicts keypoint locations, scores and descriptors from a raw radar scan.The overview says its design is not tailored to radar and can also apply to vision or LIDAR.Source: 1) Keypoint Prediction · p. 2, §III · p. 2
2) Pose Estimation:
- For two proximal scans, a point-based estimator finds the optimal transform and uses its errors to train keypoint prediction..Source: 2) Pose Estimation · p. 2
3) Metric Localisation:
The method predicts keypoints, scores and descriptors, then uses descriptor-and-score-weighted matches in a differentiable pose estimator trained on translation and rotation error. The descriptors also support metric loop closure.
3) Metric Localisation:
- For metric localisation, the same keypoint descriptors serve as local-scene summaries to detect and solve metric loop closures.Source: 3) Metric Localisation · p. 2
A. Keypoint Prediction
- A U-Net-style encoder-multidecoder predicts full-resolution keypoint locations, scores and descriptors.The network uses concatenation skip connections.Source: A. Keypoint Prediction · p. 2
- Spatial softmax within each cell and a weighted coordinate sum produce sub-pixel keypoint locations.Source: A. Keypoint Prediction · p. 2
- Scores represent a keypoint’s usefulness for motion estimation, while dense descriptors identify real-world locations for matching.Scores are mapped to [0,1]; descriptors are formed by resizing and concatenating encoder features into a 248-channel map.Source: A. Keypoint Prediction · p. 2
B. Pose Estimation
- Matches combine keypoint scores and descriptor cosine similarity, retaining points predicted to belong to the static scene and the same real-world location.A dense search for destination matches improves results while remaining well above real-time speed.Source: B. Pose Estimation · p. 3
- Differentiable SVD pose estimation lets transform error backpropagate to keypoint prediction, trained with translation and rotation error.The training loss weights the rotation term with α = 10.Source: B. Pose Estimation · p. 3
C. Metric Localisation
The framework reuses dense descriptors for metric loop closure: it retrieves candidate places by embedding similarity, then estimates a metric transform. The evaluation targets odometry and place recognition on challenging Oxford Radar RobotCar Dataset [1] data.
C. Metric Localisation
- Max-pooling dense descriptors across spatial dimensions produces a 248-D location embedding with practically no inference-speed overhead.Source: C. Metric Localisation · p. 3
- Embedding cosine similarity above a threshold identifies a topological loop closure, whose keypoints are then used to solve a full metric loop closure.Source: C. Metric Localisation · p. 3
IV. EXPERIMENTAL SETUP
- The evaluation uses challenging Oxford Radar RobotCar Dataset [1] data for odometry estimation and place recognition.Source: §IV · p. 3
A. Network Training
Training uses Oxford Radar RobotCar traversals and a differentiable point-matching and pose-estimation pipeline. The network is optimized from pose error after radar scans are converted to Cartesian images and augmented for varied inter-scan rotations.
A. Network Training
- Training uses 25 10 km Oxford Radar RobotCar traversals with Navtech CTS350-X radar data and ground-truth radar poses.Scans are converted to Cartesian at 0.7 or 0.35 m/pixel resolution, and ground-truth odometry is augmented for reliable pose solving across rotations.Source: A. Network Training · pp. 3–4
- Differentiable point matching uses source points, descriptor maps, and score maps to produce destination point locations and match weights.The procedure extracts descriptors and scores at point locations and uses descriptor cosine distance to match source points to destination locations.Source: A. Network Training · p. 3
- Differentiable pose estimation converts matched pixel locations to world coordinates and computes a weighted rigid transform minimizing point-set residuals.The algorithm forms weighted centroids and centered vectors, then uses covariance and singular value decomposition to obtain rotation and translation.Source: A. Network Training · p. 3
- Training uses TensorFlow and Adam with a 10^-3 learning rate until validation task loss remains minimized for at least 150k steps.The training setup uses the pose-augmented ground truth to solve for poses between scans at any rotation.Source: A. Network Training · p. 4
B. Evaluation Metrics
The approach is evaluated on seven additional dataset traversals, totaling approximately 70 km.
B. Evaluation Metrics
- Evaluation adds seven traversals beyond the training data.These are further dataset traversals used to evaluate the approach.Source: B. Evaluation Metrics · p. 4
- The additional evaluation traversals cover approximately 70 km.The passage reports the combined distance for the seven traversals.Source: B. Evaluation Metrics · p. 4
- The reported evaluation scope is approximately 70 km across seven further dataset traversals.This is the stated extent of the additional evaluation data.Source: B. Evaluation Metrics · p. 4
1) Odometry:
Odometry is evaluated using translational and rotational drift rates against ground-truth radar odometry. The errors are averaged over subsequences spanning 100 to 800 metres.
2) Odometry:
- Odometry evaluation measures average normalized end-point translational and rotational errors against ground-truth radar odometry.The drift-rate calculation follows the KITTI odometry benchmark approach.Source: 1) Odometry · p. 4
- Errors are computed for subsequences of length 100, 200, through 800 metres.The reported errors are averaged across all such subsequences.Source: 1) Odometry · p. 4
- The odometry metrics are normalized end-point errors expressed as translational and rotational drift rates.The evaluation uses the KITTI odometry benchmark approach.Source: 1) Odometry · p. 4
2) Localisation:
Localisation is evaluated by top-N scan retrieval against a 5 m ground-truth proximity criterion, excluding loop closures within the same trajectory. The section also describes a NetVLAD place-recognition comparison trained with hard positive and negative locations.
2) Localisation:
- A query is correctly localised when at least one of the top N retrieved scans lies within 5 m of its ground-truth position.Retrieval is ranked by descriptor cosine distance; 5 m is used instead of 25 m because nearer matches allow a full metric loop closure to be solved reliably.Source: 2) Localisation · p. 4
- Localisation results exclude loop closures to the same trajectory and evaluate localisation to other test datasets.The stated evaluation criterion uses 5 m ground-truth proximity.Source: 2) Localisation · p. 4
- The NetVLAD comparison freezes the highest-odometry-performing model before adding and fine-tuning a 512-D, 64-cluster layer for place recognition.Training uses batch-hard triplet loss with five positives within 5 m and five negatives farther than 25 m per sample.Source: 2) Localisation · p. 4
V. RESULTS
The learned keypoints improve point-based radar odometry over the state-of-the-art benchmark while running faster. Ablations and visualisations show how learned locations and scores select points on static scene structure across varied vehicle motion.
A. Odometry Performance
- 45% lower translational error and 29% lower rotational error than state-of-the-art point-based radar odometry, while running at 28.5Hz—an order of magnitude faster.Table I marks the best-performing model in bold; the comparison is against ‘RO Cen’.Source: A. Odometry Performance · p. 4, Table I · p. 5
- Across stationary, forward-travel, slight-bend, and aggressive-turn examples, the highest-scoring matches align with static structures such as walls and buildings.In the stationary example, the only moving match belongs to a vehicle passing through the scene.Source: Figure 4 · p. 4
- The additional dense-radar benchmark is not surpassed, but unlike that method, this approach is not limited by rotation, which is crucial for metric loop closures.The additional benchmarks are not directly comparable to the proposed method.Source: A. Odometry Performance · p. 5
- The best performance at both test resolutions comes from enabling both Location and Score heads, with scores more essential because they can suppress empty, unobserved, or noisy regions.The resulting keypoints localise to static scene structure.Source: Figure 3 · p. 4, A. Odometry Performance · p. 5
B. Localisation Performance
Odometry-trained location embeddings perform better for place recognition than embeddings fine-tuned with NetVLAD. They also detect topological loop closures without place-recognition training, while random initialisation performs much worse.
B. Localisation Performance
- Even without place-recognition training, ‘Max Descriptors’ reliably detect topological loop closures and far exceed randomly initialised weights.The keypoint variant with random weights falls off the plotted graph.Source: B. Localisation Performance · p. 5
- NetVLAD embeddings generalise worse to the test set than odometry-trained embeddings despite better training convergence; increasing descriptor dimensionality brings negligible gains at runtime cost.The odometry model is frozen before the NetVLAD layer is added and fine-tuned for place recognition.Source: B. Localisation Performance · p. 5
- Odometry-trained ‘Max’ embeddings exceed the place-recognition performance of embeddings fine-tuned with an additional NetVLAD layer.Fig. 5 reports recall versus number of top candidates between test datasets.Source: Figure 5 · p. 5
C. Mapping and Localisation
The system combines radar odometry, topological loop-closure detection, and pose-graph optimisation for full mapping and localisation. The paper reports improved point-based radar odometry while also solving metric loop closures.
C. Mapping and Localisation
- Three parallel processes produce an optimised map: odometry adds trajectory edges, embedding comparisons find topological closures and solve metric poses, and pose-graph optimisation merges the edges.The loop-closure threshold is set for 100% precision on a small validation set; embeddings are stored in a KDTree for lookup.Source: C. Mapping and Localisation · p. 5
- Location-specific embeddings detect loop closures across different route traversals; the caption contrasts radar recognition with vision challenges from limited field of view, lens glare, and environmental conditions.The figure shows the radar input on the map and in the top row, with corresponding colour-coded scans in the third row.Source: Figure 6 · p. 6
- Fig. 7 shows scored keypoints highlighting static structures, metric loop closures, and constraints merged into a pose-graph-optimised map at well over real-time speeds.The displayed open-loop trajectories come from three test-dataset sections at approximately the same location.Source: Figure 7 · p. 6
VI. Conclusions
- By up to 45%, errors are reduced against state-of-the-art point-based radar estimation, with an order-of-magnitude speedup; dense radar odometry remains ahead.The approach also solves poses at any rotation and detects metric loop closures as part of a radar-based mapping and localisation system.Source: §VI · p. 6
- The authors propose applying the flexible architecture to most sensor modalities and reusing detected points for downstream tasks such as object-velocity estimation.These are proposed future directions.Source: §VI · p. 6