Source-linked AI summary
Variational End-to-End Navigation and Localization
Alexander Amini, Guy Rosman, Sertac Karaman, Daniela Rus
TL;DR
End-to-end driving and map-conditioned methods have limited treatment of long-term plans and uncertainty about localization and actions. This paper integrates camera data and roadmaps in a variational network for probabilistic control, route-conditioned steering, and map-based localization. The reported evaluations demonstrate navigation and pose-uncertainty reduction, including near intersections.
Problem
End-to-end driving has limited ability to handle long-term plans, while map-conditioned methods do not address uncertainty about location and longer-term plans.
Source: §I · p. 1Method
A variational network combines camera imagery with unrouted maps for probabilistic control and uses routed maps to produce route-conditioned deterministic steering, with map–visual correspondences for pose estimation.
Source: §I · pp. 1–2, §III · p. 3, B. Driving with Navigational Inputs · p. 4Results
The model demonstrated route-conditioned steering and probabilistic control estimates, and reduced spatial uncertainty by an average of 0.31 nats near intersections.
Source: §V · p. 6, C. Reducing Localization Uncertainty · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
Deep learning has revolutionized the ability to learn "end-to-end" autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-to-point navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior.
I. INTRODUCTION
End-to-end driving networks can struggle with long-term plans, and map-conditioned methods do not account for uncertainty about location or plans. This work integrates maps with sensory data to support navigation and localization while preserving uncertainty.
I. INTRODUCTION
- Coarse maps support higher-level reasoning and matching road patterns despite appearance changes, provided the overall road network structure matches.The paper distinguishes this topology-level matching from fine-scale place recognition and loop closure.Source: §I · p. 1
- Map-conditioned driving methods capture longer-term action structure but ignore uncertainty about the vehicle’s location and plan.The paper identifies this as a limitation of prior approaches to end-to-end driving.Source: §I · p. 1
- The variational network combines raw sensory data with routed and unrouted maps while preserving reasoning about uncertainty.Unrouted maps support probabilistic control estimates; a routed map provides deterministic control along the specified route.Source: §I · pp. 1–2
- The localization algorithm infers pose by matching road topology in the map to visual road appearance.The introduction presents this as a way to reason about localization from map–image correspondences.Source: §I · pp. 1–2
- The algorithms were evaluated on real-world data, demonstrating steering-control navigation and improved pose localization with severely limited GPS information.The paper lists this evaluation as a key contribution.Source: §I · p. 2
II. RELATED WORK
Prior work spans modular driving systems, end-to-end control, map conditioning, variational inference, planning, and localization. The paper positions its contribution at the intersection of these lines, emphasizing map-related uncertainty and high-level visual matching.
II. RELATED WORK
- Earlier map-conditioned end-to-end driving methods do not connect steering uncertainty to maps or reason about discrepancies between input modalities.The paper contrasts these methods with its approach to uncertainty and cross-modal reasoning.Source: §II · p. 2
- Variational-inference driving networks address multiple possible actions and robustness; this work extends that approach by conditioning on maps.The cited line of work is identified as.Source: §II · p. 2
- The paper’s map matching is higher-level than conventional place recognition, closer to semantic localization using driving affordances.The related work also connects the approach to planning hierarchies and localization research.Source: §II · p. 2
III. MODEL
The model uses camera images and a coarse unrouted roadmap to estimate a probability distribution over steering commands, while a routed map enables deterministic steering. Its architecture combines image and map features, with the stochastic output represented by a Gaussian mixture model.
III. MODEL
- The network uses camera images and a pose-centered crop of the unrouted map to estimate a distribution over steering commands.It uses a Gaussian mixture model, with K = 3 chosen empirically to capture most driving situations encountered.Source: §III · p. 2
- Steering command is used interchangeably with road curvature, rather than steering angle, because steering angle depends on road slip and vehicle-specific control-plant parameters.Source: §III · p. 3
- A routed map adds a second network function that outputs a deterministic steering command for the desired route.The image-processing output is combined with features from a convolutional module processing the routed map.Source: §III · pp. 2–3
- The network uses fewer convolutional layers for map patches because they contain fewer features and preserving map position matters.The design aims to avoid translational-invariance effects associated with convolution and subsampling.Source: §III · p. 3
- Training penalizes mixture-component standard deviations and uses the negative log-likelihood of the steering command under the Gaussian mixture.The standard-deviation penalty is per component; the supplied passage does not give the complete loss expression.Source: §III · p. 3
A. Localization via End-to-End Networks
The model estimates pose by relating the steering distribution predicted for a pose to the steering distribution across possible poses. The posterior can be updated by marginalizing over steering and pose, with pose marginalization tractable for localized beliefs or discrete road networks.
A. Localization via End-to-End Networks
- The network computes a steering distribution conditioned on pose, image, and map, and pose estimation uses double marginalization over steering and pose.The posterior update incorporates a prior belief about pose.Source: A. Localization via End-to-End Networks · p. 3
- Pose marginalization is tractable when prior observations have localized the pose or when poses are sampled over a discrete road network.Source: A. Localization via End-to-End Networks · p. 3
- The posterior update evaluates how each pose and its image/map explain steering-angle samples, then normalizes the distribution.The same algorithm can be used with modifications in a map-matching framework,.Source: A. Localization via End-to-End Networks · p. 4
IV. RESULTS
The results section presents evaluation on real-world train and test data, covering routed and unrouted steering estimates and pose-uncertainty reduction from map-camera agreement.
IV. RESULTS
- The evaluation uses a real-world training and test dataset of rich driving environments.Source: §IV · p. 4
- The section demonstrates driving with a roadmap image and steering-angle estimation using both routed and unrouted maps.Source: §IV · p. 4
- The section examines reducing pose uncertainty based on agreement between the map and camera feeds.Source: §IV · p. 4
A. System Setup
The system was evaluated onboard a drive-by-wire vehicle using camera, GPS, and IMU sensing. The authors built rendered road maps from OSM and trained on one set of suburban roads before testing on unseen segments and intersections.
A. System Setup
- A Toyota Prius V with drive-by-wire capabilities used three forward-mounted cameras and an onboard NVIDIA Drive PX2.The cameras captured 1080p RGB images at approximately 30 Hz; GPS and IMU provided coarse global localization.Source: A. System Setup · p. 4
- The road network was built from OSM as a directed graph whose intersections are vertices and directed roads are edges.Edge weights use great-circle length, with possible extensions to account for road type or real-time traffic delays.Source: A. System Setup · p. 4
- To handle route ambiguity, the authors split driving sequences into non-ambiguous subroutes before rendering routed maps.This avoids self-crossings when the vehicle traverses the same intersection in different directions; unrouted maps show all edges, while routed maps highlight traversed edges.Source: A. System Setup · p. 4
- Training used 25 km of suburban driving data, and testing used different road segments and intersections not seen during training.Training routes included turns, intersections, roundabouts, and dynamic obstacles such as vehicles and pedestrians.Source: A. System Setup · p. 4
B. Driving with Navigational Inputs
The network represents multiple plausible steering commands from unrouted maps and selects deterministic control when given a routed map. It also navigated an unseen test track onboard at 15 Hz, including a roundabout absent from training.
B. Driving with Navigational Inputs
- The network produces a continuous steering-control distribution from an unrouted map and deterministic control for a routed-map path.The figure’s projected trajectories show the vehicle path if it executes the predicted steering command.Source: B. Driving with Navigational Inputs · p. 4
- Fig. 4 evaluates steering-distribution coverage by z-score and plots true-command density across test-roadmap locations; density decreases before intersections as probability spreads across paths.Points with gross GPS failures are omitted from the visualization.Source: Figure 4 · p. 5, B. Driving with Navigational Inputs · p. 5
- The onboard system ran at 15 Hz on an unseen 1 km test track with nine intersections, and the network generalized to a roundabout absent from training.With a routed path, it disambiguated multiple steering modes and selected a control command toward the route.Source: B. Driving with Navigational Inputs · pp. 4–5
C. Reducing Localization Uncertainty
The method updates pose beliefs by comparing camera observations with map road topology, under assumed GPS-derived pose priors. Around intersections, it reduces spatial and angular uncertainty on held-out roads, although spatial localization is weaker on linear segments.
C. Reducing Localization Uncertainty
- Fig. 5 maps changes in four uncertainty measures across test-route GPS readings; Fig. 6 reports reduction in spatial variance, angular variance, total variance, and entropy across prior-uncertainty levels.Decreasing variance corresponds to increasing confidence; Fig. 6 reports positive average improvement for samples preceding intersections.Source: Figure 5 · p. 5, Figure 6 · p. 6, C. Reducing Localization Uncertainty · p. 6
- Spatial pose information is strongest in rich map areas and weak on linear road segments, whose geometry is insufficient for 3DOF localization.Heading angle was corrected at both straight segments and more complex turns and intersections.Source: C. Reducing Localization Uncertainty · pp. 5–6
- 0.31 nats of average uncertainty reduction was achieved near intersections; with initial angular uncertainty σ = 0.8 radians (45 degs), standard deviation fell by 0.2 radians (11 degs).Spatial uncertainty with a 2 m prior typically fell about 20 meters before an intersection, where the forward camera can still see it.Source: C. Reducing Localization Uncertainty · p. 6
- Across tested prior-uncertainty values, average improvement around intersections was positive for both spatial and heading uncertainty, so localization did not worsen on average.The evaluation starts from a GPS-derived pose with assumed Gaussian uncertainty; training and test roads and intersections were disjoint.Source: C. Reducing Localization Uncertainty · pp. 5–6
D. Coarse Grained Localization
The system uses camera-implied steering direction to match visual road topology against map locations, supporting coarse localization without a pose prior. In five test examples, it selected the correct map location, while some ambiguities corresponded to similar road patterns.
D. Coarse Grained Localization
- The comparison conditions each image from location i on map patches from location j, showing camera–map correspondence and identifying gross map-pose errors.The task targets rough localization from visual surroundings without a pose prior.Source: Figure 7 · p. 6, D. Coarse Grained Localization · p. 6
- The system correctly matched all five example images to their map locations, producing a strong diagonal in the confusion matrix.It sometimes assigned high probability to maps with similar upcoming turns or a driveway resembling a turn.Source: D. Coarse Grained Localization · p. 6, Figure 7 · p. 6
V. CONCLUSION
The conclusion presents a variational model that combines coarse roadmaps and perceptual data for steering, navigation, and pose estimation. It reports reduced pose uncertainty, while online discrete road map-matching remains future work.
V. CONCLUSION
- The variational model combines coarse roadmaps with raw perceptual data to learn steering, estimate possible control commands, and navigate using a routed map.The learned network also supports pose estimation, localization correction, and place recognition from map and visual information.Source: §V · p. 6
- The pose-estimation algorithm uses the learned network to reason about robot localization and demonstrates reduced uncertainty in the resulting pose.Source: §V · p. 6
- Online integration of localization with discrete road map-matching onboard the full-scale vehicle is left for future work.The authors also call for a more robust comparison with human-driver localization.Source: §V · p. 6