Source-linked AI summary
3D Visual Perception for Self-Driving Cars using a Multi-Camera System: Calibration, Mapping, Localization, and Obstacle Detection
Christian Häne, Lionel Heng, Gim Hee Lee, Friedrich Fraundorfer, Paul Furgale, Torsten Sattler, Marc Pollefeys
TL;DR
The paper addresses 3D visual perception for self-driving cars using multiple fisheye cameras to obtain surround coverage without treating cameras independently. It presents a calibrated pipeline for mapping, localization, and obstacle detection, and reports practical feasibility on V-Charge vehicles. The system uses four fisheye cameras and directly processes fisheye images with less than 10% runtime increase over pinhole processing.
Problem
Self-driving cars need 3D environmental perception for localization and obstacle detection, but multi-camera systems and fisheye images require adapted processing and accurate calibration.
Method
The paper presents a multi-camera pipeline combining calibration, ego-motion estimation, sparse and dense mapping, map-based localization, and online obstacle detection for four fisheye cameras.
Results
The system was used successfully on V-Charge self-driving cars, while direct fisheye-image matching increased runtime by less than 10% over equal-resolution pinhole processing.
Takeaways & Limitations
A four-fisheye-camera system can provide a practical surround-perception pipeline for autonomous valet parking while keeping camera count, cost, and complexity low.
Takeaways & Limitations
The pipeline does not yet address dynamic objects or semantic segmentation, which are identified as future work.
Abstract
from arXiv · showhide
Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction.
1. Introduction
The paper presents a multi-fisheye-camera 3D perception pipeline for autonomous driving, targeting full surround coverage while supporting calibration, mapping, localization, and obstacle detection. Developed for V-Charge, the system integrates offline and online components for autonomous navigation and valet parking.
- 3D visual perception is required to estimate environmental structure around the car for visual localization and obstacle detection.
- A multi-camera system provides 360° surround perception, addressing blind spots that limit narrower camera configurations.
- The pipeline calibrates cameras, estimates vehicle motion, builds sparse maps, localizes against them, generates dense maps, and detects obstacles.
- The paper emphasizes an overview of design choices and mathematical concepts because the complete implementation cannot be covered in detail.
- V-Charge developed the system for fully autonomous valet parking and charging, including indoor parking garages without GPS.
- The four-camera platform uses front, rear, and side-view-mirror fisheye cameras to support the multi-camera system.Each camera has a nominal 185° FOV and produces 1280 × 800 images at 12.5 fps.
2. Calibration
The paper presents automatic SLAM-based calibration for multi-camera systems, combining camera modeling, naturally occurring features, vehicle motion, and joint optimization. The method avoids fiducial targets and overlapping fields of view while producing calibration parameters and a sparse map for localization.
- Motivation: Calibration is essential because inaccurate or drifting intrinsic and extrinsic parameters degrade mapping, localization, and obstacle detection.Environmental changes, wear, vibrations, encoder resolution, and odometry drift motivate frequent or automatic recalibration.
- SLAM-based Extrinsic Calibration: The SLAM-based method estimates extrinsic parameters from naturally occurring features and vehicle motion instead of a fiducial target, while recovering metric scale from wheel odometry.It jointly estimates inter-camera transformations and a sparse environmental map without requiring overlapping fields of view or initial extrinsic guesses.
- Camera Model: The fisheye camera model combines unified projection with radial-tangential distortion, using intrinsic parameters to map 3D points to image coordinates.The model includes the mirror parameter, distortion parameters, focal lengths, principal point coordinates, and intrinsic projection matrix.
- Joint Optimization: The calibration solves a robust non-linear least-squares problem combining image reprojection residuals with relative pose residuals from wheel odometry.The projection function relates scene points, camera transformations, and vehicle poses; a robust cost reduces the influence of outliers.
- Initialization and Refinement: Because the optimization is highly non-convex, initialization proceeds through planar hand-eye calibration, triangulation, and iterative refinement of extrinsics and scene points.Vehicle poses are later corrected using loop closures and pose-graph optimization, while temporally close inter-camera correspondences provide stronger constraints.
- Feature Matching and Results: Rectifying images onto a common plane improves fisheye-camera feature matching under viewpoint differences and distortion, while optimized parameters produce a more well-defined sparse map.The system maintains local frame histories for exhaustive cross-camera matching, with longer histories increasing matching time.
3. Sparse Mapping
The sparse-mapping framework jointly processes calibrated multi-camera imagery as a generalized camera, using minimal motion solvers and loop closures to build large-area 3D maps efficiently. The resulting trajectory stays locally close to GPS/INS and has below-3 m global error in the reported example, while multi-camera imagery increases loop-closure detections in opposing-direction paths.
- 3.1. Ego-Motion Estimation: Joint generalized-camera ego-motion estimation keeps the motion of all rigidly mounted cameras consistent instead of processing each camera independently.The approach uses Plücker line representations because generalized-camera rays do not share one projection center.
- 3.1. Ego-Motion Estimation: Ackermann-constrained motion reduces the relative-motion model to 2 degrees of freedom, enabling a two-correspondence solver with up to 6 solutions.The unknowns are the relative yaw angle θ and translation scale ρ.
- 3.1. Ego-Motion Estimation: For inlier ratio v = 0.5, the two-point solver requires 16 RANSAC iterations, compared with 603606 iterations for the 17-point solver.The authors report that the two-point solver is more efficient inside RANSAC than the 17-point and 6-point alternatives.
- 3.1. Ego-Motion Estimation: The solver integrated with SfM, bundle adjustment, and loop-closure detection produced a trajectory with locally close GPS/INS agreement and global error below 3 m.The reported global shape difference partly reflects that the mapped parking lot is not exactly planar.
- 3.2. Loop-Closure Detection: Loop-closure detection and pose-graph optimization correct accumulated drift by adding constraints between revisited poses and optimizing the trajectory and map.Vocabulary-tree retrieval followed by geometric verification supplies candidate loop-closure edges.
- 3.2. Loop-Closure Detection: The multi-camera system detects many more loop closures than a forward-looking monocular camera, especially when the car travels opposite to a previously visited path.In another example, detections were similar when most paths faced the same direction, and optimized 3D points aligned well with buildings in the satellite image.
- 3.3. Summary: Known camera calibration lets the sparse-mapping framework treat rigidly mounted cameras jointly, enabling efficient mapping of large areas.The minimal solvers form the backbone of the offline sparse-mapping approach.
4. Localization
Localization estimates the rigid pose between the map’s world frame and the multi-camera frame from 2D–3D correspondences, while exploiting the cameras’ known geometric relations. The reported visual localization results are accurate with only small errors, although scene changes can cause gaps that wheel odometry can bridge.
- 4. Localization: Localization determines the rigid transformation between the map’s fixed world frame and the multi-camera frame from 2D–3D correspondences.The method estimates the generalized-camera pose rather than solving separately for each camera.
- 4. Localization: The generalized-camera pose solver requires three 2D–3D correspondences, matching the requirement stated for calibrated perspective-camera pose estimation.Plücker lines represent rays that do not meet at a single projection center.
- 4. Localization: Equal distances between corresponding map-frame and multi-camera-frame points provide equations for the unknown depths, after which absolute orientation computes the global pose.The depth variables λ_i define the 3D point positions in the multi-camera frame.
- 4. Localization: The evaluation builds a sparse map from one GPS/INS-referenced image set and tests localization on a separate image set collected at another time.Figure 13 reports translational and rotational errors against GPS/INS ground truth.
- 4. Localization: The visual localization algorithm produces accurate results with only small errors, but scene changes can cause missing trajectory segments.The paper states that wheel odometry can bridge these missing parts in practice.
5. Dense Mapping
The dense mapping system directly matches fisheye images to compute real-time depth maps, then fuses them into dense height maps for later automated driving. Its parking-garage evaluation achieved a mean error of 7.06cm over nine measurements.
- Depth Map Computation: Dense multi-view stereo matching directly processes fisheye images and produces real-time depth maps for automated driving.The method avoids requiring prior stereo rectification and incorporates camera models for fisheye images.
- Depth Map Computation: Plane sweeping compares warped images against a reference across proxy planes, selecting the plane with lowest pixel-wise dissimilarity.The system uses negative ZNCC and aggregates costs across multiple images.
- Depth Map Computation: Online mapping averages dissimilarity costs for efficiency, while offline mapping uses temporal averages that are more robust to occlusions.The offline procedure separately averages images before and after the reference image and keeps the lower dissimilarity.
- Depth Map Computation: Direct fisheye matching increases running time by less than 10% compared with same-resolution pinhole processing.This preserves the fisheye camera’s full field of view without requiring initial unwarping.
- Height Map Fusion: Depth maps are fused into a voxel grid, where signed weights indicate free or occupied space, before extracting a regularized height map.A two-pass optimization identifies observed regions and determines their height values.
- Height Map Fusion: 7.06cm mean error was measured over nine parking-garage distances, indicating accuracy sufficient for autonomous driving.The resulting height-map method supports mapping large parking spaces and storing them efficiently.
6. Applications
The pipeline applies sparse maps to localization and structure-based calibration, while dense depth maps support real-time obstacle extraction. Structure-based calibration avoids infrastructure, and obstacle maps achieved errors below 10cm in low-speed driving and parking experiments.
- Structure-Based Calibration and Localization: Sparse maps generated during calibration and mapping support visual localization and structure-based calibration of multi-camera systems.Structure-based calibration estimates camera poses from 2D-3D correspondences and optimizes extrinsic parameters and vehicle poses.
- Structure-Based Calibration and Localization: Structure-based calibration uses many 2D-3D correspondences across frames to calibrate multiple systems accurately, quickly, and without infrastructure.The authors attribute its accuracy and repeatability to the accurate sparse map and the large number of correspondences.
- Obstacle Detection: Obstacle detection converts each depth map into a 2D occupancy grid, extracts the closest occupied cell along each ray, and refines its position using 3D points.Obstacle maps are fused over time and across cameras to remove remaining noise.
- Obstacle Detection: 12.5Hz processing runs on a single NVIDIA GeForce GTX 680 computer for the left, right, and front-facing cameras.The obstacle map is reported as accurate enough for low-speed driving and parking, with errors below 10cm.
- Obstacle Detection: Errors below 10cm were measured for the obstacle map in low-speed driving and parking experiments.The full evaluation is referenced separately.
7. Conclusions
The paper concludes that a four-fisheye-camera pipeline can provide calibration, sparse and dense mapping, localization, and online obstacle detection for self-driving cars. Its cost-minimal, minimally overlapping camera arrangement covers 360 degrees, while dynamic objects and semantic segmentation remain future work.
- Conclusions: The complete pipeline supports calibration, metric sparse and dense mapping, map-based localization, and online obstacle detection using four fisheye cameras.The system was used successfully on the self-driving cars of the V-Charge project.
- Conclusions: Four minimally overlapping fisheye cameras provide 360-degree coverage while keeping camera count, cost, and system complexity low.The paper notes that more cameras would provide denser sensing but increase complexity and cost.
- Conclusions: Direct processing of fisheye images avoids unwarping to a pinhole model and preserves usable field of view.The approach handles the strong distortion of fisheye lenses with an appropriate projection model.
- Conclusions: Dynamic objects and semantic segmentation are identified as future work for improving sparse or dense maps, localization, and obstacle detection.Semantic class information is given as a possible source of valuable information for these tasks.