Source-linked AI summary
An Overview of Depth Cameras and Range Scanners Based on Time-of-Flight Technologies
Radu Horaud, Miles Hansard, Georgios Evangelidis, Clement Menier
TL;DR
The paper addresses how TOF cameras measure scene depth and how their designs, calibration, and integration with color cameras can be understood across pulsed-light and continuous-wave technologies. It reviews physical principles, scanners, cameras, prototypes, and commercial systems. The review distinguishes pulsed-light systems suited to outdoor and long-range sensing from CW systems suited mainly to indoor short-range measurement, while identifying resolution, accuracy, and interference limitations.
Problem
Range-sensing applications need arrays of depth measurements, but TOF technologies differ in measurement principles, operating conditions, and limitations.
Method
The paper reviews pulsed-light and continuous-wave TOF principles, device designs, calibration models, commercial systems, prototypes, and TOF-color fusion.
Results
Pulsed-light LIDAR generally supports outdoor operation and ranges up to a few kilometers, whereas CW cameras usually operate indoors over shorter distances.
Takeaways & Limitations
TOF LIDAR systems are used for autonomous vehicle driving and robot navigation, including obstacle, car, and pedestrian detection and road following.
Takeaways & Limitations
Spatial resolution is typically 10 to 100 times lower than video cameras, while depth accuracy can range from a few centimeters to several meters.
Abstract
from arXiv · showhide
Time-of-flight (TOF) cameras are sensors that can measure the depths of scene-points, by illuminating the scene with a controlled laser or LED source, and then analyzing the reflected light. In this paper, we will first describe the underlying measurement principles of time-of-flight cameras, including: (i) pulsed-light cameras, which measure directly the time taken for a light pulse to travel from the device to the object and back again, and (ii) continuous-wave modulated-light cameras, which measure the phase difference between the emitted and received signals, and hence obtain the travel time indirectly. We review the main existing designs, including prototypes as well as commercially available devices. We also review the relevant camera calibration principles, and how they are applied to TOF devices. Finally, we discuss the benefits and challenges of combined TOF and color camera systems.
1 Introduction
The introduction motivates range sensing as a broadly useful technology and frames TOF systems as LIDAR devices that produce depth measurements through pulsed-light or continuous-wave methods. It also previews the paper’s review of technologies, calibration, 3D reconstruction, and TOF-color fusion.
- Range sensing supports mapping, surveying, planetary landing, 3D modeling, navigation, recognition, motion capture, and 3D reconstruction.
- LIDAR estimates distance by illuminating an object with a collimated laser beam and detecting reflected light, but highly specular materials are excluded.
- Most applications require arrays of depth measurements, motivating scanning mechanisms or scannerless devices for spatial coverage.
- Pulsed-light sensors measure round-trip pulse time and support adverse outdoor and long-distance operation, whereas CW sensors infer travel time from phase differences and face phase-wrapping ambiguity.
- The paper reviews TOF technologies, designs, calibration models, 3D image and point-cloud construction, and fusion with color cameras.
2 Pulsed-Light Technology
Pulsed-light TOF systems emit laser pulses and measure their return time using photodetector arrays and timing circuits. The section covers scanner and flash-LIDAR architectures, along with SPAD-based detection and the timing, signal, and semiconductor principles underlying them.
- Pulsed-light sensors use emitted laser pulses, reflected-light photodiode arrays, and time-to-digital or time-to-amplitude circuitry.
- TOF range scanners use rotating mirrors, while 3D flash LIDAR cameras diffuse one laser beam across a scene and image it with a 1-D or 2-D detector array.
- A TOF camera requires each photodetector to have timing circuitry, with received energy decreasing as the inverse square of distance and being divided further by optical diffusion.
- Centimeter-scale depth precision requires subnanosecond timing, while high detection bandwidth also increases noise and competes with weak received signals.
- SPADs operate in Geiger mode, detect individual photons, generate fast electrical pulses, and can directly trigger integrated digital CMOS circuitry.
3 LIDAR Cameras
LIDAR cameras include mechanically scanned range scanners and Flash LIDAR systems that form depth images without mechanical scanning. The reviewed devices span laboratory prototypes and commercial products with different fields of view, ranges, resolutions, frame rates, and calibration-relevant measurement characteristics.
- Velodyne Range Scanners: Velodyne HDL-64E uses 64 semiconductor lasers on a rotating head to generate a 360° horizontal field of view.Each laser-detector pair is aligned at a predetermined vertical angle; 16- and 32-laser variants offer reduced vertical resolution.
- Velodyne Range Scanners: Over 1.3 million data points are generated each second, independent of the spin rate, at rates an order of magnitude above conventional designs.The HDL-64E manufacturer specification is approximately 2 cm accuracy, while experiments reported 3.0–3.5 cm measurement noise.
- Toyota’s Hybrid LIDAR Camera: Toyota’s prototype combines a pulsed laser, polygonal mirror, concave imaging mirror, and a 32-macro-pixel vertical CMOS line sensor for scanned depth measurements.Its polygonal mirror covers a contiguous 4.5° vertical field of view while horizontal scanning spans 170°.
- 3D Flash LIDAR Cameras: 3D Flash LIDAR forms a depth value at each pixel from a single flood-illuminating laser pulse, eliminating the mechanical scanning mechanism used by standard LIDAR devices.Flash LIDAR can operate as a 3D video camera delivering images at up to 30 FPS, although reflected energy is divided among multiple detectors.
- 3D Flash LIDAR Cameras: Commercial Flash LIDAR cameras vary widely in range, field of view, resolution, and frame rate across available designs.Examples include Advanced Scientific Concepts cameras with 128×128 sensors and 10–30 FPS, and Odos Imaging’s 1280×1024 camera with ranges up to 10 m and frame rates up to 450 FPS.
4 Continuous-Wave Technology
Continuous-wave TOF cameras infer depth indirectly from the phase shift between emitted and received modulated light. Their pixels demodulate the signal to estimate phase, amplitude, and background offset, while phase wrapping, integration time, and other effects constrain measurements.
- Measurement principle: CW sensors estimate distance from the phase difference between emitted and received sinusoidally modulated light, obtained through cross-correlation demodulation.The known modulation frequency relates the measured phase to distance.
- Measurement principle: Each pixel independently demodulates the received signal and measures phase delay, amplitude, and background-light offset.The received signal is affected by reflectivity, optical attenuation, and background illumination.
- Demodulation: Four equally spaced samples within one modulation period suffice to compute the received signal’s offset, amplitude, and phase.The four-bucket method samples at 0, π/2, π, and 3π/2.
- Pixel structure: Electro-optical demodulation pixels perform light detection, correlation-based demodulation, clocking, and charge storage before producing an integrated capacitor voltage proportional to correlation.Incoming photons are converted into electron charges and then photocurrent.
- Limitations: Long integration over multiple modulation periods improves signal-to-noise ratio but introduces motion blur and prevents fast-shutter operation.CW systems also face errors from temperature, illumination, scattering, multiple paths, and simultaneous-camera interference.
- Depth estimation: Depth is derived from measured phase using light speed and modulation frequency, but phase periodicity creates an inherent wrapping ambiguity.At 30 MHz, the unambiguous range is 0 to 5 meters; lower frequency increases range while reducing accuracy.
5 TOF Cameras
The paper surveys commercially available CW-TOF cameras and summarizes their specifications, operating ranges, synchronization options, and practical trade-offs. Models differ in resolution, modulation strategy, range, frame rate, and power requirements.
- Overview: The survey selects commercially available cameras with readily available technical and scientific documentation and summarizes their main specifications in Table 3.Manufacturer-reported accuracy is omitted because precision depends on surface properties, illumination, frame rate, and other factors.
- Mesa Imaging cameras: SR4000 and SR4500 cameras provide depth, amplitude, and confidence images at 176×144 pixels; more accurate SR4000 measurements are obtained at 10–15 FPS rather than the possible 30 FPS.Both cameras are continuous-wave TOF devices manufactured by Mesa Imaging.
- Mesa Imaging cameras: SR4000 modulation settings provide maximum depths from 4.84 to 10.34 m, depending on the selected frequency.The listed settings range from 14.5–15.5 MHz and 29–31 MHz.
- Multi-camera operation: Up to six SR4000 cameras can operate synchronously, while an arbitrary number of SR4500 cameras can be combined because each unit encodes its modulation frequency differently.Both arrangements can be used with color cameras.
- Kinect v2: Kinect v2 uses a TOF sensor and multiple modulation frequencies from 10–130 MHz, while weighing 970 g and consuming 15 W.Its predecessor used structured light, weighed 170 g, and consumed 2.5 W.
- Other cameras: Other listed CW-demodulation devices include SoftKinetic DS311 and DS325, Fotonic E70 and E40, a PMD sensor chip, and Panasonic’s D-imager.The Panasonic D-imager had a range up to 15 cm and was discontinued in March.
6 Calibration of Time-of-Flight Cameras
TOF cameras are calibrated using projective camera models with intrinsic, extrinsic, and lens-distortion parameters, supplemented by TOF-specific corrections for depth errors and optical effects.
- Camera model: TOF cameras use pinhole-camera and projective-geometry models, augmented in practice by intrinsic, extrinsic, and distortion parameters.
- Intrinsic parameters: Intrinsic calibration maps normalized image coordinates to pixel coordinates using scale factors, the principal point, and a 3 × 3 matrix A.
- Extrinsic parameters: The measured distance and image ray are combined to recover 3D coordinates in the camera frame, then transformed to a common world frame using rotation and translation.
- Lens and image calibration: A shared calibration board can estimate lens parameters, while standard color-camera calibration can be applied to CW-TOF amplitude-plus-offset images.
- Depth correction: TOF depth estimates exhibit systematic nonlinear wiggling errors that can be corrected with lookup tables or regression models fitted to calibrated depth data.
- Practical considerations: Depth errors are difficult to predict because they depend on unknown object materials and scene complexity, including possible multiple-path distortion.
7 Combining Multiple TOF and Color Cameras
Multiple TOF and color cameras can be cross-calibrated and combined to address viewpoint dependence, sparse or low-resolution depth, and failures of stereo matching in poorly textured scenes.
- Motivation: A single TOF camera’s viewpoint-dependent point cloud can contain holes at depth discontinuities when viewed from another direction.
- Synchronization: Infrared band-pass filtering and distinct modulation frequencies can permit simultaneous capture by multiple TOF cameras, although user-defined frequencies are not universally available.
- Mixed systems: A TOF camera paired with two color cameras combines active range sensing with passive stereo, whose dense reconstruction requires sufficient texture and nonrepetitive patterns.
- Mixed systems: TOF measurements remain useful in untextured areas where stereo matching commonly fails, while mixed systems are recommended when high-resolution 3D maps are required.
- Cross-calibration: Multiple TOF and color cameras can be cross-calibrated through geometric models, detected calibration vertices, stereo reconstruction, or calibration-plane constraints.
- System configurations: Several TOF systems can be combined for 3D reconstruction, and synchronized captures can align color images with TOF depth and amplitude images.
8 Conclusions
The paper surveys pulsed-light and continuous-wave TOF technologies, their calibration and applications, and the trade-offs governing practical sensor choice. TOF devices offer broad utility, but resolution, accuracy, synchronization, and scene-dependent errors remain important constraints.
- Core technologies: TOF estimates depth by measuring light’s round-trip travel time, using pulsed-light or continuous-wave modulation technologies.
- Limitations: Spatial resolution is typically 10 to 100 times lower than video-camera resolution, while depth accuracy ranges from a few centimeters to several meters.
- Calibration: TOF cameras are modeled as pinhole cameras, allowing standard intrinsic, extrinsic, and distortion calibration techniques, including calibration from amplitude-plus-offset and depth images.
- Applications: TOF cameras and scanners support applications including multimedia interfaces, autonomous navigation, industrial sensing, robotic systems, and planetary or space exploration.
- Technology trade-offs: Pulsed-light systems support adverse outdoor lighting and long-distance measurements, whereas continuous-wave systems usually operate indoors over shorter distances and suffer phase-wrapping ambiguity.
- Conclusion: The paper concludes that TOF technologies are used in many configurations and may increasingly affect everyday life through refinement, commoditization, and miniaturization.