Evaluating and Improving the Depth Accuracy of Kinect for Windows v2

Lin Yang; Longyu Zhang; Haiwei Dong; Abdulhameed Alelaiwi; Abdulmotaleb El Saddik

Source-linked AI summary

Evaluating and Improving the Depth Accuracy of Kinect for Windows v2

Lin Yang, Longyu Zhang, Haiwei Dong, Abdulhameed Alelaiwi, Abdulmotaleb El Saddik

arXiv:2212.13844v1Dec 25, 2022cs.MM

TL;DR

The paper addresses the unanswered accuracy of Kinect v2 by measuring its depth behavior, modeling spatial accuracy, evaluating temporal variance, and applying multi-sensor trilateration. Experiments characterize Kinect v2 performance and report that trilateration works with three sensors even beyond four meters, while reflective or light-absorbing materials can make depth unreliable.

Problem
Prior work had not evaluated Kinect v2 accuracy, despite its use of depth measurements in applications such as 3D data fusion and body tracking.
Method
The paper measures accuracy distribution, depth resolution, depth entropy, edge noise, and structural noise, and applies least-squares trilateration to measurements from multiple Kinect v2 sensors.
Results
Three-sensor trilateration works well even when the sensors are positioned more than four meters from the target.
Takeaways & Limitations
Kinect v2 shows acceptable performance and potential for applications including entertainment, education, and medicine.
Takeaways & Limitations
Reflective or light-absorbing materials can make Kinect v2 depth values unreliable or unavailable, and opposing sensors can cause image regions to disappear.

Abstract

from arXiv · show

Microsoft Kinect sensor has been widely used in many applications since the launch of its first version. Recently, Microsoft released a new version of Kinect sensor with improved hardware. However, the accuracy assessment of the sensor remains to be answered. In this paper, we measure the depth accuracy of the newly released Kinect v2 depth sensor, and obtain a cone model to illustrate its accuracy distribution. We then evaluate the variance of the captured depth values by depth entropy. In addition, we propose a trilateration method to improve the depth accuracy with multiple Kinects simultaneously. The experimental results are provided to ascertain the proposed model and method.

I. INTRODUCTION

The paper addresses the lack of prior Kinect v2 accuracy evaluation by characterizing depth accuracy and variance, then proposes multi-sensor trilateration to improve measurements.

Kinect v2 uses Time-of-Flight sensing, improving captured depth-image quality compared with Kinect v1’s structured-light method.
The proposed multi-Kinect method applies trilateration to improve depth accuracy using measurements from multiple sensors.Three Kinects measure the same target, whose position is estimated from intersecting spheres under a least-squares solution.
The paper measures Kinect v2 depth accuracy and models its spatial distribution with a 3D cone.
Depth entropy is used to evaluate variance in captured depth values over time.
The study builds on prior work using multiple sensors to address limitations in single-sensor resolution, field of view, or accuracy.

III. KINECT ACCURACY ASSESSMENT

The assessment defines depth accuracy relative to a planar surface and evaluates its spatial distribution using perpendicular planar measurements at multiple distances and orientations.

A. Accuracy Distribution: The experiment uses a 19-inch screen or flat wall as a planar surface for collecting Kinect v2 depth data.The surface is positioned perpendicular to the sensor or at a specified angle.
A. Accuracy Distribution: Depth accuracy is the difference between the true planar-surface depth and the mean of Kinect v2’s measured depth values.The true distance is obtained with a laser distance meter, while the measured depth matrix comes from the Kinect SDK.
A. Accuracy Distribution: Accuracy distribution is evaluated in both horizontal and vertical directions at distances of 1m, 2m, 3m, and 4m.Distance-value matrices in millimeters are exported from the Kinect SDK at each tested location.

B. Depth Resolution

Depth resolution is defined as the smallest detectable depth difference and is measured across distances and surface angles. The assessment also characterizes temporal variation, edge noise, and structural noise in Kinect v2 depth measurements.

B. Depth Resolution: Depth resolution is the minimum detectable difference between adjacent depth values over a continuous distance range.Smaller detectable differences indicate more precise depth measurements and higher resolution.
B. Depth Resolution: The resolution experiment uses planar surfaces angled at 45 or 60 degrees and positioned 1m to 4m from Kinect v2.Measurements are repeated at 1m, 1.5m, 2m, 2.5m, 3m, 3.5m, and 4m.
C. Depth Entropy: Depth entropy uses temporal depth-value variation to illustrate Kinect v2 stability and reliability.The variation can reach several millimeters even when the sensor setup is stationary.
D. Edge Noise / E. Structural Noise: Edge noise denotes differing depth values near sharp object contours, while structural noise appears as ring-shaped depth distributions on planar surfaces.Edge-noise tests use circular planar surfaces; structural-noise tests use walls positioned 1m, 2m, 3m, and 4m away.

IV. MULTI-KINECT TRILATERATION

The multi-Kinect trilateration method estimates a target position from redundant distance measurements represented as sphere equations. It linearizes these equations and solves them in the least-squares sense, with SVD available for ill-conditioned systems.

A. Trilateration Principle: Multiple Kinects measure distances to the same target, modeling each sensor as a sphere center with the measured distance as its radius.The target is estimated from the common intersection of the spheres.
B. Trilateration for Improving Multi-Kinect Accuracy: The target position is defined by sphere equations using each Kinect’s coordinates and its measured target range.The setup assumes n Kinect v2 sensors can simultaneously observe the target.
B. Trilateration for Improving Multi-Kinect Accuracy: Measurement noise is represented by error terms associated with each Kinect’s sphere equation.The formulation accounts for hardware accuracy limits, clock offsets, and other disturbances.
B. Trilateration for Improving Multi-Kinect Accuracy: Subtracting a reference sphere converts the system into n − 1 linear equations with three unknown target coordinates.The resulting coefficient matrix contains differences between Kinect sensor coordinates.
B. Trilateration for Improving Multi-Kinect Accuracy: The pseudo-inverse solution gives the least-squares target position that minimizes the summed residual errors.Singular value decomposition can compute the pseudo-inverse when A^T A is singular or ill-conditioned.

C. A Case Study

The case study uses three Kinect v2 sensors and trilateration to estimate a target’s optimized position from inter-sensor distances and depth-angle measurements.

Set-up: Three Kinect v2 sensors measure the same planar target, using averaged depth values from 20×20-pixel center regions.The target is positioned more than 4m away, where a single sensor may not generate accurate depth values.
Inputs: The trilateration input combines measured Kinect-to-Kinect distances with each sensor’s observed depth-angle pair.The measured distances are lk1k2, lk1k3, and lk2k3; the observed pairs are (θk1, Dk1), (θk2, Dk2), and (θk3, Dk3).
Procedure: The method places the three Kinects as sphere centers, computes angles and radiuses from sensor measurements, and solves for the target position O′.The Kinect sensors are configured on the same horizontal plane, with z_k1 = z_k2 = z_k3 = 0.
Evaluation: The estimated target position is compared with three positions calculated independently from Kinects k1, k2, and k3.Each single-Kinect position uses that sensor’s location and corresponding depth-angle pair according to geometry rules.
Visualization: The three-sphere construction identifies O′ as the intersection point associated with Kinects k1, k2, and k3.Figure 4 presents both the coordinate-system setup and the sphere-intersection visualization.

A. Accuracy Evaluation

This subsection explains experiment results in correspondence with the subsections of Section III.

A. Accuracy Evaluation: The subsection explains the experiment results.
A. Accuracy Evaluation: The results correspond to the subsections in Section III.
A. Accuracy Evaluation: Section III provides the subsections used for organizing the experiment-results discussion.

1) Accuracy Distribution:

The accuracy evaluation maps depth measurements of a planar surface across key image positions and models the resulting error distribution as an elliptical cone.

1) Accuracy Distribution:: Depth frames are recorded with the planar surface placed at key positions, then summarized by mean depth, standard deviation, and depth range.The depth frame contains 512×424 pixels, and Figure 5 illustrates the central key position at 2m.
1) Accuracy Distribution:: Kinect v2’s accuracy error distribution follows an elliptical cone with 60° vertical and 70° horizontal angles.The distribution is summarized across all key positions and visualized in Figure 6.
1) Accuracy Distribution:: The accuracy regions classify average error as less than 2mm, between 2mm and 4mm, or more than 4mm.Green, yellow, and red identify the three corresponding accuracy areas in the horizontal and vertical planes.
1) Accuracy Distribution:: Figure 5 depicts the depth-value distribution for a planar surface represented by a 70×50-pixel region.The figure provides a spatial view of captured values rather than the cone-level error summary.

2) Depth Resolution:

The depth-resolution study examines planar surfaces at different distances and tilt angles, finding coarser measurements and faster variability growth beyond 2m.

2) Depth Resolution:: A larger tilt angle produces lower depth resolution and larger standard deviation.The comparison uses planar surfaces positioned at 45° and 60° toward Kinect v2.
2) Depth Resolution:: Beyond 2m, maximum resolution and standard deviation increase faster for both 45° and 60° tilt conditions.This marks a sharper degradation regime in the reported distance trends.
2) Depth Resolution:: Depth values vary within 6mm over time across repeated frames recorded at planar-surface distances of 1m, 2m, 3m, and 4m.Thirty depth frames are recorded at each position, with changing pixel-value distributions over time.
2) Depth Resolution:: Mean entropy increases rapidly after the distance exceeds 2m.The entropy analysis uses repeated depth frames to characterize temporal variation in pixel values.

4) Edge Noise:

The Kinect v2 exhibits edge and structural noise in planar-surface depth captures, while trilateration combines measurements from three sensors to estimate target position.

Edge Noise: Zero-value contours appear around planar edges even within Kinect v2’s measurable range, with widths of roughly 2–4mm at 1m.The observed contour is 1–2 pixels wide, although the plane lies within the stated 0.5m–4m range.
Structural Noise: Recorded depth values form rings around a planar surface, decreasing as pixels move farther from the center.The paper attributes this phenomenon tentatively to diffraction from random depth-pixel variance.
Multi-Kinect Trilateration: Three Kinect v2 sensors are arranged at the vertices of an isosceles triangle, and trilateration computes an optimized target position from their measurements.The method first calculates sensor-vertex positions and radiuses before computing the optimized position.
Multi-Kinect Trilateration: 10.66mm is the smallest reported measurement error for Kinect k3, compared with 64.23mm for Kinect k1 and 78.99mm for Kinect k1.The passage reports the proposed trilateration error as lying between the individual sensor errors.
Multi-Kinect Trilateration: 51.22mm is the mean measurement error of the three Kinect sensors, and the multi-Kinect trilateration result is reported as better than this mean.Kinect k3’s lower error is associated with its perpendicular orientation, whereas the other two sensors use a 60-degree angle.

VI. CONCLUSION

The study finds Kinect v2 acceptable for practical depth sensing under favorable positioning and reports that multi-Kinect trilateration works beyond four meters. Reflective or light-absorbing materials, sensor interference, and system requirements constrain use.

VI. CONCLUSION: Kinect v2 has good accuracy when objects are positioned within the green regions shown in Fig. 6.This conclusion is based on the paper’s experimental evaluation of accuracy distribution and related depth-camera properties.
VI. CONCLUSION: Multi-Kinect trilateration works well when sensors are positioned more than four meters from the target object.The approach uses three Kinect sensors to improve accuracy.
VI. CONCLUSION: Reflective or light-absorbing materials can make Kinect v2 depth values unreliable or impossible to determine, and facing sensors can cause image regions to disappear.The paper attributes these effects to insufficient or interfering infrared light.
VI. CONCLUSION: Kinect v2 offers acceptable performance at a relatively low price for potential use in entertainment, education, and medicine.The paper contrasts its price with professional depth cameras and tracking systems such as Vicon.