PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable iToF Depth Sensing via Sensor-Intrinsic Uncertainty Modeling and State-Space Restoration

Yansong Du, Yutong Deng, Yuting Zhou, Zhancong Xu, Yingjia Lu, Mengdi Wang, Feiyu Jiao, Bangyao Wang, Zhaoxiang Jiang, Xun Guan

arXiv:2609.05507v1physics.opticscs.CV

TL;DR

iToF restoration faces a synthetic-to-real gap because common noise assumptions miss sensor-dependent depth uncertainty. The paper derives pixel-wise uncertainty from calibrated tap statistics for heteroscedastic synthesis and uncertainty-aware DVSS restoration; the full objective reaches 40.85 dB PSNR and 2.54 mm MAE.

  • Problem

    Spatially uniform or range-only Gaussian noise does not capture real iToF’s range- and signal-dependent uncertainty, contributing to a synthetic-to-real distribution gap that limits robustness.

    Source: §I · p. 1

  • Method

    The framework derives pixel-wise depth uncertainty from tap statistics and uses it for heteroscedastic depth synthesis and uncertainty-aware DVSS restoration.

    Source: §I · pp. 1–2

  • Results

    40.85 dB PSNR, 0.9987 SSIM, 82.30 mm2 MSE, and 2.54 mm MAE are achieved by the full objective.

    Source: C. Ablation Study · p. 8

  • Takeaways & Limitations

    The framework couples sensor-level uncertainty modeling with restoration for accurate, structure-preserving depth recovery.

    Source: §VI · p. 9

Abstract

from arXiv · show

Indirect time-of-flight (iToF) cameras provide compact and cost-effective dense depth measurements, but their ranging accuracy is often degraded by sensor-intrinsic uncertainty under practical imaging conditions. Spatially uniform or range-only Gaussian perturbations cannot accurately reproduce the range-dependent and signal-dependent noise characteristics of real iToF measurements, leading to a synthetic-to-real gap for learning-based restoration. To address this problem, we propose a joint depth-uncertainty modeling and restoration framework for reliable iToF sensing. A sensor-intrinsic depth-uncertainty model is first developed from calibrated tap responses, returned-signal levels, and sensor noise statistics through a depth-oriented weighted least-squares formulation. The resulting pixel-wise uncertainty is used for heteroscedastic depth synthesis and uncertainty-aware restoration supervision. Based on this heteroscedastic data synthesis, we further develop a U-shaped restoration network with Depth Visual State Space (DVSS) blocks, which combine long-range state-space modeling with convolutional spatial-channel refinement for structure-preserving depth recovery. Experiments on synthetic data and measurements captured by an in-house iToF prototype validate the proposed uncertainty model under varying range and returned-signal conditions. Controlled comparisons with fixed and range-aware Gaussian noise, together with evaluations on U-Net, Restormer, and DVSS, further demonstrate that the proposed synthesis consistently benefits different restoration backbones. The complete framework achieves 40.85~dB PSNR and 2.54 mm MAE on the synthetic test set, and 35.42 dB PSNR and 4.87 mm MAE on real iToF measurements.

I. INTRODUCTION

iToF depth uncertainty varies with range and returned-signal strength, while simplified noise models fail to match real measurements. The paper models pixel-wise uncertainty from tap statistics and uses it for heteroscedastic synthesis and uncertainty-aware restoration.

I. INTRODUCTION

  • Spatially uniform or range-only Gaussian perturbations cannot represent the joint effects of photon statistics, sensor noise, range, and local signal strength.This mismatch limits robustness across devices, distances, and acquisition conditions.Source: §I · p. 1
  • The framework derives per-pixel depth uncertainty from signal-dependent counting statistics and sensor-intrinsic noise using depth-oriented weighted least squares.The resulting uncertainty supports heteroscedastic noisy-depth synthesis that reflects range- and signal-dependent measurement behavior.Source: §I · p. 2
  • A U-shaped DVSS restoration network uses propagated uncertainty for heteroscedastic synthesis and uncertainty-aware training, while combining long-range modeling with local geometric refinement.Its convolutional refinement branch preserves local details and depth discontinuities.Source: §I · pp. 1–2
  • Experiments on synthetic data and in-house prototype measurements validate the uncertainty model across range and returned-signal conditions, with synthesis benefits across U-Net, Restormer, and DVSS.The reported benefit is independent of a specific restoration architecture.Source: §I · p. 2

II. RELATED WORK

Prior ToF work addresses sensor errors and learning-based restoration, but common synthetic noise assumptions do not capture coupled iToF uncertainty. This work uses calibrated tap statistics to construct pixel-wise uncertainty for training and evaluates a restoration network built around that synthesis.

II. RELATED WORK

  • Prior work spans physical modeling, calibration, learned restoration, sensor-noise statistics, and efficient long-range modeling, while simplified synthetic noise can leave a distribution shift and unstable deployment.Existing correction methods may depend on camera configuration, calibration, or acquisition assumptions; CNNs and attention also face long-range context or computational constraints.Source: §II · pp. 2–3

III. METHOD

  • The synthesis pipeline uses clean depth, returned-signal level, and sensor parameters to estimate local depth variance from tap statistics, then trains a U-shaped network with DVSS blocks.The variance determines each pixel’s perturbation scale for heteroscedastic depth synthesis.Source: §III · p. 3

A. iToF Tap Response Model

  • The tap model represents measurements using depth-dependent calibrated responses and signal-dependent counting noise plus a signal-independent sensor noise floor.Sensor parameters are calibrated independently; the response uses a phase sweep, while dark offset and residual noise are estimated from emitter-off captures.Source: A. iToF Tap Response Model · p. 3

B. Depth-Oriented Weighted Least-Squares Noise Modeling

The depth-oriented weighted least-squares formulation estimates local uncertainty directly for depth and returned-signal strength, rather than treating phase as an intermediate target. Its covariance yields a per-pixel depth variance governed by signal, response sensitivity, and sensor noise.

B. Depth-Oriented Weighted Least-Squares Noise Modeling

  • The weighted least-squares objective treats depth and returned-signal strength as variables, evaluating local uncertainty at the operating point defined by the clean depth and signal-level maps.The signal level is included as an auxiliary variable.Source: B. Depth-Oriented Weighted… · p. 4
  • Tap measurements and their covariance feed a locally linearized weighted least-squares estimator, whose parameter covariance provides the depth-variance component.The tap expectation vector and measurement covariance characterize the measurements used by the estimator.Source: B. Depth-Oriented Weighted… · p. 4
  • Depth uncertainty depends jointly on signal level, tap-response sensitivity, and sensor noise variance; greater response sensitivity reduces variance, while weak signals or higher noise floors increase it.The formulation evaluates uncertainty around the calibrated operating point and does not replace the camera’s standard demodulation and depth-decoding pipeline.Source: B. Depth-Oriented Weighted… · p. 4

C. Physics-Guided Noisy Depth Synthesis

The synthesis process converts the per-pixel depth variance into spatially varying noisy-depth observations. Its uncertainty scale follows scene depth and signal strength rather than imposing one global noise level.

C. Physics-Guided Noisy Depth Synthesis

D. Denoising Network Architecture

The denoising network uses a U-shaped encoder–decoder with stacked DVSS blocks and skip connections. Each DVSS block combines long-range state-space modeling with convolutional refinement of local spatial and channel responses.

D. Denoising Network Architecture

  • The U-shaped network encodes multiscale features through stacked DVSS blocks, then uses symmetric upsampling and skip connections to recover fine geometric details.A Patch Embedding layer projects the input into feature space, and a Depth Head produces the denoised depth output.Source: D. Denoising Network Architecture · p. 4
  • DVSS processes features through parallel branches: a VSSB models long-range dependencies, while depthwise convolution and SCR refine local structures and channel responses.The branches are fused by channel concatenation and a 1 × 1 convolution with a residual connection.Source: D. Denoising Network Architecture · pp. 4–5
  • Fig. 2 depicts the U-shaped architecture on the left and the DVSS block and its VSSB submodule on the right.Source: Figure 2 · p. 5
  • SCR combines global channel-level context with locally detailed spatial responses to suppress stochastic depth noise while maintaining edges, thin structures, and depth discontinuities.The module’s global and local branches interact in a coarse-to-fine manner.Source: D. Denoising Network Architecture · p. 5, Figure 3 · p. 5

IV. NOISE MODEL VALIDATION

Validation with an in-house iToF camera shows that the sensor-intrinsic model captures depth-noise variation across range and returned-signal levels. Across pooled conditions, it outperforms a range-aware Gaussian baseline.

IV. NOISE MODEL VALIDATION

  • The noise-model validation uses real-scene acquisition with an in-house iToF camera and measurement system.The experimental setup and imaging system are shown in Fig. 4.Source: §IV · p. 5, Figure 4 · p. 5
  • Across 500–1500 mm, the proposed model tracks the increase in measured temporal noise, while the spatially uniform range-aware baseline deviates from the real noise.The range validation uses 100 mm intervals and compares empirical repeated-capture noise with the proposed model and a fitted range-aware baseline.Source: §IV · pp. 5–6
  • At 1500 mm, the proposed model follows the measured reduction in depth uncertainty as returned-signal level increases, unlike the baseline’s common uncertainty for pixels at that distance.The comparison varies black, gray, and white surfaces across four acquisition settings, using 100 repeated captures per condition.Source: §IV · p. 6
  • 7.7% MAPE and 0.93 R2: the proposed model substantially outperforms the range-aware Gaussian baseline across pooled range and returned-signal conditions.The evaluation compares predicted and measured depth standard deviations.Source: §IV · p. 6

V. EVALUATION OF THE PROPOSED NETWORK

Training uses synthetic iToF depth data with amplitude-derived signal cues for noise synthesis. The restoration model is optimized with uncertainty-aware reconstruction and gradient consistency.

A. Training Details

  • Training uses ToF-FlyingThings3D's 6,250 aligned 640×480 samples, with amplitude maps supplying per-pixel returned-signal cues for noise synthesis.All depth values are converted to millimeters for training and evaluation.Source: A. Training Details · p. 6
  • Models train on random 320×320 crops for 300 epochs with AdamW, a 2 × 10−4 initial learning rate, and batch size 4; evaluation uses full-resolution inputs.The checkpoint with the lowest validation MAE is selected.Source: A. Training Details · p. 6
  • The final loss combines uncertainty-weighted robust reconstruction with a gradient-consistency term for local geometric consistency.The robust penalty is a smooth approximation to L1, and the uncertainty weight derives from propagated depth uncertainty.Source: A. Training Details · p. 6
  • Uncertainty weighting prioritizes low-signal, high-uncertainty regions, while the gradient term helps preserve structural discontinuities.The horizontal and vertical gradients use finite-difference operators.Source: A. Training Details · p. 6
  • The gradient-loss weight is 0.1; controlled noise-model baselines are rescaled to match the proposed synthesis's average training-set perturbation variance.The experiments also use β = 0.5, εc = 10−3, and η = 10−6, with other training settings held unchanged for controlled comparisons.Source: A. Training Details · p. 6

B. Quantitative Comparison

The proposed method outperforms the compared iToF denoising baselines on the synthetic test set. Qualitative comparisons report cleaner, more spatially consistent depth maps with preserved contours and discontinuities.

B. Quantitative Comparison

  • 40.85 dB PSNR, 0.9987 SSIM, 82.30 mm2 MSE, and 2.54 mm MAE; versus RCF, PSNR improves 1.32 dB, SSIM 0.0015, MSE 26.1%, and MAE 45.1%.The final model uses the proposed noise-aware restoration loss; RCF is the strongest baseline under the unified PSNR definition.Source: Table I · p. 7, B. Quantitative Comparison · p. 7
  • Compared with DeepToF, ToF-KPN, and RCF, the proposed method produces cleaner, more spatially consistent depth while preserving contours and structural discontinuities.The same DVSS backbone trained with range-aware Gaussian noise also performs worse, indicating that the training distribution contributes beyond the architecture.Source: Figure 6 · p. 7, B. Quantitative Comparison · p. 7

C. Ablation Study

The ablations show that both local refinement and uncertainty-aware training contribute to depth restoration, with the full DVSS model and full objective achieving the strongest reported results.

C. Ablation Study

  • Adding SCR without DWConv improves PSNR from 39.76 dB to 40.00 dB and reduces MAE from 3.55 to 3.19 mm.Removing the entire local refinement branch produces the largest reconstruction error; replacing SCR with SE remains inferior to full DVSS.Source: C. Ablation Study · pp. 7–8
  • 40.85 dB PSNR, 0.9987 SSIM, 82.30 mm2 MSE, and 2.54 mm MAE are achieved by the full DVSS model with the full objective.Compared with L1-trained full DVSS, this improves PSNR by 0.47 dB and reduces MSE and MAE by 10.3% and 8.6%.Source: Table II · p. 8, C. Ablation Study · p. 8
  • 40.37 dB PSNR, 0.9985 SSIM, 91.75 mm2 MSE, and 2.78 mm MAE make full DVSS the best structure-ablation variant under L1 training.DWConv strengthens local spatial interactions, while SCR refines spatial and channel responses.Source: C. Ablation Study · p. 8
  • The full objective combines uncertainty-aware reconstruction, which emphasizes high-uncertainty low-signal regions, with gradient consistency for preserving local geometric discontinuities.The uncertainty-aware term alone improves PSNR from 40.37 dB to 40.66 dB versus L1 training.Source: C. Ablation Study · p. 8

D. Real-World Evaluation

On real prototype measurements, the proposed method outperforms the compared baselines quantitatively and preserves boundaries and planar structures in representative scenes. Evaluation uses a high-exposure multi-frame reference and a shared valid region.

D. Real-World Evaluation

  • 35.42 dB PSNR and 4.87 mm MAE are the best results on real captured data, surpassing ToF-KPN by 0.66 dB PSNR and 8.3% MAE.The method also improves on default demodulation by 3.74 dB PSNR and reduces MAE by 32.5%.Source: Table III · p. 8, D. Real-World Evaluation · p. 8
  • Real-data metrics average per-scene PSNR and MAE over common valid ROIs, using a high-exposure reference averaged from 100 frames.The reference is used only for evaluation, not network training; the test set contains 200 static scenes.Source: D. Real-World Evaluation · p. 8
  • Compared with baselines, the method more effectively suppresses depth noise while preserving object boundaries, planar structures, and geometric details.DeepToF leaves residual boundary artifacts, while RCF and ToF-KPN can blur local discontinuities and planar structures.Source: Figure 7 · p. 9, D. Real-World Evaluation · p. 9

E. Effect and Generality of the Proposed Noise Model

Controlled comparisons show that heteroscedastic synthesis improves restoration across noise baselines and backbones. The paper concludes that the complete framework performs best among evaluated methods, while identifying non-ideal sensing factors and embedded deployment as future work.

VI. CONCLUSION AND FUTURE WORK

  • The complete framework achieves the best synthetic and real depth accuracy among evaluated methods, and proposed synthesis consistently improves different restoration architectures.The full objective improves DVSS from 40.37 to 40.85 dB PSNR on synthetic data and from 35.12 to 35.42 dB on real measurements.Source: §VI · pp. 9–10, E. Effect and Generality of the… · p. 9

E. Effect and Generality of the Proposed Noise Model

  • 40.37 dB PSNR and 2.78 mm MAE on synthetic data, and 35.12 dB PSNR and 5.04 mm MAE on real measurements, are achieved with proposed synthesis under L1 training.Fixed Gaussian training yields 39.40 dB PSNR and 3.42 mm MAE on synthetic data, while range-aware Gaussian yields 39.89 dB PSNR and 3.08 mm MAE.Source: E. Effect and Generality of the… · p. 9, Table IV · p. 10
  • Proposed synthesis lowers synthetic MAE from 4.18 to 3.76 mm for U-Net, 3.58 to 3.22 mm for Restormer, and 3.08 to 2.78 mm for DVSS.On real measurements, MAE decreases from 6.34 to 5.84 mm, 5.91 to 5.46 mm, and 5.36 to 5.04 mm, respectively.Source: E. Effect and Generality of the… · p. 9, Table IV · p. 10

VI. CONCLUSION AND FUTURE WORK

  • Future work will extend the framework to multipath interference, motion artifacts, and thermal drift, and investigate deployment efficiency on embedded platforms.Source: §VI · p. 10
Loading 2609.05507v1…