PaperlayerView exact revision on arXiv

Source-linked AI summary

TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis

Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, Matthew O'Toole

arXiv:2109.15271v2cs.CV

TL;DR

Monocular dynamic scene reconstruction is under-constrained, so existing methods rely on data-driven priors. TöRF replaces those priors with a continuous-wave ToF neural representation that models raw sensor measurements alongside color, improving dynamic view synthesis and robustness to ToF ambiguities.

  • Problem

    Monocular dynamic NVS is under-constrained and therefore relies on data-driven priors to reconstruct dynamic content.

  • Method

    TöRF models raw continuous-wave ToF measurements with a physically based neural radiance field optimized jointly with color images.

  • Results

    The method improves view synthesis for few-view and especially dynamic scenes, with better results than NeRF [32] and two dynamic-scene baselines [26] [52].

  • Takeaways & Limitations

    Raw ToF phasor supervision helps RGB and ToF sensing resolve reconstruction errors, limitations, and ambiguities more effectively than derived depth alone.

  • Takeaways & Limitations

    C-ToF sensing can struggle on large-scale scenes, outdoors, and with fast-moving objects because measurements require multiple rapidly captured images.

Abstract

from arXiv · show

Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g., NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known to be an under-constrained problem, and so methods rely on data-driven priors for reconstructing dynamic content. We replace these priors with measurements from a time-of-flight (ToF) camera, and introduce a neural representation based on an image formation model for continuous-wave ToF cameras. Instead of working with processed depth maps, we model the raw ToF sensor measurements to improve reconstruction quality and avoid issues with low reflectance regions, multi-path interference, and a sensor's limited unambiguous depth range. We show that this approach improves robustness of dynamic scene reconstruction to erroneous calibration and large motions, and discuss the benefits and limitations of integrating RGB+ToF sensors that are now available on modern smartphones.

1 Introduction

Novel-view synthesis from monocular dynamic video is under-constrained and often requires learned priors for depth and motion. TöRF instead combines color with raw continuous-wave ToF measurements in a neural radiance-field representation, improving view synthesis while handling ToF-specific ambiguities.

  • Monocular dynamic NVS is ill-posed, producing visual artifacts and motivating learned priors for scene depth and motion.
  • TöRF represents scene appearance with an implicit neural model that leverages both color and ToF images.
  • Raw ToF phasor supervision handles limited unambiguous range, low reflectance, and multi-path interference better than directly using derived depth.
  • The method uses a physically based raw continuous-wave ToF rendering model and evaluates dynamic scenes against NeRF [32] and two dynamic-scene baselines [26] [52].

2 Related Work

Neural scene representations span structured grids, image-based representations, point clouds, and implicit functions. Although depth improves NVS, explicitly reconstructed depth can fail around thin structures, edges, complex reflectance, and noise, motivating raw ToF modeling.

  • Neural scene representations combine learnable components with physics-based models and classical rendering processes for NVS.
  • Neural scene representations include voxel grids, multiplane and multi-sphere images, point clouds, and implicit functions of geometry and appearance.
  • Implicit neural representations avoid voxel grids’ cubic memory growth while flexibly encoding scene geometry and appearance.
  • Raw ToF modeling addresses depth-map problems involving thin structures, depth edges, complex reflectance, and noisy measurements.

3 Neural Volume Rendering of ToF images

The method extends neural volume rendering to continuous-wave ToF cameras by modeling collocated active illumination and raw complex phasor measurements. This formulation accounts for multi-point light contributions and avoids limitations of depth-derived supervision.

  • 3 Neural Volume Rendering of ToF images: The neural field predicts density and radiance, while differentiable ray marching integrates scene contributions along each camera ray.Density controls opacity and radiance represents direction-dependent appearance; the same framework is extended to learn from raw ToF images.
  • 3.1. Collocated Point Light Source.: The ToF rendering model modifies standard volume rendering for collocated active illumination and uses squared transmittance with inverse-square light falloff.The radiance depends on the camera-positioned point source, and the resulting integral can be approximated with quadrature and optimized using a neural network.
  • 3.2. Continuous-Wave ToF Model.: Continuous-wave ToF rendering weights each light path with a complex phasor, producing complex-valued images whose phase encodes distance-related information.The phasor is formed from the path-length weighting function, and practical phasor images combine four real-valued captures.
  • 3.2. Continuous-Wave ToF Model.: Raw phasor supervision models multiple points along a ray, whereas conventional depth recovery assumes a single reflecting point and can suffer multi-path interference.Multiple surfaces can produce mixed phasors and flying-pixel artifacts near depth edges, including for semi-transparent or specular objects.
  • 3.2. Continuous-Wave ToF Model.: Direct phasor optimization avoids depth ambiguity beyond c/(2f), improves robustness when reflected light is weak, and handles mixed responses near depth edges or complex reflectance.For f = 30 MHz, the typical unambiguous range is approximately 5 m; modeling the raw response avoids forcing these cases into a single depth value.

4 Optimizing Dynamic ToF + NeRF = TöRF

TöRF models dynamic scenes with separate static and dynamic neural fields, using raw phasor measurements alongside color rather than derived depth. Its joint optimization and raw-phasor supervision address wrap-around and dark-object noise while accommodating camera-pose uncertainty.

  • 4.1. Dynamic Neural Radiance Fields.: TöRF represents static appearance with a position-and-direction network and dynamic appearance with a position-, direction-, and time-dependent network using per-frame latent codes.
  • 4.1. Dynamic Neural Radiance Fields.: Raw phasor supervision avoids wrap-around errors because only one phase offset remains consistent across multiple camera views.
  • 4.1. Dynamic Neural Radiance Fields.: For dark objects, TöRF recovers scene geometry more accurately than ToF-derived depth by using triangulation when phasor magnitudes are small.
  • 4.2. Optimization.: Training samples color and phasor-camera rays and minimizes a weighted sum of RGB and ToF reconstruction errors, with the ToF weight reduced later to prioritize color.
  • 4.3. Camera Pose Optimization.: For real scenes, TöRF optimizes camera poses and the relative color–ToF calibration within training because COLMAP may fail and does not recover metric scale.
  • 4.2. Optimization.: The model traces separate rays for non-collocated color and ToF cameras and alternates color and ToF losses during optimization.

5 Experiments

Experiments evaluate TöRF on synthetic and real static and dynamic scenes using RGB and raw ToF measurements. Results show stronger few-view static reconstruction, improved dynamic reconstruction against depth-prior and ToF-depth baselines, and the highest reported real-scene visual quality.

  • 5.1. Datasets and Setup.: The real-scene benchmark uses RGB and phasor cameras with a 41 mm baseline, while the iPhone 12 Pro comparison provides depth but not raw measurements.
  • 5.1. Datasets and Setup.: Training takes 12–24 hours to converge, and rendering a 256×256 novel view takes 3–5 seconds per frame on an RTX 2080 Ti.
  • 5.2. Few-View Reconstruction of Static Scenes.: With two input views, TöRF reconstructs synthetic static scenes better than NeRF, while four-view RGB results are comparable and TöRF depth remains significantly more accurate.
  • 5.3. Few-View Reconstruction of Static Scenes.: TöRF produces better depth and RGB views than NSFF, VideoNeRF, and their ToF-depth variants on the synthetic dynamic DinoPear sequence, while NSFF has slightly higher PSNR but worse LPIPS.
  • 5.4. Dynamic Scenes.: On real-world scenes, TöRF achieves the highest visual quality and most accurate depth maps, while competing methods show ghosting, stretching, disocclusion, or wrapped-range failures.

6 Discussion

The evaluations show that ToF supervision improves few-view and dynamic reconstruction, while ToF sensing and neural rendering impose practical limitations. Remaining errors concentrate in difficult reflectance, motion, and viewing configurations, alongside computational and application constraints.

  • Limitations: C-ToF sensing is constrained by large scenes, outdoor illumination, and rapid motion because measurements require four or more quickly captured images.Multiple modulation frequencies can extend unambiguous range, and coding methods can improve depth precision.
  • Results: TöRF significantly outperforms NeRF on synthetic static scenes, particularly from two input views, while producing more accurate depth estimates.Metrics use 10 hold-out views; mirror reflections affect the bathroom depth-error measure.
  • Results: Compared with VideoNeRF+ToF and NSFF, TöRF produces higher visual quality and more accurate depth, reducing errors in static and some dynamic components.VideoNeRF+ToF shows disocclusion and warped-edge artifacts, while NSFF exhibits ghosting and stretching.
  • Results: NSFF achieves the highest PSNR on DinoPear, but this does not match perceived visual quality; TöRF instead produces better depth reconstructions from raw phasor images.The baselines receive manually unwrapped ToF depth, whereas TöRF models raw phasor images.
  • Limitations: Even with ToF data, dark dynamic objects and grazing-angle views remain difficult, while neural networks limit the duration of modeled dynamic sequences.Dark hair is an example of motion-related reconstruction failure.
  • Potential Social Impact: Integrating ToF improves reconstruction accuracy but restricts use to scenarios where active illumination is detectable, with possible misuse for falsified media, surveillance, or copying designs.These concerns are stated as potential negative impacts of higher-fidelity reconstruction.
  • Practical constraints: Neural reconstruction is more computationally expensive than classic image-based rendering, requiring 12–24 hours of GPU optimization and producing reported CO2 emissions.The reported estimates are 1.5–3 kg CO2-equivalents per scene for optimization and 0.01–0.02 kg per sequence for rendering.

7 Conclusion

The conclusion presents TöRF as a physically based RGB+ToF neural radiance-field model that improves few-view and dynamic novel-view synthesis. It emphasizes raw phasor supervision as more effective than derived depth and identifies additional sensing modalities as future directions.

  • Conclusion: TöRF combines RGB and ToF image formation in a neural model and improves novel-view synthesis for few-view static scenes and especially dynamic scenes.The method recovers TöRF volumes through an optimization procedure.
  • Conclusion: Raw ToF phasor supervision performs better than derived depth by helping resolve sensing errors, limitations, and ambiguities.The conclusion frames this as a benefit of using both sensing modes together.

8 Additional Results

Additional experiments include real smartphone capture and further real-scene visualizations. The smartphone sequence uses processed depth converted to synthetic C-ToF data because raw measurements are unavailable.

  • Additional Results: Additional results cover StudyBook, Dishwasher, and DinoPear, including animated comparisons and an explicit demonstration of handling multi-path interference.The reported visualizations appear in Figures 8–10 and accompanying website videos.
  • Smartphone Capture: The Dishwasher sequence was captured with an iPhone 12 Pro, whose ARKit pipeline supplies aligned RGB video and dense metric depth maps but not raw ToF measurements.The consumer setup provides a practical camera comparison while limiting direct raw-measurement evaluation.
  • Smartphone Capture: For processing the iPhone sequence, the estimated metric depth maps are converted to synthetic C-ToF sequences by assuming constant infrared albedo.The RGB and ToF data are collocated because the depth maps are aligned with the color video.

9 Dynamic Field Blending

Dynamic-field blending combines static and dynamic radiance-field predictions through a time- and position-dependent opacity blend. The resulting blended radiance and intensity are then used in RGB and ToF rendering.

  • Rendering: Quadrature evaluates the dynamic-scene rendering integral, with the ToF case paralleling RGB rendering using collocated reflected radiant intensity.The same field-blending framework supports both sensing modalities.
  • Dynamic Field Blending: The blended opacity combines static and dynamic network opacities using the blending weight b(x_k, τ).The blend is α_blend(x_k, τ) = (1 − b(x_k, τ)) · α_stat(x_k) + b(x_k, τ) · α_dyn(x_k, τ).
  • Rendering: The blended radiance is premultiplied by blended opacity, while static and dynamic networks provide the scattered radiance components.The corresponding radiant intensity is computed for use in the ToF rendering equation.

10 Continuous-wave Time-of-Flight Image Formation Model

The paper models continuous-wave ToF measurements through convolution-based image formation, producing complex phasors whose magnitude reflects returned light and phase relates to distance. It then describes RGB+ToF acquisition, calibration, phasor construction, and comparisons showing improved reconstruction quality over video baselines.

  • 10 Continuous-wave Time-of-Flight Image Formation Model: The C-ToF measurement convolves the scene’s temporal response with the light-source modulation and integrates the result over repeated exposure periods.The exposure and source functions are periodic, with N periods occurring during one exposure.
  • 10 Continuous-wave Time-of-Flight Image Formation Model: The convolution of exposure and source modulation defines h(t), a path-length importance function weighting each light path by its path length.This formulation determines how scene responses contribute to the sensor measurement.
  • 10 Continuous-wave Time-of-Flight Image Formation Model: The assumed phasor camera uses sinusoidal modulation and four phase offsets to linearly reconstruct a complex response at every pixel.The phase offsets are φ ∈ {0, π/2, π, 3π/2}.
  • 10 Continuous-wave Time-of-Flight Image Formation Model: The resulting phasor magnitude represents reflected-light amount, while its phase is related to the distance of the observed scene point.Real and imaginary phasor components are visualized separately in the experimental setup.
  • 11 Experimental C-ToF Setup: The experimental system combines a high-resolution color camera with a lower-resolution ToF camera, both operating at 30 frames per second.The color sensor has 2056×1542 pixels and the ToF sensor has 320×240 pixels; color images are resized to 640×480 for optimization.
  • 11 Experimental C-ToF Setup: Calibration undistorts and aligns the stereo cameras while estimating a common zero-phase offset from a calibration target.The setup calibrates intrinsics, extrinsics, and distortion coefficients before optimization.
  • 11 Experimental C-ToF Setup: TöRF reduces reconstruction errors in static and some dynamic components compared with VideoNeRF and NSFF baselines using ToF-derived depth.The compared models were trained for comparable times; the figure also presents TöRF against modified baselines receiving unwrapped depth images.
  • 11 Experimental C-ToF Setup: Raw phasor supervision addresses multi-path interference that makes C-ToF depth inaccurate near specular reflections such as a metallic fridge door.The measured phasors mix contributions from the door surface and reflected virtual objects, biasing depth values.
Loading 2109.15271v2…