PaperlayerView exact revision on arXiv

Source-linked AI summary

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, James Tompkin

arXiv:2008.06534v1cs.CV

TL;DR

Stereo 360° video lacks motion parallax and has disparity errors, while 6DoF view synthesis must also estimate unseen regions. MatryODShka learns a multi-sphere representation from stereo imagery; it reports real-time rendering and high quantitative performance, with quality limits at larger extrapolations.

  • Problem

    Stereo 360° video lacks motion parallax and has disparity errors, while view extrapolation requires estimating unseen regions.

    Source: §1 · p. 1

  • Method

    MatryODShka learns depth and fills disoccluded regions through a layered multi-sphere image representation inferred from stereo 360° imagery.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    80 Hz stereo VR rendering; test results cover baselines up to 5.6× ODS interpupillary distance and achieve the highest quantitative performance.

    Source: §5 · pp. 12, 14

  • Takeaways & Limitations

    The system’s fast inference matters when preprocessing is unavailable, including communications and robotic teleoperation.

    Source: §5 · p. 14

    Insufficient training data constrains the method, which works best on indoor scenes given the available training data.

    Source: §5 · p. 14

Abstract

from arXiv · show

We introduce a method to convert stereo 360{\deg} (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360{\deg} imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.

1 Introduction

Stereo 360° video provides binocular disparity but lacks motion parallax and has disparity errors, making prolonged viewing uncomfortable. MatryODShka learns a layered spherical representation to synthesize 6DoF views in real time, including unseen regions.

1 Introduction

  • Stereo 360° lacks motion parallax and has direction-dependent disparity problems, contributing to low viewing comfort and VR sickness.Side-by-side ERP disparity is incorrect except in one direction, while ODS disparity diminishes toward the zenith and nadir.Source: §1 · p. 1
  • Extrapolating views beyond a camera system’s baseline requires estimating unseen regions, and live 360° video calls for real-time synthesis.The target is 6DoF video with accurate motion parallax and disparity cues over a reasonably sized headbox.Source: §1 · pp. 1–2
  • MatryODShka learns to estimate depth and inpaint holes using a layered multi-sphere image representation from stereo 360° input.The network uses spherical-aware convolutions and losses and maintains temporal consistency without additional network parameters.Source: §1 · p. 2
  • The system is designed for real-time 4K video inference and rendering on modern GPUs, with an open-source training and rendering system.The system includes synthetic-data renderers, TensorFlow models, TensorFlow and TensorRT inference in Unity, and a multi-sphere video renderer.Source: §1 · p. 2

2 Related Work

Prior 360° video formats and view-synthesis methods offer depth or motion-parallax capabilities with limitations in viewpoint range, occlusions, or speed. MatryODShka extends layered perspective representations to spheres for omnidirectional 6DoF synthesis.

2 Related Work

  • ODS uses a circular projection with left- and right-eye panoramic views for disparity, while monocular 360° video provides no depth perception.ODS is popular because it fits existing processing, compression, and transmission pipelines.Source: §2 · pp. 2–3
  • Depth-map view synthesis is limited by reconstruction accuracy and cannot reveal content behind occlusions.Structure-from-motion and multi-view stereo assumptions often fail for stationary cameras or dynamic environments.Source: §2 · p. 3
  • Depth-augmented ODS, flow-based blending, and layered representations enable motion-parallax synthesis but not up/down motion; a prior 6DoF rig is not real-time, while another approach is offline.MatryODShka is presented as a fast learning-based method applicable to in-the-wild ODS videos or streams.Source: §2 · p. 3
  • Multi-plane images are semi-transparent appearance layers that can be reprojected into novel perspective views; MatryODShka extends this idea to spherical layers.Prior approaches optimize RGBA layers from stereo or four to five input images.Source: §2 · p. 3
  • Spherical image convolution methods can add inference expense or trade model capacity for spherical awareness.Full rotation-equivariance is not necessarily desirable because videos can exploit a fixed gravity vector.Source: §2 · p. 3
  • The illustrated pipeline forms left/right sphere sweep volumes from an ODS image, then predicts per-layer blending weights and alpha maps for a high-resolution MSI.The caption identifies the resulting MSI as supporting real-time 6DoF view synthesis.Source: Figure 2 · p. 4

3 Method

The method infers a multi-sphere image (MSI) from stereo omnidirectional imagery, then renders it from novel headset viewpoints. Its representation, inference, training losses, and rendering pipeline are designed for omnidirectional 6DoF view synthesis.

3 Method

  • The pipeline infers an MSI from paired ODS images and renders each video frame from novel left- and right-eye headset viewpoints.Training uses synthetic ODS image pairs.Source: §3 · p. 4

3.1 Multi-Sphere Image Representation

  • MSIs represent scenes as concentric spheres with RGBA values, enabling omnidirectional view synthesis from camera positions and orientations within the innermost sphere.The implementation uses 32 layers spanning radii from 1 m to 100 m, with more layers near the camera.Source: §3.1 · p. 5

3.2 Model Architecture

  • The network predicts per-layer alpha and left/right blending weights, using ODS reprojection to provide view colors and fill holes from at least one input view.The blending can also accommodate content without natural disparity, including specular highlights, reflections, and transparent content.Source: §3.2 · pp. 5–6

3.3 Training Losses

  • Training combines spherical-area-weighted re-rendering error with transform-inverse regularization to encourage temporal consistency without explicitly interpolating MSIs.The regularizer compares rendered views from transformed inputs with views from the original MSI at a transformed target pose.Source: §3.3 · pp. 6–7

3.4 High-resolution Rendering

  • High-resolution rendering combines the input ODS images using inferred blending weights and upsamples low-resolution alpha layers to match 4K×2K RGB images.Joint bilateral upsampling also reduces checkerboard artifacts and improves occlusion-edge quality.Source: §3.4 · p. 8

4 Experiments

The experiments evaluate MSI view synthesis using synthetic Replica supervision, image-quality metrics, baseline comparisons, ablations, and temporal-consistency tests. Results include stronger occluded-content synthesis than Serrano et al., a trade-off between temporal consistency and image quality, and sensitivity to baseline training data and rendering choices.

Training data

  • Replica scenes provide synthetic supervision through ERP target views rendered at interpolation and extrapolation poses, with 30,000 positions split by scene into training and test sets.Each position also supplies left- and right-eye ODS inputs; the target views are rendered at 640×320 after 4K×2K rendering, blur, and downsampling.Source: §4 · pp. 8–9

Baseline Comparisons

  • The double-plane-sweep model trained on RealEstate10K performs clearly better than the same model trained on Replica, despite the mismatch between perspective training and ODS testing.RealEstate10K is described as more varied and non-synthetic than Replica.Source: §4 · p. 10
  • ODS-Net fails to produce metrically accurate depth on Replica test data and performs worse than mesh-based rendering with ground-truth depth.The ground-truth-depth mesh baseline serves as an upper bound for depth-based re-rendering.Source: §4 · p. 10
  • Our method better synthesizes occluded content than Serrano et al., while their RGBD input and depth preprocessing yield more accurate disparity in textureless regions.Our method runs at 30 Hz, while Serrano et al. require one minute per frame; their layer computation assumes a static camera, whereas ours can support arbitrary camera motion with sufficient training data.Source: §4 · p. 10, Figure 3 · p. 11

Ablations

  • Adding a background RGB layer improves PSNR and E-LPIPS, but its inference-resolution limit can make it look blurry and explicitly blending it into disocclusions slows rendering.The authors characterize this as an application-level trade-off.Source: §4 · p. 11

Temporal consistency

  • Increasing transform-inverse regularization range improves frame-to-frame RGB consistency and more substantially improves depth consistency, while slightly degrading image metrics through increased blur.Depth consistency matters for consistent re-rendering and disparity cues across consecutive frames and different VR eye IPDs.Source: §4 · p. 11

5 Discussion and Conclusion

The system converts stereo 360° video into a real-time 6DoF MSI representation, with stereo VR rendering and strong quantitative performance at large synthesis baselines. Its range is constrained by computational trade-offs, depth discretization, and the available training data.

5 Discussion and Conclusion

  • Real-time ODS inference and 80 Hz stereo VR rendering support results at baselines up to 5.6× the ODS interpupillary distance, with the highest quantitative performance.The representation still provides correct disparity and motion parallax for objects with accurate alpha, while large disocclusions can reveal its layers.Source: §5 · pp. 12, 14
  • The approach can handle online in-the-wild ODS videos and live streams from off-the-shelf cameras, while adapting it to raw per-camera feeds is left for future work.Some multi-camera systems capture scene content omitted from the final ODS projection; using those feeds could contribute to reconstruction.Source: §5 · p. 14
  • Quality at larger extrapolations is limited by computational trade-offs, insufficient stereo 360° training data, and the method’s strongest results on indoor scenes with moving cameras.The authors also note that more work is needed for dynamic scene objects.Source: §5 · p. 14
  • MSI depth discretization limits the range of non-aliased views; beyond it, the layered representation becomes visible, an effect increased by raising only layer RGB resolution.The authors describe this rendering approach as potentially a better quality-and-comfort trade-off than retaining low-resolution imagery.Source: §5 · p. 14
  • The end-to-end system converts stereo 360° camera images into a 6DoF MSI representation in real time, distributing scene content across depths with temporal consistency.The authors report competitive image-quality metrics while maintaining fast inference for settings where preprocessing is unavailable.Source: §5 · p. 14

MatryODShka: Appendices

The appendices provide further comparisons and document implementation choices, from hardware and network details to rendering pseudocode.

MatryODShka: Appendices

A Additional Results and Comparisons

The appendix presents additional MSI decompositions and comparisons with the double-plane-sweep baseline. The reported novel views are better at larger synthesis baselines, while a figure illustrates a limitation of the layered representation.

A Additional Results and Comparisons

B Implementation Details

The implementation combines GPU inference and rendering with a filtering step to support real-time MSI viewing. The appendix also documents the network, training hyperparameters, and rendering pipeline inputs and outputs.

B Implementation Details

  • On the workstation, MSI inference runs at 30 Hz and novel-view rendering at 250+ Hz, or 125+ Hz in VR; the laptop reaches 30 Hz inference and 60+ Hz rendering, or 30+ Hz in VR.The workstation uses two NVIDIA GeForce 2080 Ti GPUs; the laptop’s rendering speed is stated as insufficient for fast VR head motion.Source: Hardware and Software · p. 23
  • Joint-bilateral upsampling removes some checkerboarding artifacts from learned transpose-convolution upsampling, which is approximately 2× faster than bilinear upsampling followed by learned convolution.The filtering uses spare compute on a dedicated rendering GPU and can be skipped on less powerful machines.Source: Joint-Bilateral Upsampling Effect · p. 23
  • The U-Net-style architecture takes left and right sphere-sweep volumes as input and concatenates the V coordinate to each convolutional layer’s input.The double-plane-sweep baseline uses the same architecture without the additional coordinate channels.Source: Table 3 · p. 24
  • The real-time Unity renderer takes high-resolution left and right ODS images plus the headset pose, and outputs a rendered perspective image.The appendix also provides dataset-generation and training hyperparameters and pseudocode for the rendering functions.Source: Network Architecture and Hyperparameters · p. 23, Table 4 · p. 24, Rendering Pseudocode · p. 25

Related papers

Loading 2008.06534v1…