PaperlayerView exact revision on arXiv

Source-linked AI summary

A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality

Ryan S. Overbeck, Daniel Erickson, Daniel Evangelakos, Matt Pharr, Paul Debevec

arXiv:1810.08860v1cs.GR

TL;DR

Consumer VR needs panoramic imagery that supports positional movement, but light-field capture and rendering must also meet demanding practical constraints. This paper builds an integrated acquisition, processing, compression, and rendering system that produces high-quality stereo views at 90Hz on commodity hardware, while retaining limitations in viewing volume and scene geometry.

  • Problem

    Panoramic VR requires light-field content that supports positional tracking while remaining practical to acquire, process, store, render artifact-free, and update at high frame rates.

  • Method

    The system combines novel spherical camera rigs, automated processing, disk-based per-view reconstruction, offline-to-real-time prefiltering, VP9-based compression, and progressive rendering.

  • Results

    The complete system reconstructs high-quality stereo views at 90Hz on commodity VR hardware, with light-field stills compressed to reasonable sizes and delivered through a public application.

  • Takeaways & Limitations

    The system provides a complete, downloadable pipeline for panoramic light-field stills aimed at consumer VR.

  • Takeaways & Limitations

    The viewing volume can feel constraining, and geometry mismatches or multilayer reflective surfaces can produce edge artifacts and ghosting.

Abstract

from arXiv · show

We present a system for acquiring, processing, and rendering panoramic light field still photography for display in Virtual Reality (VR). We acquire spherical light field datasets with two novel light field camera rigs designed for portable and efficient light field acquisition. We introduce a novel real-time light field reconstruction algorithm that uses a per-view geometry and a disk-based blending field. We also demonstrate how to use a light field prefiltering operation to project from a high-quality offline reconstruction model into our real-time model while suppressing artifacts. We introduce a practical approach for compressing light fields by modifying the VP9 video codec to provide high quality compression with real-time, random access decompression. We combine these components into a complete light field system offering convenient acquisition, compact file size, and high-quality rendering while generating stereo views at 90Hz on commodity VR hardware. Using our system, we built a freely available light field experience application called Welcome to Light Fields featuring a library of panoramic light field stills for consumer VR which has been downloaded over 15,000 times.

1 INTRODUCTION

Panoramic light fields are well suited to immersive VR because positional tracking lets users move within photorealistic scenes, but practical adoption is limited by acquisition, storage, quality, and speed challenges. The paper presents an integrated system addressing these constraints for real-time consumer VR.

  • Positional tracking lets panoramic light fields provide motion parallax, view-dependent reflections, and more comfortable, immersive VR experiences.
  • Light fields have seen limited commercial adoption because they require compelling use cases, portable acquisition of thousands of viewpoints, compact storage, artifact-free rendering, and high frame rates.
  • 90Hz rendering is required by current desktop VR platforms to maximize user comfort.
  • The system combines portable camera rigs, cloud processing, prefiltering, VP9-based compression, and real-time reconstruction to address these technical challenges.
  • 50-200MB datasets with few noticeable compression artifacts can be downloaded in seconds, while reconstructed views run at 90Hz on entry-level VR-enabled machines.

2 RELATED WORK

Prior work spans light-field compression, panoramic VR, image-based rendering, acquisition, and geometry, but the paper targets a complete system supporting downloadable panoramic light-field experiences on entry-level VR hardware.

  • The system extends prior panoramic-light-field efforts by documenting acquisition, processing, compression, and rendering for downloadable experiences on entry-level consumer VR hardware.
  • Simpler image-based representations reduce data dimensionality but restrict viewer motion, whereas panoramic light fields support full 6 degrees of freedom.
  • The work uses a disk-based reconstruction basis with per-view geometry for high-quality rendering of loosely unstructured light fields.
  • Existing learned view-interpolation approaches were limited to offline usage, restricting their immediate relevance to real-time applications.
  • The paper presents mechanical gantry rigs for outward-looking spherical light fields, addressing limitations of lenslet arrays and conventional gantries.
  • Previous light-field compression methods commonly require decoding the full light field before rendering, conflicting with immersive VR’s need for fast access.
  • The paper implements a prior compression strategy in VP9 and demonstrates it within a high-quality light-field rendering algorithm.

3 ACQUISITION

The system uses two portable rigs to capture spherical light fields: a fast 16-camera GoPro rig and a higher-quality two-camera DSLR rig. Their capture volumes support seated VR viewing and are produced in minutes rather than hours.

  • Two camera rigs record rays incident upon a spherical light-field volume.
  • The 16×GoPro rig prioritizes speed, rotating 16 cameras around a full circle in 30, 60, or 120 seconds.
  • The GoPro rig produces 16 rows of 72 pictures, or approximately 6 Gigapixels, from evenly spaced video frames.
  • The 2×DSLR rig prioritizes image quality with two Sony a6500 cameras and 8mm fisheye lenses mounted on an 80cm pivoting platform.
  • HDR acquisition yields 7,776 photos in 33 minutes, while continuous operation yields 2,592 single-shot photos in nine minutes.
  • The rigs provide light-field viewing volumes of 60cm for the GoPro system and about 70cm for the mirrorless-camera system.

4 LIGHT FIELD RECONSTRUCTION WITH DISK-BASED BLENDING FIELD AND PER-VIEW GEOMETRY

The system reconstructs novel light-field views by projecting each image onto per-view geometry and blending the results with a disk-based basis. This representation supports efficient GPU rendering, tile streaming, and higher-quality reconstruction than a single global geometry proxy.

  • Rendering with Disks: Each light-field image is projected onto its own mesh, masked and weighted by a disk, then blended into the novel view.The mesh is derived from a multi-view stereo depth map, while disk size controls overlap between neighboring views.
  • Rendering with Disks: The disk-based basis answers ray queries by sampling each image at the ray–mesh intersection and weighting samples at the ray–disk intersection.The disk weight uses a reconstruction kernel that peaks at the disk center and falls to zero at its edge.
  • GPU Implementation: The reconstruction is implemented in two GPU passes: accumulate weighted samples and weights, then divide the accumulated color by the accumulated weight.The first pass uses texture-mapped meshes, shader-computed disk weights, and additive blending in an RGBA framebuffer.
  • Tile Streaming and Caching: Tile streaming and caching avoid loading the entire light field into GPU memory, which would otherwise be prohibitively expensive at typical resolutions.The naïve implementation renders every textured mesh for every image each frame.
  • Comparison to Camera-Mesh Based Blending: The disk representation places disk parameters in shader inputs and reserves the GPU triangle stream for scene geometry, enabling efficient per-view geometry rendering.This design provides higher reconstruction quality than a single global geometry proxy and maps each rasterized triangle to exactly one tile for caching.
  • Mesh Generation: At δp = 5, meshes average 10 triangles per tile and produce sharp results; increasing δp further does not noticeably improve reconstruction.At δp = 1, meshes have 2 triangles per tile and show visible ghosting artifacts.

5 PREFILTERING TO IMPROVE LIGHT FIELD RECONSTRUCTION

The disk-based reconstruction can produce depth-edge artifacts when geometry proxies are inaccurate. A synthetic prefilter projects from a higher-quality offline model into the real-time representation to reduce these artifacts, while geometry precision and reconstruction kernels control sharpness and blur.

  • Artifact reduction: Geometry inaccuracies can create depth-edge artifacts that flicker as the viewer moves.The disk-based reconstruction works well with accurate geometry but is vulnerable where the proxy deviates from the scene.
  • Prefilter design: Synthetic prefiltering applies a large-aperture model after acquisition and before encoding to suppress artifacts from the simplified real-time model.The offline process treats the original light field as a synthetic scene and uses sufficiently large apertures to band-limit the signal.
  • Geometry precision: At δp = 5, reconstructions are sharp with about 10 triangles per tile, while δp = 1 uses 2 triangles per tile and shows ghosting.Increasing precision beyond δp = 5 does not noticeably improve reconstruction, so the system uses δp = 5 by default.
  • Prefilter design: The prefilter computes a Monte Carlo approximation of a 4D integral over each light-field pixel and synthetic aperture, using reconstruction disks as the aperture.The integral combines the disk-shaped kernel, pixel reconstruction basis, and high-quality offline light-field ray queries.
  • Prefilter behavior: The Lanczos-2 pixel basis avoids both ringing and over-blurring, whereas a box filter works but introduces additional blur.The basis function σ(s,t) determines the pixel reconstruction behavior in the prefilter.
  • Prefilter behavior: When real-time and offline geometries agree, rays converge and produce sharp results; disagreement spreads rays across different image regions and blurs them.This behavior is illustrated by the close-geometry and differing-geometry cases in Figure 8.

6 LIGHT FIELD COMPRESSION USING VP9

The system adapts motion-compensated prediction to light-field tile streaming, avoiding the long reference chains of conventional video compression. A modified VP9 codec uses sparse references and independent tile access to preserve compression while enabling fast random decoding.

  • Motivation: About 1,000–1,500 1280×1024 images require 4–6 GB without compression, motivating exploitation of light-field image coherence.The large dataset size would be inconveniently slow to download over a standard 10–20 Mbit/s connection.
  • Compression challenge: Standard video motion-compensated prediction prevents fast random access to arbitrary light-field tiles.The dependency on earlier translated pixel blocks makes conventional video techniques problematic for tile streaming.
  • Reference structure: A conventional row-based GOP creates long reference chains, requiring intermediate images to be decoded before accessing a tile in a later image.This is inefficient for rendering algorithms that need immediate access to individual tiles.
  • VP9 implementation: The VP9 implementation adds arbitrary reference-image selection and a compact tile lookup table for random access.The two-level lookup table adds about 1 byte per tile and contributes negligible decoding overhead.
  • Reference structure: Sparse reference images ensure every other image depends on its nearest reference, limiting reference chains to length one.Reference images are loaded into memory so tiles from other images can be decoded immediately.
  • Reference structure: Hierarchical clustering selects center images as references, with 25 images per cluster providing good compression for most light fields.The approach supports irregular panoramic datasets by clustering 3D image locations using a bounding volume hierarchy.

7 LIGHT FIELD PROCESSING AND RENDERING SYSTEM DETAILS

The processing and rendering system combines automated cloud processing with tiled, cached GPU rendering and progressive refinement. Its pipeline regularizes and enhances acquired light fields before compression, while runtime mechanisms preserve responsive rendering under variable tile workloads.

  • Processing pipeline: The automated processing pipeline has eight stages: calibration, synthesis, subsampling, equiangular conversion, geometry generation, prefilter, color enhancement, and compression.The pipeline is designed to scale across many light fields.
  • Processing pipeline: Calibration estimates camera intrinsics and extrinsics for every captured light field because these parameters vary between captures.The calibration step uses approximate extrinsic and intrinsic priors as inputs.
  • Processing pipeline: Synthesis fills the capture sphere’s top and bottom blind spots by interpolating new views from other light-field images.The bottom pole is obstructed by the tripod, while the top may lack images when the rig is not wound fully upward.
  • Processing pipeline: Subsampling removes excessive sampling near the sphere’s poles, reducing data size and rendering cost while making quality more uniform.The densest image set remains available as input to prefiltering before outputting the subsampled set.
  • Processing pipeline: Locally linear equiangular conversion warps tile corners while retaining rectilinear mapping within tiles, providing most EAC benefits without significant rendering cost.The conversion supports wide-field-of-view cameras without sending excessively large images to the renderer.
  • Runtime rendering: The renderer culls visible tiles, decodes uncached tiles, streams them to GPU caches, and performs disk-based reconstruction on the GPU.Multiple CPU decoder threads handle ordinary tile loads, which are generally fewer than 50 per frame during normal head motion.
  • Runtime rendering: Two-level progressive rendering uses coarse and fine disk hierarchies so a complete coarse light field remains available during tile-decoding spikes.The coarse level is small enough to preload into GPU RAM, allowing incremental refinement with fine tiles.
  • Runtime rendering: A valid-view-volume fade hides distracting image edges as the viewer approaches the boundary of the dataset’s supported viewing region.The fade is applied instead of exposing the light field outside its limited viewing volume.

8 RESULTS

The evaluation measures rendering quality, compression, rendering speed, and practical system limitations. The proposed system reconstructs high-quality views from sparse imagery, compresses light fields substantially, and renders within the 90Hz frame budget, while retaining artifacts and operational constraints.

  • Evaluation scope: The evaluation covers rendering quality, file size, rendering speed, and limitations of the proposed system.These correspond to technical challenges (3)–(5) identified by the paper.
  • Render quality: With 466 light field images, the proposed method achieves high-quality reconstruction, whereas the planar-proxy baseline exhibits significant ghosting.The planar proxy requires 29,722 images, approximately 64× more, to approach the proposed reconstruction quality.
  • Render quality: The per-view geometry approach is more tolerant of calibration and geometry errors than the global-geometry baseline, especially near edges.The global-geometry implementation requires near-perfect calibration, geometry generation, and registration for high reconstruction quality.
  • Render quality: Per-view geometries handle reflective and other non-diffuse surfaces reasonably well because they need only approximate nearby views.The global-geometry baseline struggles because photogrammetry cannot construct a model consistent across all views for such surfaces.
  • Artifact reduction: Prefiltering replaces many flickering ghosting artifacts with smooth blur, preserving sharp imagery where per-view geometry is accurate.This reduces the visible impact of rendering artifacts while retaining detail in well-modeled regions.
  • Compression: At approximately 45 dB PSNR, most evaluated light fields achieve approximately 40×–200× compression, while the difficult Discovery Flight Deck reaches approximately 29×.Compared with standard image compression, the method achieves approximately 1.5×–2× better compression on most scenes.
  • Compression: At 45 dB PSNR, the dense planar Bulldozer light field reaches 178× compression, increasing to 1,000× at 38 dB.Its dense planar structure yields greater compression improvement from the modified codec.
  • Compression: The disk-based reconstruction and per-view geometry reduce the image density needed for rendering but also reduce compression coherence.The paper identifies this as a trade-off between rendering sparsity and compression performance.

9 CONCLUSION AND FUTURE WORK

The paper presents an end-to-end panoramic light field system for VR that combines portable acquisition, reconstruction, prefiltering, compression, and 90 Hz rendering. It also demonstrates public distribution through Welcome to Light Fields while identifying light field video as an important future challenge.

  • The complete system acquires, processes, compresses, and renders panoramic light field stills for VR, reconstructing stereo views at 90 Hz on commodity hardware.The renderer decodes images on demand while maintaining the frame rate required for comfortable VR.
  • Welcome to Light Fields provides a freely downloadable library of panoramic light field stills and has been downloaded over 15,000 times.
  • Future work: Portable light field still acquisition is practical, but building a portable and easy-to-use light field video rig may require as many as 100 cameras.
  • Future work: Light field video also poses compression and decoding challenges, particularly maintaining 90 Hz rendering speed.The authors identify hardware decoding and reducing the number of rendered tiles as possible directions.
Loading 1810.08860v1…