PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Field Video Capture Using a Learning-Based Hybrid Imaging System

Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, Ravi Ramamoorthi

arXiv:1705.02997v1cs.CVcs.GR

TL;DR

Consumer light field cameras capture at low frame rates, and interpolation can fail when motion is missing between samples. This paper combines sparse light field video with standard 30 fps video in a learning-based synthesis system. It reports generating 30 fps light field video with consumer cameras, enabling applications such as refocusing and parallax-view generation during playback.

  • Problem

    Low-frame-rate light field capture makes direct interpolation unreliable when motion is entirely missing between adjacent frames.

    Source: §1 · p. 2

  • Method

    A hybrid system combines sparse light field video and 30 fps 2D video, using learned flow estimation to warp images and appearance estimation to combine them for target views.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 6

  • Results

    The paper reports an algorithm for generating 30 fps light field video with consumer cameras, which current video interpolation and depth-from-video methods cannot achieve.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The resulting video supports digital refocusing and parallax-view generation while it plays.

    Source: §1 · p. 2

    Artifacts remain, mostly around occlusions, and are usually caused by improper flow estimation.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Light field cameras have many advantages over traditional cameras, as they allow the user to change various camera settings after capture. However, capturing light fields requires a huge bandwidth to record the data: a modern light field camera can only take three images per second. This prevents current consumer light field cameras from capturing light field videos. Temporal interpolation at such extreme scale (10x, from 3 fps to 30 fps) is infeasible as too much information will be entirely missing between adjacent frames. Instead, we develop a hybrid imaging system, adding another standard video camera to capture the temporal information. Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps. We adopt a learning-based approach, which can be decomposed into two steps: spatio-temporal flow estimation and appearance estimation. The flow estimation propagates the angular information from the light field sequence to the 2D video, so we can warp input images to the target view. The appearance estimation then combines these warped images to output the final pixels. The whole process is trained end-to-end using convolutional neural networks. Experimental results demonstrate that our algorithm outperforms current video interpolation methods, enabling consumer light field videography, and making applications such as refocusing and parallax view generation achievable on videos for the first time.

1 INTRODUCTION

Light field cameras enable post-capture changes such as refocusing, but their bandwidth limits consumer capture to about 3 fps, where interpolation can fail because motion is missing between frames. The paper combines sparse light field capture with standard video to generate 30 fps light field video using a learning-based synthesis pipeline.

1 INTRODUCTION

  • At 3 fps, light field capture is limited by bandwidth, and interpolation can be impossible when motion is entirely missing between neighboring frames.The Lytro ILLUM's raw output is nearly 20 times 1080p resolution under the stated comparison, limiting capture to 3 fps.Source: §1 · pp. 1–2
  • The hybrid system combines a low-frame-rate light field video with standard 30 fps 2D video to produce a full light field video at the standard video rate.The light field stream captures angular information; the 2D stream captures temporal information.Source: §1 · p. 2
  • The generated video supports post-capture refocusing and parallax-view generation as it plays.The paper also describes an interactive interface for focusing on points and tracking objects through video.Source: §1 · p. 2
  • For each target view at a missing light field frame, the system estimates spatio-temporal flows to warp images, then estimates appearance to combine them into final pixels.The pipeline propagates angular information from the light field frames to the 2D-only frames.Source: §1 · p. 2
  • The proposed algorithm generates 30 fps light field video using consumer cameras, a capability current video interpolation and depth-from-video methods cannot achieve.The paper presents this as its first such algorithm.Source: §1 · p. 2
  • The authors train disparity and optical-flow CNNs without ground-truth labels and cascade them to combine angular and temporal information.The introduction identifies these networks as components of its CNN architecture.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses light field video capture, frame interpolation, view synthesis, hybrid imaging, and 2D-to-3D conversion, but these approaches leave gaps for consumer light field video at increased temporal resolution. The paper positions its method as temporal synthesis using both sparse light field and 2D video information.

2 RELATED WORK

  • Existing light field video devices either have lower spatial or angular resolution or are expensive and aimed mainly at research, while the consumer-friendly Lytro ILLUM cannot capture light field video.The paper combines the ILLUM with an additional 2D camera.Source: §2 · p. 2
  • Most video interpolation methods target slow motion from standard 30 fps input, while 3 fps input may omit motion and separate view interpolation may not yield a consistent light field video.The paper contrasts its approach with these limitations in Section 5.Source: §2 · p. 2
  • Light field super-resolution methods reconstruct spatial or angular detail, but none of the cited approaches is designed for temporal upsampling.The passage distinguishes temporal upsampling from novel-view synthesis methods.Source: §2 · p. 3
  • Earlier hybrid camera systems target spatial-resolution increase, deblurring, or hyperspectral imaging rather than increased temporal resolution.Source: §2 · p. 3
  • The paper adapts the depth-warp-and-combine pattern of view synthesis to temporal image synthesis rather than generating only new viewpoints.It uses learning-based geometry and appearance estimation.Source: §2 · p. 3
  • Prior 2D-to-3D methods generate depth sequences, whereas this method produces a light field sequence.One cited approach also requires its entire training dataset at runtime, with substantial memory and time costs.Source: §2 · p. 3

3 ALGORITHM

The system generates light-field frames between sparse keyframes by propagating angular information through standard 2D video. Spatio-temporal flow CNNs warp input views to each target, and an appearance CNN combines them into the output image.

3 ALGORITHM

  • The system fills frames between sparse light-field keyframes by propagating angular information through 2D video and synthesizing target views with flow and appearance CNNs.Source: §3 · p. 3
  • The setup assumes aligned camera viewpoints for explanation; in practice, the system estimates an additional flow for viewpoint calibration and downsamples DSLR images to Lytro resolution.Spatial super-resolution is outside the paper's focus.Source: §3 · p. 3

3.1 Spatio-temporal flow estimation network

  • A disparity CNN predicts central-view disparity from shifted light-field views, using their means and variances as features.The approach trains disparity estimation through view reconstruction rather than requiring ground-truth depth maps.Source: §3.1 · pp. 3–4
  • For view synthesis, an imperfect depth map can produce a more visually pleasing result than a perfect one near occlusion boundaries.The learning-based approach does not explicitly model occlusions, but the network can still learn to handle them.Source: §3.1 · pp. 4–5, Figure 6 · p. 5
  • Temporal flow estimation computes optical flow between keyframes and intermediate 2D frames, cascading neighboring-frame flows when needed.Because temporal displacement can be large, the network estimates flow hierarchically from coarse to fine.Source: §3.1 · p. 5
  • The predicted flows warp five inputs—the current 2D frame, two keyframe 2D frames, and two keyframe target views—to the missing target view.The current 2D frame often contributes most, but keyframe views can matter more in static regions.Source: §3.1 · p. 6
  • A CNN combines disparities borrowed from the two keyframes and temporal position to predict target-view disparity, since depth changes make direct combination difficult.Source: §3.1 · p. 6

3.2 Appearance estimation network

  • The appearance CNN combines the five warped images with estimated disparities, flows, and position cues to generate the target image.These cues help the network handle occlusion boundaries and weight images that differ in both angular and temporal domains.Source: §3.2 · p. 6

4 IMPLEMENTATION

Implementation details the flow architecture and its image-reconstruction objective, training and evaluation procedures, and a calibrated, synchronized dual-camera prototype.

4.1 Network architecture

  • The temporal-flow CNN uses a hierarchical image pyramid, while image-reconstruction training yields fewer warping artifacts around occlusions despite less accurate flow.FlowNet performs better at flow estimation, but the paper reports fewer artifacts from its own flow when warping images.Source: §4.1 · pp. 6–7, Figure 8 · p. 7

4.2 Training and Evaluation

  • Training uses about 120 slow-motion scenes; components are trained stage-wise before joint optimization of the flow and appearance networks.Each clip has about 10 frames, patch extraction yields over 10 million training patches, and the authors state that code, model, and datasets will be released upon publication.Source: §4.2 · p. 7
  • Quantitative evaluation compares reconstruction error on 30 test scenes captured using the same slow-camera-motion and object-motion approach as the training data.The passage identifies the comparison with prior approaches but gives no error values.Source: §4.2 · p. 7

4.3 Testing in real-world scenarios

  • The real-world prototype pairs a 30 fps DSLR video with 3 fps Lytro light fields, calibrates viewpoint and color using NRDC, and synchronizes frames using recorded shutter sounds.The DSLR is downsampled to Lytro resolution, and all 64 keyframe views are used to compute disparity maps.Source: §4.3 · pp. 7–8

5 RESULTS

The system outperforms video interpolation and depth-from-video methods in quantitative and qualitative comparisons, and demonstrates light-field video applications. Its limitations include occlusion artifacts, restricted working distances, motion-blur sensitivity, and unused DSLR resolution.

5.1 Quantitative comparison

  • Our method significantly improves average PSNR and SSIM over video interpolation and depth-from-video methods across in-between frames and four corner views.Table 1 reports the comparison; the supplied passages do not provide its individual metric values.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 Qualitative comparison

  • Video interpolation methods show temporal artifacts around moving objects, while depthTransfer lacks correct parallax in test and real-world comparisons.The accompanying comparisons show ghosting and artifacts for Epicflow and FlowNet, and less parallax for depthTransfer.Source: §5.2 · p. 9, Figure 11 · p. 10, Figure 12 · p. 11

5.3 Applications

  • The 30 fps light-field video supports refocusing during playback, tracking a selected point in focus, and changing effective aperture.The system demonstrates these applications using an interactive interface.Source: §5.3 · p. 9

5.4 Limitations and future work

  • Occlusion regions remain prone to artifacts because flows are undefined there and can cause errors in warped images.The authors identify improper flow estimation as the usual cause of these artifacts.Source: §5.4 · p. 9
  • The small-baseline Lytro and DSLR setup currently limits capture to objects that are neither too close nor too far away.Very close objects produce large inter-camera occlusions that make alignment hard.Source: §5.4 · p. 9
  • Motion blur often makes flow estimation fail, so exposure must be short enough to avoid blur, a condition harder to meet in dark scenes.The authors suggest deblurring as a possible option for dark scenes.Source: §5.4 · pp. 9–10
  • The system does not yet use the DSLR's extra resolution to generate higher-resolution light-field video.The authors leave high-resolution output to future work.Source: §5.4 · p. 10
  • The authors identify extending visual-understanding tasks such as saliency, material estimation, and matting to dynamic scenes as future work.The proposed extensions concern dynamic objects and scenes.Source: §5.4 · p. 10

6 CONCLUSION

The paper presents a hybrid-camera pipeline that combines sparse light-field frames with 2D video to generate light-field video. The authors report realistic results that outperform current video interpolation and depth-from-video methods, supporting applications including dynamic refocusing and focus tracking.

6 CONCLUSION

  • The method combines a spatio-temporal flow CNN, which propagates temporal information and warps images to a target view, with an appearance CNN that generates the final image.The pipeline combines the 2D video with a sparse light-field sequence.Source: §6 · p. 11
  • The authors report realistic light-field videos and better performance than current video interpolation and depth-from-video methods.Source: §6 · p. 11
  • The generated videos support applications such as dynamic refocusing and focus tracking.Source: §6 · p. 11

A CONCATENATION OF FLOWS

This section describes warping light-field and 2D-video inputs to a target view. The resulting light fields and disparities also support changing focus distances through a video.

A CONCATENATION OF FLOWS

  • The section describes warping five images—It, I0, IT, L0, and LT—to the target view Lt.Source: A CONCATENATION OF FLOWS · p. 11
  • For L0, the warp uses the flow from frame 0 to t to locate z, then evaluates the light-field view at the disparity-adjusted coordinate.The passage defines z as y + f_0→t(y) and rewrites the warp using L0 and d0.Source: A CONCATENATION OF FLOWS · p. 11
  • The same section next gives the warp flows for IT and LT.Source: A CONCATENATION OF FLOWS · p. 11
  • With light fields and disparities at each frame, focus distances can be changed throughout the video.The figure contrasts focus on the front with focus on the back.Source: Figure 14 · p. 12

B TRAINING DETAILS

Training samples use selected angular views from two key-frame light fields and one intermediate 2D frame because the full light-field sequence does not fit in memory. The network predicts a view at the intermediate frame and is trained against its reconstruction target.

B TRAINING DETAILS

  • Because the full light-field sequence cannot fit in training memory, each example randomly samples four angular views plus the central view and selects one intermediate 2D frame.The paper describes these selections as angular and temporal crops.Source: B TRAINING DETAILS · p. 12
  • The application examples compare fixed-plane focus with tracking a user-selected point, and show changing depth of field by using a larger virtual aperture.In the aperture example, the foreground man remains in focus while the background blurs.Source: Figure 15 · p. 12, Figure 16 · p. 12
  • In a challenging scene, the method misses window-line structure occluded by a twig and produces elongated leaves relative to ground truth.Source: Figure 17 · p. 12
  • A training example supplies six views from the two key-frame light fields and one 2D frame, and predicts one angular view at that intermediate time.Source: B TRAINING DETAILS · p. 12
  • Training minimizes Euclidean image-reconstruction distance between the generated intermediate view and its target light-field view.Source: B TRAINING DETAILS · p. 12
Loading 1705.02997v1…