Sparse Light Field Sampling Improves Casual 3D and 4D Reconstruction

1 Introduction

• Novel-view-synthesis methods generally assume monocular input, while camera motion trades angular for temporal resolution and provides effective multi-view cues only when it outpaces scene motion.Even multi-camera phone and tablet datasets often use only one camera for training, while sparse-view methods commonly turn to learned priors or regularizers.Source: §1 · p. 2
• The study frames multi-view capture as sensor-limited when one sensor trades spatial for angular resolution, and exposure-limited when synchronized sensors observe each event simultaneously.Matching simultaneous views with a moving monocular camera costs capture time and, in dynamic scenes, observes a different scene state.Source: §1 · p. 2
• Using all available cameras substantially improves few-shot static and dynamic reconstruction; fixed-budget angular sampling helps when exposures are scarce, with the largest gains in single-shot and dynamic scenes.These hardware-derived gains complement learned priors rather than being subsumed by them.Source: §1 · p. 3
• The paper presents static and dynamic real-world datasets captured with iPhone 15 Pro, Apple Vision Pro, and Lytro Illum cameras, and benchmarks 3DGS, 4DGS, and prior-based sparse-view methods against monocular baselines.Source: §1 · pp. 2–3
• The multiplexed light field prototype outperforms monocular and standard light field capture from a single exposure on real captures.Its higher-resolution sub-lens images overlap on the sensor, and the design improves the angular-spatial tradeoff in simulation and on the physical prototype.Source: §1 · p. 3

2 Related Work

• Radiance-field methods assume one monocular image per camera pose, relying on camera motion for angular coverage; sparse views, short trajectories, and dynamic scenes make reconstruction increasingly ill-posed.Source: §2 · p. 3

Sparse-view, casual capture, and dynamic novel view synthesis.

• Feed-forward models accept one or a few views, and structural, generative, and dynamic priors constrain reconstruction, but these approaches do not replace insufficient per-timestep angular measurements [10,11,46,39, 48, 56, 57, 65, 17, 33, 53–55, 61,8,14,29,41,50,52,58].Source: Sparse-view, casual capture, and… · p. 4
• Existing NVS benchmarks assume monocular input, and none evaluates reconstruction quality against angular sampling density or compares consumer camera types on the same scenes.Source: Benchmarks and datasets · p. 4

Light field sampling.

• Light-field cameras trade spatial resolution for angular resolution, while consumer stereo, dual-pixel, and multi-camera phones already provide sparse simultaneous views whose reconstruction benefits have not been systematically evaluated.Plenoptic sampling theory specifies alias-free angular rates, but these typically require dedicated hardware or dense trajectories.Source: Light field sampling · p. 4

3.1 Camera Models

• The benchmark models synchronized exposures as calibrated sub-views and evaluates smartphone arrays, stereo, light-field, and multiplexed light-field cameras.Each exposure’s sub-views supervise the same Gaussians through the camera forward model.Source: §3.1 · p. 5, Figure 2 · p. 5
• Angular baseline θ is the scale-invariant maximum inter-view angle from the scene center; real-data medians are 5.06° (iPhone), 15.1° (stereo), and 0.74° (Lytro).It increases with wider baselines or closer scenes, unlike metric baseline, which depends on scene depth.Source: §3.1 · pp. 5–6

3.2 Sensor-Limited Multi-View

• With a fixed sensor budget, an M × M light-field grid trades per-view resolution of 1/M^2 for M^2 directions, favoring angular samples when exposures are scarce.Those angular samples triangulate geometry that additional spatial resolution in a single view cannot recover.Source: §3.2 · p. 6
• The multiplexed camera overlaps sub-lens footprints to retain larger pixel footprints, then reconstructs by simulating superposed sub-views; experiments report the tradeoff pays off.Its calibration estimates each sub-view’s camera parameters and pose, while the forward model linearly superimposes rasterized sub-lens images for single-measurement supervision.Source: §3.2 · p. 6

3.3 Exposure-Limited Multi-View

• Synchronized physical-camera sub-views provide single-timestep parallax unavailable to sequential monocular capture of dynamic scenes, so gains are expected greatest for dynamic, single-exposure capture and to shrink with camera motion.To isolate multi-view effects, each camera model is compared with a monocular stream from the same camera; dynamic evaluation uses 4D Gaussian Splatting on smartphone and stereo cameras only.Source: §3.3 · p. 7

4.1 Multi-View Dataset

• The released real-world datasets cover 16 scenes—eight static and eight dynamic—and the static captures include iPhone 15 Pro, Vision Pro stereo, and Lytro Illum views.Static data include one exposure per camera and handheld videos from the iPhone and stereo cameras; the Lytro provides 81 sub-views per exposure. Device descriptions retain their printed markers:,, and.Source: §4.1 · p. 8, Static scene dataset · p. 8

Dynamic scene dataset.

• Dynamic videos cover eight real-world scenes using the iPhone’s three cameras and Vision Pro’s two cameras, with poses estimated per rig and propagated across sub-views.Videos are time-subsampled to 10 FPS for iPhone and 15 FPS for stereo; pose estimation uses VGGT and MegaSaM.Source: Dynamic scene dataset · pp. 8–9

4.2 Synthetic Experiments

• With scarce exposures, light field cameras outperform monocular capture under the same sensor budget, while iPhone and stereo multi-view models also outperform their matched-exposure monocular baselines.Multiplexing further improves sensor-limited performance; as exposures increase, camera designs converge. The results show angular diversity can outweigh reduced spatial resolution when views are limited.Source: Figure 4 · p. 8, Table 1 · p. 9, Figure 5 · p. 9
• All four light field sampling cameras reduce depth error relative to monocular capture at both exposure counts, with the largest improvements from light field and multiplexed light field models.The evaluation reports absolute relative error and root mean squared error on NeRF Synthetic test views at θ = 10°; Fig. 6 shows the trend qualitatively.Source: Table 2 · p. 10, Figure 6 · p. 10, §4.2 · p. 11

Sensor-limited.

• Using all Lytro sub-views improves mPSNR and mSSIM over the central view across reconstruction methods, while low sub-view resolution can soften textures and affect mLPIPS.The comparison matches the total pixel count to the monocular image; full light field input also resolves occlusion-boundary failures seen in monocular reconstructions.Source: Table 3 · p. 12, Figure 8 · p. 12, Sensor-limited · p. 13

Exposure-limited.

• For real single-exposure static scenes, synchronized iPhone and stereo views improve all reported metrics over their monocular baselines and complement learned sparse-view regularizers.The devices are compared against the iPhone wide camera and the stereo right camera, respectively.Source: Table 3 · p. 12, Figure 9 · p. 13, Exposure-limited · p. 14
• In casual handheld video, both multi-view cameras outperform their monocular counterparts, but gains are smaller because camera motion already supplies angular diversity.The study also evaluates training with Difix3D+ [53], which uses diffusion-based artifact correction.Source: Table 4 · p. 13, Exposure-limited · p. 14, Figure 10 · p. 14

4.4 Real-World Dynamic Scenes

• Synchronized multi-camera input produces the largest gains in dynamic reconstruction, recovering geometry and motion that stationary monocular input smears across time.The comparison uses each device’s multi-view stream against its monocular stream for 4DGS on held-out frames; the limitation of monocular input is also noted in the 4DGS paper.Source: Figure 11 · p. 14, §4.4 · p. 15, Table 5 · p. 15

5 Discussion

• Synchronized multi-view capture improves single-exposure and dynamic reconstruction across synthetic and real-world experiments, complementing learned sparse-view priors.Source: §5 · p. 15
• With a fixed sensor budget and scarce exposures, angular samples are more valuable than spatial resolution, while multiplexing recovers part of the lost resolution.Source: §5 · p. 15
• Benefits increase with denser angular sampling, wider baselines, and nearby objects, but may diminish for distant content in large, unbounded scenes with minimal disparity.Source: §5 · p. 15
• The authors urge future datasets to record all available camera streams and hope the results inform camera designs jointly built with reconstruction algorithms.Source: §5 · p. 15

A Background on Gaussian Splatting

• 3DGS represents a scene as anisotropic 3D Gaussians, each with a center, opacity, view-dependent color, and covariance.Source: A Background on Gaussian Splatting · p. 21
• Each Gaussian’s covariance is formed from its rotation and scale as Σ_i = R_i S_i S_i^T R_i^T.Source: A Background on Gaussian Splatting · p. 21
• For a given camera, each Gaussian is projected to a 2D mean.Source: A Background on Gaussian Splatting · p. 21
• A projected Gaussian contributes opacity at a pixel according to its distance from the 2D mean and its 2D covariance.Source: A Background on Gaussian Splatting · p. 21
• 3DGS sorts contributing Gaussians front to back and renders pixel color with standard alpha compositing.Source: A Background on Gaussian Splatting · p. 21
• Accumulated transmittance T_i handles occlusion within each rendered sub-view.Source: A Background on Gaussian Splatting · p. 21

A.2 FSGS

• FSGS supervises rendered depth maps against a pretrained monocular depth estimator with a scale-invariant depth correlation loss.This constrains scene geometry in regions left underdetermined by sparse training views.Source: §A.2 · p. 21
• FSGS [16] densifies sparse SfM initialization through proximity-guided Gaussian unpooling, placing new Gaussians between existing neighbors to fill coverage gaps.Source: §A.2 · p. 22
• FSGS [16] synthesizes pseudo-views near training cameras and supervises them with the same depth prior to regularize unseen viewpoints.Source: §A.2 · p. 22

A.3 SparseGS

• SparseGS combines monocular-depth correlation, diffusion-model score distillation for novel-view appearance, and floater pruning to address few-shot reconstruction artifacts.Its patch-based Pearson correlation loss aligns rendered and predicted depth while remaining robust to unknown scale and shift.Source: §A.3 · p. 22
• 4DGS models dynamic scenes by deforming canonical Gaussians over time with a learned residual function, avoiding per-frame Gaussian duplication.Its deformation field uses K-planes factorization and a decoder that predicts position, rotation, and scaling residuals from spatiotemporal features.Source: §A.3 · p. 22

B.1 Optical Design

• The multiplexed optical system uses a main objective, lens array, aperture array, and sensor; its design draws particularly on and.The aperture array blocks stray high-angle light and limits each sub-view’s effective aperture to manage throughput, depth of field, and optical crosstalk.Source: §B.1 · pp. 22–23
• The lens array forms overlapping sub-images, while the aligned aperture array controls light and crosstalk; the main lens and object placement maximize captured perspective shift.Unlike a standard plenoptic camera, the sensor is brought closer to the lens array to increase sub-image size.Source: §B.1 · p. 23

B.2 Forward Model

• The forward model treats each lenslet as an independent camera and represents the sensor measurement as a normalized, weighted sum of lenslet images.The weights indicate whether each lenslet footprint covers a pixel, and normalization uses the maximum number of overlapping lenses.Source: §B.2 · p. 23

B.3 Camera Calibration

• Calibration isolates each lenslet in turn with physical masks, then uses VGGT and bundle adjustment to estimate fixed intrinsic and extrinsic parameters.White-field images with uniform illumination determine the lenslet spatial masks for dataset collection.Source: §B.3 · p. 23

C Additional Implementation Details

• For Blender scenes without SfM point clouds, 100K Gaussians are initialized inside a sphere rather than a cube, reducing floater artifacts, especially in backgrounds.The spherical support avoids placing Gaussians in empty corners; training for approximately 3,000 iterations was sufficient for convergence in the experiments.Source: C Additional Implementation Details · p. 24, Figure 1 · p. 24
• For real-world experiments, Gaussians are initialized from sparse SfM points obtained during calibration, following 3DGS.Source: C Additional Implementation Details · p. 24
• Figure 2 shows single-exposure Lego measurements across camera models: light-field and multiplexed cameras split a fixed sensor, while iPhone and stereo cameras add full-resolution synchronized sensors.Source: C Additional Implementation Details · p. 24, Figure 2 · p. 25

• Table 4 reports per-scene metrics for 4DGS reconstruction on real dynamic scenes captured with a fixed camera.Source: D Additional Per-Scene Results · p. 25, Table 4 · p. 29
• Table 1 reports per-scene metrics for all NeRF synthetic Blender scenes [10] at 10°; one-exposure methods use adjacent-camera metrics, while three-exposure methods use full test-camera metrics.Source: Table 1 · p. 26
• Table 2 reports per-scene masked metrics for single-exposure captures of real-world static scenes.Source: D Additional Per-Scene Results · p. 25, Table 2 · p. 27
• Table 3 reports per-scene masked metrics for casual-video reconstruction, including reconstructions using learned priors.Source: D Additional Per-Scene Results · p. 25, Table 3 · p. 28