PaperlayerView exact revision on arXiv

Source-linked AI summary

Light Field Primitive for Novel View Synthesis

Liang Chen, Jiahui Ning, Xun Jiang, Xing Xu, Jimmy Ren, Fenglei Fan, Heng Tao Shen

arXiv:2609.31198v1cs.CV

TL;DR

Novel view synthesis methods that anchor scenes in 3D space must mediate between their representation and camera measurements made along rays, while classical light fields require dense ray capture. LFP instead uses differentiable primitives in 4D ray space; it performs competitively on standard benchmarks and supports several imaging effects at quality comparable to specialized frameworks.

  • Problem

    3D-anchored NVS representations must mediate between scene variables and ray-based camera measurements, while classical light fields require dense ray capture and cannot be optimized against posed images.

    Source: §1 · p. 1

  • Method

    LFP represents a scene with compact differentiable primitives in the two-plane ray parameterization, each sharing one learned record across a group of rays.

    Source: §1 · p. 2

  • Results

    LFP is effective on standard NVS benchmarks and matches specialized frameworks in quality on multi-scale anti-aliasing, defocus deblurring with refocusing, fisheye rendering, and transparent-object reconstruction.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Because primitives reside in ray space, optical and appearance effects can be handled by the shared renderer without rebuilding it for each effect.

    Source: §5 · p. 9

    LFP records rays rather than matter, so tasks requiring explicit geometry are not served by the representation as it stands.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present Light Field Primitives (LFP), a formulation for novel view synthesis that replaces the dense ray database with a compact set of differentiable primitives in the classical two-plane parameterization. Each primitive condenses a group of rays into one learned record, and its response to a query is governed by how closely that query belongs to the group. Rendering a camera ray then reduces to compositing all responses it elicits, and a scene can be optimized directly from posed images and rendered in real time with rays. Beyond its competitive performance on standard benchmarks, the main advantage of LFP is structural: its primitives reside directly in the 4D ray space, so optical and appearance effects that are already operations on the light field become behaviors of a single shared renderer. With minimal changes to that renderer, LFP supports multi-scale anti-aliasing, defocus deblurring with refocusing, rendering for fisheye cameras, and even transparent object reconstruction with ray refraction, matching specialized frameworks that devote substantial machinery to these effects.

1 INTRODUCTION

NeRF and 3DGS anchor scenes in 3D space, while camera measurements are radiance along rays, requiring a bridge between the representation and the measurement. LFP places differentiable primitives in 4D ray space, where a shared renderer supports several imaging effects.

1 INTRODUCTION

  • In NeRF and 3DGS, ray-related departures such as coarser pixel resolution and finite aperture require additional renderer complexity for anti-aliasing and defocus.The methods otherwise optimize scene parameters under an ideal model in which each pixel is represented by one ray.Source: §1 · p. 1
  • Classical light fields encode radiance by ray intersections with two planes, but dense ray capture brings high storage costs and limits views to a fixed volume.Without a continuous representation, they interpolate recorded rays rather than optimize against posed images.Source: §1 · p. 1
  • LFP evaluates primitive-query interactions directly in ray space and renders by compositing responses, enabling optimization from posed images and real-time rendering.The representation avoids projecting primitives or translating queries into a separate coordinate system.Source: §1 · p. 2
  • LFP represents scenes as compact differentiable primitives in the classical two-plane parameterization, each holding one learned record for a group of rays.A 4D anchor coordinate and disparity define the ray group's affine coupling; covariance grades query response, while the record stores opacity and radiance.Source: §1 · p. 2
  • LFP is effective on standard NVS benchmarks and extends to four effects at quality comparable to specialized frameworks.The extensions are multi-scale anti-aliasing, defocus deblurring with post-hoc refocusing, fisheye rendering, and transparent-object reconstruction.Source: §1 · p. 3

2 LIGHT FIELD REPRESENTATION

The representation indexes rays by their crossings of two parallel planes, then groups rays through a shared crossing on a third plane. LFP primitives attach a learned record to each such group.

Rays as light field coordinates

  • A ray is indexed in 4D by its tangential coordinates where it crosses two parallel planes, mapping each camera pixel into ray space.The planes are one unit apart and share an orthonormal tangent basis.Source: §2 · p. 3

Epipolar geometry in ray space

  • Rays that see the same content form a ray group when they share a crossing on a third plane determined by content disparity.In two-plane coordinates, the group satisfies an affine coupling between the coordinates on the two planes; its disparity and any one ray specify it.Source: §2 · p. 3

Primitives

  • Each primitive stores an anchor ray and group disparity, an anisotropic covariance, opacity, and base radiance.The covariance controls how sharply response falls off outside the ray group; the directional term captures residual radiance variation within it.Source: §2 · p. 3

Epipolar geometry in ray space

  • Figure 2 shows the progression from two-plane ray coordinates, to a group sharing a third-plane crossing, to query response measured against an anchor ray.The response reflects how far the query departs from the represented ray group.Source: §2 · p. 3, Figure 2 · p. 4

3 METHOD

LFP evaluates each query directly against ray groups in 4D light-field space, then composites primitive responses. Its renderer also incorporates view-dependent appearance and uses ray-space-specific optimization, atlas, and initialization choices.

3.1 RENDERING

  • For query x, LFP measures ray-group displacement on the primitive’s disparity plane, scales it angularly, and maps it through a Gaussian response.The displacement is computed from stored and query coordinates without explicitly intersecting the plane.Source: §3.1 · p. 4
  • The covariance sets response extent, while a half-pixel residual-width floor prevents responses narrower than a view can resolve.The width floor addresses the risk that a primitive becomes difficult for one view to capture and appears as flicker in another.Source: §3.1 · p. 4
  • Primitive responses are opacity-capped and front-to-back composited; disparities set order within one atlas, while multi-atlas scenes sort by camera-space depth.The remaining transmittance contributes to the background.Source: §3.1 · pp. 4–5

3.2 MODEL DETAILS AND OPTIMIZATION

  • Each ray group has constant base radiance and three ASG lobes for view-dependent appearance, with independent bandwidths and a shading axis untied to surface geometry.The lobes can model narrow directional highlights with anisotropic bandwidths.Source: §3.2 · p. 5
  • Sixteen atlases handle rays near-parallel to a single plane pair; sparse SfM points initialize ray-group anchors but are not part of the representation.Points are assigned to atlases by neighboring-normal alignment, and covariance is initialized isotropically at a scale set by local point spacing.Source: §3.2 · p. 5
  • Optimization assigns primitives to affected tiles using response-derived screen radii, regularizes appearance-lobe amplitudes, and densifies from gradients of ray-group parameters.The densification signal uses gradients with respect to the anchor and disparity rather than a projected 2D mean.Source: §3.2 · p. 6

4 EXPERIMENTS

Experiments evaluate LFP on standard novel-view synthesis and additional rendering settings; results show competitive performance while reusing a shared renderer across tasks.

Standard NVS results

  • LFP ranks first on PSNR and SSIM on both standard datasets and second on LPIPS; against 3DGS, it renders at about two thirds of the frame rate with under half the memory.The reevaluated 3DGS run lands within 0.02 dB of its cited result.Source: §4 · p. 7

Experimental scope

  • LFP avoids both ray marching and image-plane projection: queries and primitives share ray coordinates, so their interaction is a subtraction.The experiments cover standard NVS and four additional settings, using the same CUDA renderer kernel across setups.Source: §4 · pp. 6–7

Multi-scale anti-aliasing

  • Without added parameters or training recomputation, LFP is competitive with Mip-Splatting in both zoom directions, ahead on PSNR at the furthest scales and second on SSIM and LPIPS.The evaluation trains once at n=1 and tests zoom-in and zoom-out at other scales.Source: §4 · p. 7, Table 2 · p. 7

Defocus deblurring and refocusing

  • LFP ranks third on PSNR and second on SSIM for deblurring while recovering sharp structures; two per-image scalars also permit post-training refocusing.The aperture term is applied when rendering blurred training views and dropped at test time, without an auxiliary network or added loss.Source: §4 · pp. 7–8, Figure 4 · p. 8, Table 3 · p. 8

Fisheye NVS

  • For fisheye NVS, LFP leads on PSNR and follows closely on the other two metrics without changing renderer stages, maintaining quality across central and peripheral regions.The pixel-to-ray step uses the fisheye model instead of a pinhole; subsequent stages remain unchanged.Source: §4 · pp. 8–9, Table 4 · p. 8

Transparent objects

  • On transparent scenes, LFP outperforms specific refraction-modeling methods by margins that leave refraction far from solved, while preserving glass and liquid contours without explicit modeling.The method represents refracted rays in the same way as other rays because its primitives group rays with coherent radiance.Source: Table 5 · p. 8, §4 · p. 9

5 DISCUSSION AND CONCLUSION

LFP represents rays rather than matter, so it does not provide explicit geometry and its atlas coverage follows the training-camera capture. The conclusion presents ray-space primitives as a shared representation for rendering and several optical effects.

5 DISCUSSION AND CONCLUSION

  • Tasks requiring explicit geometry are not served by LFP as it stands because the representation records rays rather than matter.Atlas normals are fixed from training-camera directions, so coverage follows the capture; primitives in different atlases are ordered using camera-frame depth.Source: §5 · p. 9
  • LFP uses compact differentiable primitives in the two-plane parameterization, with each learned record representing a group of rays.Queries are answered according to their departure from the coupling defining the ray group, without projecting or intersecting primitives or marching along rays.Source: §5 · p. 9
  • The conclusion presents wider tolerance as supporting multi-scale anti-aliasing and defocus deblurring with post-hoc refocusing, while fisheye rendering changes only ray computation and refraction needs nothing.The paper reports these effects as matching methods built specifically for each effect.Source: §5 · p. 9
  • The authors report using generative AI to assist with writing and evaluation scripts, but not with research ideation or experimental design.They state that authors verified the claims, numbers, equations, and citations, and read and tested AI-assisted code.Source: AI USE STATEMENT · p. 10

A RELATED WORK

Prior NVS approaches and learned light-field successors either anchor scenes in 3D or give up the dense explicit ray table. LFP presents sparse differentiable ray-space primitives as a way to retain explicit light-field operations while optimizing from posed images.

A RELATED WORK

  • NeRF integrates a field along camera rays, while 3DGS projects 3D Gaussians onto the image plane; both anchor scenes in 3D space.LFP instead defines primitives natively in 4D ray space and couples them to camera rays through an epipolar response.Source: A RELATED WORK · pp. 12–13
  • Classical light-field rendering parameterizes radiance by rays’ intersections with two planes, enabling aperture integration for refocusing and prefiltering for antialiasing.Its dense ray table requires high sampling rates, bounds views by captured planes, and cannot be optimized against sparse images.Source: A RELATED WORK · p. 13
  • Learned light-field successors compress or restructure the ray record, but optical effects are no longer direct operations on an explicit ray structure.Layered representations approximate these effects with warps and blur kernels; neural light fields rely on learned regularities or repeated queries.Source: A RELATED WORK · p. 13
  • LFP replaces the dense table with sparse differentiable primitives, each storing one learned record for a group of rays and enabling optimization against posed images.Epipolar geometry gathers rays sharing a primitive’s record rather than serving only as a depth cue or view-sampling rule.Source: A RELATED WORK · p. 13
  • LFP retains explicit light-field operations: defocus is an integration over rays, and anti-aliasing is a prefiltering of them.The paper frames this as retaining light-field optical advantages without the density burden or separate mechanisms for these effects.Source: A RELATED WORK · p. 13

B MODEL DETAILS

The model details describe constrained primitive fields, directional appearance lobes, adaptive primitive-count control, and atlas assignment based on training-camera directions and local surface normals.

B MODEL DETAILS

  • The primitive’s disparity is stored in log form and exponentiated to stay positive, while its opacity logit passes through a sigmoid to remain between zero and one.The primitive fields include μᵢ, pᵢ, Σᵢ, oᵢ, and cᵢ; Sec. 3.2 adds directional lobes.Source: B MODEL DETAILS · p. 13
  • The covariance is formed from three scalars with distinct geometric roles, and its two exponentials give squared half-axes oriented by R(θᵢ).Its closed-form Cholesky factor is consumed by the renderer.Source: B MODEL DETAILS · p. 14
  • For the covariance and appearance lobes, separating scale from anisotropy keeps the two covariance directions from competing through one shared parameter.The lobe matrix is assembled from scalar parameters, including an angle and two log bandwidths represented as a mean and ratio.Source: B MODEL DETAILS · p. 14
  • Each primitive’s atlas index is assigned at initialization and remains fixed; the shared atlas basis and camera-dependent projections are precomputed once per frame.The caption depicts atlases sharing the scene centre while differing in orientation.Source: B MODEL DETAILS · pp. 14–15, Figure 5 · p. 15
  • Every 100 iterations from 500 to 10,000, primitives selected by averaged placement gradients are cloned or split according to their larger radius.Opacity resets occur every 3,000 iterations from 3,000 to 10,000; clones are used below a 0.005 scene-extent radius threshold and splits otherwise.Source: B MODEL DETAILS · p. 14
  • A clone retains its parent’s shape, orientation, and atlas but is displaced in the st plane; parent and child each keep half the parent’s opacity.A split replaces the parent with two children that halve its longer radius while retaining the shorter radius; round primitives shrink isotropically.Source: B MODEL DETAILS · p. 14
  • Primitives are pruned when opacity falls below 0.005 or their larger radius exceeds 0.1 of the scene extent..Source: B MODEL DETAILS · p. 15
  • Atlas normals are fixed before optimization as k-means centres of training-camera directions, using 16 clusters on every benchmark.The vertical axis is taken from the direction of least variance, and initialization points are assigned among eligible atlases by alignment with a local-PCA surface normal.Source: B MODEL DETAILS · p. 15, Figure 5 · p. 15

C TILE ASSIGNMENT WITHOUT PROJECTION

The tile-assignment method derives a primitive’s image-space reference position and falloff extent from its ray-space residual rather than projecting a 3D primitive. It assigns tiles using a locally estimated extent and recentres cases whose reference position falls outside the image.

C TILE ASSIGNMENT WITHOUT PROJECTION

  • LFP assigns primitives to tiles from the pixel-dependent response rather than a projected covariance: the residual’s zero gives the centre, and its slope gives the extent.The resulting rectangle is used to list a primitive only in tiles it can affect; tile lists are then sorted by depth and composited.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · pp. 15, 17
  • The reference position is the image location of the query that belongs to the primitive’s ray group, where its response reaches its maximum.It is computed directly by inverting the pixel-to-ray map, without searching over pixels.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · p. 15
  • Three pixel-to-ray evaluations estimate how quickly the residual changes per pixel near the reference position, avoiding differentiation of the projective composition.This linearization determines tile listing only; shading computes the residual directly.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · pp. 15–16
  • The rasterizer uses a five-standard-deviation response cutoff together with the residual rate to estimate the pixels that matter.The resulting pixel-space quadratic form describes an ellipse around the reference position.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · p. 16
  • The ellipse is bounded by an axis-aligned rectangle whose half-widths come from its inverse covariance, then clamped to 3–1024 pixels.The lower bound keeps a primitive in at least its peak tile, while the upper bound limits emitted entries in near-singular cases.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · p. 16
  • When the reference position falls outside the image margin, the method clamps it and uses one Newton step to recenter the rectangle at the linearized residual’s zero.This can retain a primitive whose rectangle still reaches the image, as illustrated in Fig. 6(b).Source: C TILE ASSIGNMENT WITHOUT PROJECTION · p. 17, Figure 6 · p. 17
  • A Newton step is rejected if it exceeds 128 pixels or the rate matrix is near singular; the method then uses the clamped position and raises both half-widths to at least 32 pixels.The near-singular case corresponds to a grazing configuration with a response nearly constant along an image line.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · p. 17
  • Primitives with camera-frame depth below 0.05 or rectangles missing the image are dropped; remaining entries are sorted by tile and depth.The tiled compositor traverses the lists in reverse during the backward pass.Source: C TILE ASSIGNMENT WITHOUT PROJECTION · pp. 16–17

D ANTI-ALIASING AND DEFOCUS DEBLURRING

A pixel is modeled as a bundle of rays, so aliasing and defocus can be handled by widening the primitive response covariance. Convolving the response with the bundle adds its covariance P and lowers the peak by a determinant-ratio factor.

D ANTI-ALIASING AND DEFOCUS DEBLURRING

  • A pixel bundle captures both aliasing from coarse spatial sampling and defocus from a finite aperture spreading rays over directions.The response is not read more narrowly than a view can resolve.Source: D ANTI-ALIASING AND DEFOCUS DEBLURRING · p. 18
  • Convolving the unnormalized Gaussian response with a pixel’s ray bundle adds P to the covariance and lowers the peak by the square root of a determinant ratio.The response, record, and compositing remain unchanged; the two optical effects are represented by terms added to Σ_i.Source: D ANTI-ALIASING AND DEFOCUS DEBLURRING · p. 18
  • The pixel-footprint scale is ρ_i = Z_cam_ij π/(2f), with f = max(f_x, f_y) when focal lengths differ.Taking a half-pixel as one standard deviation is a convention; the ratio t_i/ρ_i² is unchanged under any convention.Source: D ANTI-ALIASING AND DEFOCUS DEBLURRING · p. 18

D.1 ANTI-ALIASING

For anti-aliasing, LFP widens primitive covariance to match the output pixel’s footprint and adjusts the peak factor. The construction supports rendering at other resolutions without re-optimization, while qualitative comparisons show fidelity across zoom-in and zoom-out settings.

D.1 ANTI-ALIASING

  • Rendering at a new resolution uses Σ_i + t_iI, and finer-than-training output holds t_i at ρ_i², allowing any resolution without re-optimization.At finer output scales, the footprint is not narrowed below the width supported by the training observations.Source: §D.1 · p. 18
  • The scale correction replaces the training footprint with the output footprint; its determinant ratio cancels the fitted covariance and equals one at the training scale.The opacity is fitted during training with the training footprint in place, so the correction is applied at render time.Source: §D.1 · p. 18
  • The output footprint is calibrated from the block of training pixels averaged at each scale, combining the box-filter variance with each pixel’s footprint variance.The supplied reproduction details specify determinant evaluation on the rasterizer’s floored Cholesky factors and set ρ_i = 0 for near-plane-culled primitives.Source: §D.1 · p. 18
  • At 1/8 output scale, LFP stays close to ground truth across resolutions, while 3DGS brightens and dissolves thin structures; in zoom-in tests it also stays faithful where baselines show artifacts.Mip-Splatting is similar to LFP in the zoom-out setting and smears colour and washes out fine texture in the zoom-in setting.Source: §D.1 · p. 19, Figure 7 · p. 19

D.2 DEFOCUS AND REFOCUS

For defocus, LFP maps the finite-aperture ray spread into residual-space covariance and applies the same covariance-widening construction. The trained scene can then be rendered sharp or refocused by changing camera focus and aperture.

D.2 DEFOCUS AND REFOCUS

  • A thin lens spreads rays by a circle of confusion whose radius vanishes at the focal plane and varies linearly with aperture.Its depth is already computed for compositing.Source: §D.2 · p. 20
  • LFP maps aperture spread into residual space using the first-order derivative Q_ij and a Gaussian with the disc’s matching second moment.Only aperture K_j and focus F_j are added as optimized per-image scalars; Q_ij follows from the parameterization and is not learned.Source: §D.2 · p. 20
  • Training with the aperture term and dropping it at test time yields a sharp rendering from defocused observations; changing F_j and K_j afterward enables refocusing and new depth of field.These camera parameters remain free after training, allowing focus sweeps and apertures not captured in the training images.Source: §D.2 · p. 20
  • Unlike 3D-anchored representations that add aperture operators, LFP applies aperture spread to the covariance already read by its renderer.The covariance also carries the pixel-spread term.Source: §D.2 · p. 20

D.3 THE PEAK FACTOR AT TRAINING AND AT TEST TIME

The peak factor is omitted while fitting defocused observations because the optimized opacity and camera parameters absorb it. For post-training refocusing, the factor is applied, but the Gaussian approximation within alpha compositing makes the result inexact.

D.3 THE PEAK FACTOR AT TRAINING AND AT TEST TIME

  • During fitting, the peak factor is omitted because opacity and per-image aperture and focus parameters optimize through the same loss and absorb its effect.Retraining with the factor raised to different powers was worse on every metric at every power, with performance returning monotonically toward released behavior as the power approached zero.Source: §D.3 · p. 20
  • For post-training refocusing, the peak factor is applied because opacity is frozen and the new aperture and focus were not present in training.The changed factor therefore affects the render rather than being absorbed during fitting.Source: §D.3 · p. 20
  • The refocusing result is not exact: the Gaussian-disc approximation makes frames a few percent darker, while a thin bright feature can lose about a third of its brightness.The passage attributes the thin-feature loss to its being carried by few primitives whose footprints grow fastest relative to their original size.Source: §D.3 · p. 20

E ABLATION STUDY

Ablations test appearance basis, lobe count, regularization, atlas design, normal settings, and entry-plane coordinates. Results favor directional ASG lobes, 16 atlases, K=3, and retaining highlight regularization, while freeing entry-plane coordinates leaves quality unchanged.

Appearance model

  • ASG restores reflection intensity and shape where RGB flattens highlights and degree-3 SH smears them despite 45 versus 27 coefficients; the same advantage appears in 3DGS.The comparison uses a matched appearance budget and shows the angular basis itself accounts for the gain, especially where radiance varies sharply with viewing direction.Source: Figure 8 · p. 21, E ABLATION STUDY · pp. 21–22, Table 7 · p. 21

Different lobes for ASG

  • K=3 peaks at 28.02 dB PSNR, 0.820 SSIM, and 0.247 LPIPS; K=1, 3, and 5 render at 120.6, 101, and 84.4 FPS, making K=3 the quality–speed knee.K=1 loses 0.15 dB, while K=5 reaches 27.97 dB without recovering that loss; its extra lobes add per-primitive work without usable angular detail.Source: E ABLATION STUDY · p. 22, Figure 9 · p. 22

Highlight regularization

  • Removing λg∥g∥2 lowers mean PSNR from 28.02 to 27.85 dB, costs 0.005 SSIM and 0.008 LPIPS, and grows mean penalized-lobe amplitude from 0.098 to 0.640.The unregularized model is 12% smaller, 1.75 M versus 2.00 M primitives, as lobes absorb photometric error that would otherwise drive densification.Source: E ABLATION STUDY · p. 22, Table 9 · p. 23

Atlas design

  • One atlas loses 6.92 dB; four recover 6.62 dB, eight another 0.18 dB, sixteen another 0.12 dB, and 32 adds 0.01 dB while trailing sixteen outdoors.Coverage is saturated by sixteen atlases, the smallest count on the plateau; doubling atlas count also raises per-camera precomputation and places more primitives on the cameraframe ordering.Source: E ABLATION STUDY · p. 22, Table 8 · p. 23

Atlas normals

  • With fixed cluster-center normals, all three metrics degrade more indoors (0.36 dB) than outdoors (0.02 dB); optimizing normals never helps and costs 0.18 dB at the largest step size.At step sizes 10^-5 and 10^-4, normals rotate only 0.10° and 0.28° after 30k iterations, with metrics unchanged within noise.Source: E ABLATION STUDY · pp. 22–23, Table 8 · p. 23

Optimizing entry-plane coordinates

  • Freeing entry-plane coordinates leaves results unchanged within run-to-run noise: 28.01 dB / 0.820 / 0.247 versus 28.02 dB / 0.820 / 0.247 for the anchored model.The response depends on the ray group, not which of its rays stores the record; fixing these coordinates removes two learnable parameters and two atomic accumulations per primitive at no quality cost.Source: E ABLATION STUDY · p. 23

Related papers

Loading 2609.31198v1…