Source-linked AI summary
X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation
Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, Tobias Ritschel
TL;DR
The paper addresses interpolation of sparse image sets across view, time, and illumination without the storage and sampling demands of dense capture. It learns a per-scene neural X-Field whose differentiable graphics structure maps coordinates to images or pixel motion. The resulting representation supports high-quality interpolation, minutes-long training, compact deployment, and real-time navigation, while remaining limited to the trained scene and compatible scene-model scope.
Problem
Sparse sampling makes view, time, and illumination exploration difficult because dense capture has excessive storage, capture, and processing requirements.
Method
The method learns a neural network that maps X-Field coordinates to images through implicit geometry or correspondence and differentiable projection, lighting, and warping.
Results
The representation achieves high-quality, high-performance interpolation across view, time, and light, outperforming several competitors across advanced light transport and fine spatial detail.
Takeaways & Limitations
A per-scene X-Field can be deployed after about 20 minutes of pre-calculation for interactive, high-quality VR playback at real-time rates.
Takeaways & Limitations
The architecture generalizes across parameters within one specific X-Field, not across scenes, and scenes beyond its primitive image-formation model may fail.
Abstract
from arXiv · showhide
We suggest to represent an X-Field -a set of 2D images taken across different view, time or illumination conditions, i.e., video, light field, reflectance fields or combinations thereof-by learning a neural network (NN) to map their view, time or light coordinates to 2D images. Executing this NN at new coordinates results in joint view, time or light interpolation. The key idea to make this workable is a NN that already knows the "basic tricks" of graphics (lighting, 3D projection, occlusion) in a hard-coded and differentiable form. The NN represents the input to that rendering as an implicit map, that for any view, time, or light coordinate and for any pixel can quantify how it will move if view, time or light coordinates change (Jacobian of pixel position with respect to view, time, illumination, etc.). Our X-Field representation is trained for one scene within minutes, leading to a compact set of trainable parameters and hence real-time navigation in view, time and illumination.
1 INTRODUCTION
The paper introduces X-Fields as compact neural representations for images varying across view, time, illumination, or combinations of these dimensions. A differentiable rendering structure enables interpolation from sparse observations, with minutes-long per-scene training and real-time decoding.
- X-Fields represent image sets conditioned on view, time, light, or other dimensions as a universal, compact, interpolatable neural-network representation.
- The method implicitly represents complete X-Fields with a trainable high-dimensional getPixel mapping coordinates to observed or interpolated images.
- A suitable network combines learned representation with differentiable projection and lighting so basic graphics operations need not be learned from scratch.
- Per-scene training takes minutes, decoding runs at about 20 Hz at 1024×1024 resolution, and parameters add only a few kilobytes over image samples.
- The implementation and training data are publicly available.
2 PREVIOUS WORK
Previous work spans classical and neural interpolation across views, time, and illumination, using blending, proxy geometry, volumes, or implicit representations. This paper extends learned warping toward a single per-scene representation that jointly handles geometry, motion, illumination, and efficient deployment.
- Prior interpolation methods include linear blending, proxy geometry, per-view geometry, multi-plane representations, occupancy volumes, and learned implicit functions across view, time, and illumination.
- Linear blending applies broadly but produces ghosting, while volumetric approaches incur massive data, distribution, and volume-rendering bandwidth costs.
- Instead of storing volumetric data, the approach uses implicit neural parameters requiring only a few additional kilobytes over input images while rendering in real time.
- Implicit methods replace the pixel basis with neural queries for occupancy, color, or flow, but many require extensive training for specific scene classes.
- The proposed representation jointly generalizes over complete geometry, motion, and illumination changes without generalizing across different scenes.
3 BACKGROUND
The background motivates neural interpolation because learned representations can avoid pixel-basis ghosting, while differentiable warping preserves spatial detail. The validation examples show that neural flow interpolation combines smooth motion with textured detail.
- Neural representations interpolate observed image structure more smoothly than pixel-basis blending, which can create fade-in, fade-out, and ghosting artifacts.
- The interpolation target lies between observed coordinates, where the unobserved result should remain within the sampled X-Field domain.
- Differentiable rendering adds projection, shading, and occlusion units that free the network from learning basic graphics concepts.
- Direct appearance regression is unnecessarily difficult, whereas warping changes view more effectively and preserves image details.
- In the logo validation, nearest and linear sampling cause blur or steps, color interpolation misses fine stripes, and flow interpolation captures them.
4 OUR APPROACH
The approach learns a neural X-Field function from sparse observed coordinates to images, then interpolates unobserved coordinates by decomposing appearance, warping observations, and combining consistent results. Its learned flow predicts pixel motion from X-Field coordinates while differentiable fixed operators perform warping and reconstruction.
- 4.1 Objective: The X-Field maps an n_d-dimensional coordinate to a 2D RGB image and is trained only on sparse observed coordinates, targeting interpolation within their convex combination.Training does not evaluate unknown coordinates because their ground-truth images are unavailable.
- 4.2.1 De-light: Appearance is decomposed into shading and albedo, each interpolated independently and recombined multiplicatively to produce radiance at an unobserved coordinate.The model treats shading and albedo as warped versions of observed components, although the factorization assumptions need not strictly hold.
- 4.2.2 Interpolation: Interpolation warps every observed image and merges the deformed results as a weighted combination, applying the same operation independently to shading and albedo.Per-pixel weights handle flow consistency during merging.
- 4.2.3 Warping: A spatial transformer uses differentiable flow maps to read source pixels, with finite X-Field changes converted into absolute pixel positions for warping.The Jacobian describes infinitesimal pixel motion, while multiplying it by a coordinate delta yields the finite motion used by the transformer.
- 4.2.4 Flow: A CNN maps only the target X-Field coordinate to a Jacobian field, encoding geometric structure in the network rather than using pixel-basis observations.This coordinate-only representation is intended to remain interpolatable across unobserved coordinates.
- 4.2.5 Consistency: Multiple flows and consistency weights resolve incompatible motion by downweighting inconsistent results; when none is consistent, normalized weights reduce to linear blending.The architecture addresses inconsistencies directly instead of relying on a consistency loss.
5 RESULTS
The evaluation compares X-Field interpolation with domain-specific baselines and ablations across view, time, light, and higher-dimensional settings. The method achieves the best quality across all tasks and metrics, while remaining compact, fast to train, and real-time at deployment.
- Comparison: Our method provides the best quality in all evaluated tasks, according to all metrics and domains.Table 2 reports the main comparison across methods, dimensions, and L2, SSIM, and VGG metrics.
- Comparison: View interpolation preserves details and plausible motion better than competing methods, which exhibit outliers, blur, or ghosted structures.The comparison covers both dense and sparse light fields, including a withheld reference view.
- Comparison: Temporal interpolation remains similar to ground truth under large motion, whereas blending and SuperSlowMo produce unusable ghosting or failures.Triplet interpolation also represents strong non-rigid changes involving transparency and scattering.
- Comparison: Light interpolation deforms images to retain sharp shadows and highlights, while direct regression blurs them; quality converges with Xu et al. [2018] at wider baselines.Xu et al. [2018] requires a custom capture setup and cannot be applied to the authors’ data.
- Evaluation: Albedo-shading splitting is critical for reproducing shadows on textured surfaces, and the method achieves coherent navigation through smooth Jacobian-based deformation and soft occlusion.The method can miss details or over-smooth, but its construction avoids color flicker during traversal.
- Evaluation: The method’s main limitations are artifacts from smooth, detail-poor depth and flow maps and difficulties with deformations that are not fully rigid.The authors leave detail recovery, such as adversarial refinement, for future work; face-like non-rigid cases can still produce plausible images.
6 DISCUSSION / LIMITATIONS
The method’s success depends on data, model assumptions, and network capacity, with sparse observations and finite capacity creating characteristic artifacts. It supports view-based effects such as depth of field and motion blur, but remains limited by insufficient data, model mismatch, and overly complex depth structure.
- Data: Sparse observations cannot reconstruct unseen information, causing occlusion-related rubber-like stretches and artifacts when scene content appears in only one view.The method learns fill-in through the X-Field Jacobian, so disoccluded pixels move like neighboring pixels; consistency weighting can reduce stochastic fine-scale artifacts.
- Model: The approach works when scene changes roughly follow its hard-wired image-formation model, but model mismatch can cause failure regardless of data or representation capacity.Dominant transparency, autoexposure changes, and non-unique indirect-shadow flow violate the flow-based assumptions.
- Model: Processing shading and albedo separately, representing the full X-Field Jacobian, and learning nonlinear inverse flow make unique-flow reasoning more applicable.If flow estimates remain wrong, consistency weighting degenerates to linear blending; layered flow is identified as future work.
- Capacity: Finite network capacity smooths flow, shape, illumination, and motion, limiting representation of complex depth structures and stochastic variation.Nonlinearities permit some sharpness, but stochastic scenes may require inserted stochasticity combined with a style loss.
- Training: After about 500 epochs, or 30 minutes, the result is usable, and convergence occurs after about 1000 epochs, or 1 hour, for the shown scene.The training examples are only 5×5, so the epochs are short.
- Applications: View interpolation also supports depth-of-field and motion-blur effects by averaging images across lens or shutter coordinates.Both effects are computed from many generated images at X-Field coordinates spanning the relevant lens or shutter range.
7 CONCLUSION
The paper represents X-Fields with a neural network that produces images conditioned on view, time, and light coordinates. The resulting interpolation outperforms several competitors across advanced light transport and fine spatial detail, while fixed-scene specialization enables interactive deployment after precomputation.
- Conclusion: The method delivers high-quality, high-performance interpolation that outperforms several competitors across advanced light transport, fine plant details, single objects, and entire scenes.The reported scope includes BRDFs, soft shadows, global illumination, caustics, reflections, transparency, still-life scenes, tabletop soccer, and parks.
- Conclusion: A learnable view-time geometry model combined with warping and consistency reasoning performs better than direct color regression, unocclusion-aware warping, and domain-adapted state-of-the-art solutions.
- Conclusion: The approach simplifies the task by interpolating a fixed set of X-Field conditions rather than generalizing across all possible image combinations.This specialization supports deployment in a VR application after about 20 minutes of pre-calculation, with interactive playback rates.
- Future work: Future work includes sparse or unstructured capture, extrapolation, faster training, interpolation over wavelength or spatial audio, and reconstruction from even sparser observations.