PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Fields in Visual Computing and Beyond

Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath Sridhar

arXiv:2111.11426v4cs.CVcs.GRcs.LG

TL;DR

Rapid growth in neural-fields research has created a need for a shared mathematical formulation and a review that can remain current. This report defines and organizes neural-field techniques, surveys applications across visual computing and adjacent fields, and provides a living community website; it concludes that neural fields are broadly applicable and may support progress across many areas of computer science.

  • Problem

    Rapid progress has produced many neural-fields papers without a shared mathematical formulation, while selective amnesia and quickly outdated surveys hinder consolidation of the area.

  • Method

    The report reviews over 250 papers, defines neural fields through fields of physical quantities, unifies common techniques mathematically, surveys applications, and provides a living database.

  • Results

    The review covers neural-field techniques and applications across graphics, vision, robotics, medical imaging, computational physics, and other areas, demonstrating broad applicability.

  • Takeaways & Limitations

    Neural fields provide a widely applicable way to represent and process visual-computing problems, with low-dimensional supervision potentially relaxing hardware constraints for 3D content creation.

  • Takeaways & Limitations

    Neural fields have limited tools for editing and manipulation, which significantly limits their use cases.

Abstract

from arXiv · show

Recent advances in machine learning have created increasing interest in solving visual computing problems using a class of coordinate-based neural networks that parametrize physical properties of scenes or objects across space and time. These methods, which we call neural fields, have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation. However, due to rapid progress in a short time, many papers exist but a comprehensive review and formulation of the problem has not yet emerged. In this report, we address this limitation by providing context, mathematical grounding, and an extensive review of literature on neural fields. This report covers research along two dimensions. In Part I, we focus on techniques in neural fields by identifying common components of neural field methods, including different representations, architectures, forward mapping, and generalization methods. In Part II, we focus on applications of neural fields to different problems in visual computing, and beyond (e.g., robotics, audio). Our review shows the breadth of topics already covered in visual computing, both historically and in current incarnations, demonstrating the improved quality, flexibility, and capability brought by neural fields methods. Finally, we present a companion website that contributes a living version of this review that can be continually updated by the community.

1. Introduction

Neural fields are coordinate-based neural networks that parameterize continuous signals across space and time, enabling diverse visual-computing applications. This report addresses rapid field growth by providing a shared formulation, broad literature synthesis, and a community-updatable database.

  • Applications: Neural fields support visual-computing tasks including synthesis, reconstruction, rendering, animation, and pose estimation across scenes and objects.The reviewed scope includes graphics, vision, HCI, and adjacent applications such as robotics.
  • Background: Neural fields use coordinate-based neural networks to represent continuous signals over arbitrary dimensions and resolutions.They parameterize physical quantities such as images, shapes, view-dependent appearance, and human bodies or faces.
  • Report structure: The report organizes common techniques using consistent notation, covering representations, learned priors, differentiable forward maps, architectures, and manipulation methods.Examples of forward maps include surface and volume renderers and partial differential equations.
  • Applications: Part II surveys applications spanning shape and appearance, reconstruction, digital humans, generative modeling, compression, image processing, robotics, medical imaging, audio, and physics-informed problems.This breadth is used to identify commonalities, connections, and trends across application areas.
  • Motivation: Rapid progress has produced over 250 papers in the last two years, while the field still lacks a shared mathematical formulation and risks repeated research.The report also notes that conventional surveys can quickly become outdated.
  • Living database: A companion website provides searchable, filterable, visualized, and exportable access to the reviewed works while allowing community submissions and automatic taxonomy-based updates.Its open-source design is intended to support future living reports and surveys.

Part I. Neural Field Techniques

Neural field methods sample coordinates, predict quantities, map them to a sensor domain, and optimize reconstruction error. The toolbox organizes techniques for conditioning, efficiency, supervision, architecture, and constraints.

  • Neural field algorithms feed space-time coordinates into a network, map predicted quantities to the sensor domain, and optimize reconstruction loss against measurements.The forward map connects the reconstruction domain to available sensor supervision.
  • The neural field toolbox contains five technique classes addressing problems in learning, inference, and control.These classes include prior learning and conditioning, hybrid representations, differentiable forward maps, network architectures, and constraints or regularization.
  • Conditioning: Conditioning introduces latent variables that encode field-specific properties, enabling variation across shape, type, size, color, or other input data.Latent variables can encode instance-specific information while shared information remains in neural field parameters, and semantic or smooth latent spaces can support interpolation or editing.
  • Encoding the Conditioning Variable z: Feed-forward encoders generate z = E(O) in one encoder pass, whereas auto-decoders optimize separate latent codes for observations during training and inference.Auto-decoding is slower at inference but introduces no encoder parameters and makes fewer assumptions about the observation format.
  • Global and Local Conditioning: Global conditioning uses one latent code across all coordinates, while local conditioning assigns coordinate-dependent codes through structures such as grids, patches, or projections.Global codes suit signals with relatively few degrees of freedom; local codes encode information within spatial neighborhoods.

3. Hybrid Representations

Hybrid representations combine neural fields with discrete structures that decompose the coordinate domain, improving scalability while introducing representation-specific trade-offs. They include tiling, embeddings, grids, point clouds, object-centric structures, and meshes.

  • Hybrid representations combine neural fields with discrete data structures that decompose space and help them scale to large signals.The structures include regular or adaptive grids, curves, point clouds, and meshes.
  • Network Tiling: Network tiling assigns separate parameterized neural fields to disjoint regions while sharing the network architecture.Given a coordinate, the data structure retrieves the parameters for its region.
  • Embedding: Embedding stores latent variables z in the data structure and maps the local embedding to neural-field parameters through a function Ψ.Tiling is a special case of embedding in which Ψ is the identity and z is the parameter set Θ.
  • Defining g: The discrete structure g maps coordinates to stored quantities using Dirac-delta basis functions, with α_i representing parameters Θ_i for tiling or latent variables z_i for embedding.Interpolation can extend g between stored coordinates using nearest-neighbor, linear, or cubic schemes.
  • Grid Structures: Regular grids are simple to index but scale poorly in high dimensions, motivating adaptive or sparse grids that focus capacity on complex or high-frequency regions.Grid tiling and embedding grids can model larger-scale signals, use smaller networks, and support faster or interpolated computation, but tiling may overfit and create boundary artifacts.
  • Other Structures: Irregular grids, point clouds, object-centric representations, and meshes provide alternative spatial decompositions with adaptive capacity, interpolation, or established geometric processing.Examples include Voronoi interpolation for point clouds and barycentric interpolation for mesh-vertex embeddings.

4. Forward Maps

Forward maps connect neural-field reconstructions to sensor measurements, enabling inverse problems to be optimized with differentiable programming. The review covers renderers, ray-surface methods, volume rendering, and physics-informed forward modules.

  • Inverse problems recover a neural-field reconstruction from sensor observations when reconstruction and sensor domains differ.The reconstruction field maps world coordinates to quantities, while the sensor field maps sensor coordinates to measurements.
  • Forward maps relate a continuous reconstruction such as a 3D radiance field to measurements such as pixels in a 2D camera image.The review identifies sphere tracing, volume rendering, meshing, Radon and Fourier transforms, and partial derivatives as differentiable forward-map types.
  • A parameter-differentiable forward map permits recovery of the neural field through differentiable programming and optimization such as stochastic gradient descent.The map may depend on additional parameters and be composed with downstream sampling or optimization operators.
  • Rendering: Renderers convert neural-field representations of 3D shape and appearance into images using camera parameters and ray-based queries.Ray queries can return geometric information such as surface normals and intersection depth or aggregate arbitrary features.
  • Ray-surface Intersection: Ray-surface intersection methods recover geometric information from surface fields using ray marching, root refinement, sphere tracing, or related procedures.Sphere tracing has guaranteed convergence for Lipschitz-bounded surfaces, while naive differentiation through iterative algorithms is computationally expensive.
  • Volume Rendering: Volume rendering uses radiative-transfer integrals and, under exponential transmittance, supports efficient cumulative-sum integration with gradients distributed throughout space.Higher sample counts improve accuracy but increase computational cost and memory, motivating coarse-field importance sampling and related methods.
  • Differentiable Rendering: Surface and volume rendering have complementary limitations: surface methods provide gradients only at surfaces, whereas volume methods can be under-constrained and noisy near surfaces.Hybrid approaches soften implicit surfaces into density fields or importance-sample around surface intersections.
  • Physics-informed Neural Networks: Partial differential equations provide forward modules that map network outputs to gradient-space supervision, including physics-informed neural networks for boundary and initial-value constraints.Signed distance functions commonly use this paradigm through the Eikonal equation.

5. Network Architecture

Neural-field architecture choices shape the quality and frequency content of learned fields. The section covers positional encodings, periodic activations, derivatives, and neural integration.

  • 5. Network Architecture: Network structure and component choices provide inductive biases that affect the quality of the parameterized field.The discussion assumes a network with sufficient capacity and focuses on design decisions beyond basic layer and width selection.
  • 5.1. Overcoming Spectral Bias: Positional encoding transforms coordinates into embedded features that help neural fields fit high-frequency signals.Sinusoidal embeddings, also called Fourier feature mappings, became widely used in novel-view synthesis.
  • 5.1. Overcoming Spectral Bias: Encoding frequency controls learned spatial frequencies: lower frequencies can produce blurry reconstructions, while higher frequencies can introduce salt-and-pepper artifacts.A coarse-to-fine schedule that progressively increases high-frequency weights can improve optimization stability.
  • 5.1. Overcoming Spectral Bias: Periodic activations such as sine provide an alternative way to fit high-frequency functions, as demonstrated by SIREN.Activation-function choice also determines derivative behavior, which matters when fields must satisfy differential equations.
  • 5.2. Derivatives and Integrals: Automatic differentiation makes neural-field derivatives readily available, supporting tasks that supervise derivatives through partial differential equations.The activation function must provide nontrivial derivatives up to the PDE order; ReLU has zero high-order derivatives and therefore cannot generally represent such solutions.
  • 5.2. Derivatives and Integrals: Directly parameterizing an antiderivative enables a single-forward-pass integral at test time after fitting its gradient network to measured derivative values.This approach addresses cases where numerical quadrature may require arbitrarily many samples.

6. Manipulating Neural Fields

Neural fields can be manipulated through coordinate transformations, explicit structural controls, temporal conditioning, and direct edits to latent features or network parameters. However, editing and manipulation tools remain limited, restricting use cases.

  • 6. Manipulating Neural Fields: Neural fields have limited editing and manipulation tools compared with established visual-computing data structures.The report identifies this limitation as an active area of research.
  • Coordinate Transformations: Spatial or temporal coordinate transformations provide a simple mechanism for editing neural fields.A rigid spatial translation is expressed as g(x) = f(x+b), while temporal conditioning supports speed changes, offsets, and reversal.
  • Explicit Structural Controls: Explicit shape information such as bounding boxes, coarse geometry, and joint angles offers direct control over object geometry.For long kinematic chains, joint-angle control may introduce spurious correlations; composing local fields can help avoid this issue.
  • Dynamic Scene Editing: Dynamic-scene deformation fields are under-constrained without 3D supervision, motivating regularizers based on physical intuition.The reviewed regularizers include smoothness, sparsity, cycle consistency, and auxiliary image-space losses.
  • Parameter Editing: Latent features and learned network weights can be edited directly for geometry, texture, and other physical quantities.Methods include latent-code interpolation or swapping and test-time fine-tuning against edited observations.

Part II. Applications of Neural Fields

Neural fields span reconstruction, rendering, shape representation, editing, and material estimation across visual-computing applications. The reviewed methods address both 3D-supervised and image-based reconstruction settings.

  • Part II. Applications of Neural Fields: Part II reviews neural fields across 3D reconstruction, novel-view synthesis, human modeling, medical imaging, physics, and engineering.The report organizes applications by domain and identifies connections and trends across them.
  • 7. 3D Scene Reconstruction: 3D-scene reconstruction is important for robotics, autonomous vehicles, games, and visual effects, but observations are often sparse, incomplete, discrete, or lower-dimensional.These conditions make reconstruction an inverse problem from imperfect measurements.
  • 3D Shape Reconstruction: Neural fields represent reconstructed geometry using signed distance or occupancy functions, often with global or local conditioning and learned priors.Examples include DeepSDF, AtlasNet, occupancy networks, voxel-grid local features, and meta-learning approaches.
  • Differentiable Rendering: Differentiable rendering enables reconstruction of shape and appearance from 2D images rather than requiring 3D supervision.This matters because 3D data is expensive to obtain while 2D images are ubiquitous.
  • Differentiable Rendering: NeRF parameterizes a 3D scene by mapping coordinates to radiance and density, which can then be rendered through volume rendering.This representation supports view-synthesis applications reviewed in the report.
  • Material Reconstruction: Material reconstruction estimates surface or participating-media properties from sparse measurements, but remains difficult because materials create complex light-transport effects.For opaque surfaces, the target may be BSDF parameters; for participating media, it may be phase functions.

8. Digital Humans

Neural fields have become a major approach for modeling digital humans, including faces, bodies, hands, clothing, geometry, appearance, and deformation. Their expressiveness supports high-quality reconstruction and synthesis while limited observations and per-subject training remain challenges.

  • 8. Digital Humans: Neural fields have produced high-quality synthesis and reconstruction of human faces, bodies, and hands, making digital humans a prominent application area.The state of the art continues to evolve quickly.
  • Face Modeling: Implicit surfaces such as SDFs and radiance fields address realism and topology limitations of explicit morphable models, including for hair and teeth.i3DMM predicts SDF and color and disentangles identity, facial expressions, hairstyle, and color for editing.
  • Face Modeling: Radiance and deformation fields support photorealistic human-head view synthesis from casual handheld video, including large topological changes through auxiliary dimensions.Nerfies and HyperNeRF exemplify this direction.
  • Digital-Human Challenges: Digital-human methods often require per-subject training, and reconstruction from limited observations remains challenging.These constraints are stated for semantic and dynamic radiance-field methods.
  • Body and Hand Modeling: Clothed-human reconstruction benefits from pixel-aligned local embeddings because substantial shape and appearance variation makes a global latent embedding implausible.Neural fields are applied to image and point-cloud inputs for clothed-body reconstruction.
  • Body and Hand Modeling: Human-body methods use parametric controls, occupancy fields, warp fields, and per-body-part representations to model articulation and non-rigid deformation.These representations support pose-driven modeling and can reduce issues associated with directly handling long kinematic chains.

9. Generative Modeling

Neural fields support generative modeling by representing samples as continuously queryable functions conditioned on latent variables. The review covers image, 3D shape and appearance, multimodal, and image-processing applications.

  • Generative modeling: Neural fields represent each sample as a function that can be densely queried across coordinates and conditioned globally or locally by latent variables.Their continuous outputs can be sampled at arbitrary resolutions, and coordinate transforms can shift input samples.
  • Generative modeling of images: Image generators map 2D pixel coordinates to RGB colors and use approaches including meta-learning, global conditioning, and hypernetworks.The reviewed methods include SIREN-based variational inference and INR-GAN with a hypernetwork predicting a low-rank weight decomposition.
  • Generative modeling of 3D shape: 3D shape generators parameterize occupancy or signed distance fields and learn distributions using latent concatenation, variational autoencoders, or directly optimized latent codes.IM-Net, Occupancy Networks, and DeepSDF exemplify these alternatives.
  • Generative modeling of 3D shape and appearance: 3D shape-and-appearance models combine neural fields with forward rendering to generate images from camera parameters.GRAF uses a neural radiance field and volume rendering, while later methods vary activations, conditioning, and rendering acceleration.
  • Other applications: Neural fields are also used for image-to-image translation, continuous and dynamic image reconstruction, patch-based image processing, and multimodal manifolds.Applications include super-resolution, denoising, inpainting, time and illumination interpolation, and audio-visual modeling.

11. Robotics

Neural fields are applied in robotics to perception, localization, planning, grasping, manipulation, and control. The reviewed methods address camera estimation, scene reconstruction, spatial planning, and learned geometric constraints.

  • Robotics: Neural fields extend to robotics because many robotic problems use 3D reconstruction for perception, navigation, planning, and control.The review discusses neural fields for robot perception, planning, and control.
  • Localization via camera parameter estimation: Camera rotation and pose require alternative parameterizations because rotation matrices lie on SO(3), where continuity is not guaranteed.Reviewed choices include exponential coordinates, Rodrigues formula, continuous 6D representations, Euler angles, and homography warps.
  • Localization via camera parameter estimation: Joint scene reconstruction and camera registration is a chicken-and-egg problem because each requires the other.Some methods assume a known reconstruction or jointly optimize scene and camera parameters; other approaches relax forward-facing initialization assumptions for inward-facing 360° scenes.
  • Planning and manipulation: Neural fields represent spatial planning problems including navigation, grasping, manipulation, and interactive perception.Examples include Bellman-equation path planning, affordance maps, learned grasp points, and locally conditioned grasp-quality fields.
  • Control: Control methods use neural fields to learn obstacle barrier functions or predict distances between robot links for collision avoidance.Signed distance functions provide one representation for obstacle barriers.

12. Lossy Compression

Neural fields offer a continuous alternative for lossy compression by storing signals as network parameters or latent-conditioned decoders. The review describes geometry and volumetric-data compression while noting that comparisons remain difficult.

  • Lossy compression: Lossy compression balances rate and distortion, while prior-based schemes decode low-dimensional latent codes into neural-field parameters.The decoder's storage cost can be amortized across many latent codes.
  • 3D geometry compression: Neural fields can encode 3D geometry with potentially reduced memory compared with conventional meshes.SIREN with scalar weight quantization compressed dense volumetric data better than state-of-the-art approaches, at the cost of higher computational requirements.
  • Limitations: Compression comparisons are difficult because studies use different signal modalities, datasets, architectures, and metrics.Few rigorous comparisons with conventional schemes exist, and many works lack design ablations or direct queries without decoding.

13. Beyond Visual Computing

Beyond visual computing, neural fields parameterize inverse problems involving medical measurements, audio, sonar, and physics-based systems. These applications use domain-specific forward maps or PDE constraints to connect continuous fields with observations and laws of physics.

  • Alternative signal modalities: Neural fields address inverse problems beyond visual computing that involve incomplete observations and flexible continuous parameterizations.The review highlights non-line-of-sight, non-visible, medical, audio, time-of-flight, and volumetric-light signals.
  • Medical imaging: Medical imaging reconstructs spatial density from CT or MRI measurements, which are Radon or Fourier transforms and are not directly human-readable.The reconstruction is ill-posed, and classical techniques can be sensitive to measurement noise.
  • Medical imaging: Medical neural fields either model the sensor domain or directly predict density values, with outputs connected to measurements through filtered back-projection, Radon, or Fourier transforms.Direct density-field methods supervise predictions by mapping them into the sensor domain.
  • Audio and sonar: Neural fields represent audio as continuous waveform or spectrogram functions and model sonar scatter distributions through a convolutional forward map.Waveforms map temporal coordinates to amplitude, while spectrograms map time and frequency to amplitude.
  • Physics-informed problems: Physics-informed neural fields parameterize solutions constrained by partial differential equations and optimize them using physical equations or gradient-based regularization.This reframes some PDE problems as optimization rather than simulation and can avoid the prohibitive computation of arbitrarily small traditional step sizes.

Discussion & Conclusion

Neural fields have developed a broad technical toolbox and expanding applications, while important challenges remain in generalization, evaluation, high-level semantics, multimodal learning, societal impact, and research culture.

  • Progress drivers: Neural fields use continuous-field parameterization, positional encoding, sinusoidal activations, and differentiable rendering to support progress across applications.Differentiable volume and voxel rendering extend beyond novel-view synthesis to tasks such as 3D reconstruction and semantic segmentation.
  • Future techniques: Generalization to unseen data remains a common limitation, motivating stronger priors, task-specific heuristics, physical laws, and architectural inductive biases.The report identifies stronger priors as a route toward better generalization and data efficiency.
  • Evaluation: Rapid progress has outpaced methodical evaluation, creating a need for shared datasets and benchmarks to compare neural-field techniques.The authors connect this need to the rapid growth of the field.
  • Application frontiers: Most applications address low- and mid-level tasks, leaving high-level semantic problems, multimodal fusion, and weakly or self-supervised learning open for further work.Examples include scene layout, scene interaction, meaningful grouping, language-conditioned fields, and joint image-text or audio modeling.
  • Research culture: The neural-fields community should promote scholarship, inclusivity, diversity, sustainable growth, and attention to researchers’ mental health while avoiding repetition and bandwagon effects.The authors specifically call for acknowledging limitations and collaborating with domain experts.
  • Societal impact: Neural fields have both positive and negative societal implications, including democratized content creation, robotic automation, deception, privacy risks, surveillance, and environmental costs.The discussion highlights realistic impersonation, possible de-anonymization, reduced surveillance costs, and GPU-related resource use.

15. Conclusion

The report synthesizes neural-fields research into a shared framework spanning techniques and applications. It also establishes a community-driven living report to support continued classification and understanding of this growing area.

  • Conclusion: The report reviews over 250 papers and organizes neural-field techniques into five classes using a shared mathematical formulation.The classes include prior learning and conditioning, hybrid representations, forward maps, network architectures, and manipulation methods.
  • Conclusion: The survey covers applications across graphics, vision, robotics, medical imaging, and computational physics.This application scope extends neural fields beyond core visual-computing problems.
  • Conclusion: A community-driven website lets authors submit papers and classify them using the report’s taxonomy, supporting a living version of the survey.The website is intended to aid continued understanding as neural-fields research grows.

Appendix A: Variable Naming Conventions

The appendix defines notation for neural fields and summarizes field quantities used to represent geometry and related physical properties. It distinguishes hard-surface, volumetric, and thickness-based representations.

  • Variable naming conventions: A neural field maps coordinates x ∈ X to field quantities y ∈ Y using parameters Θ, with notation also covering encoders, positional encoding, hypernetworks, and latent variables.The conventions include input and output dimensionalities m and n, observations, domains, surfaces, view directions, and the plenoptic function.
  • Field outputs: Neural fields can represent geometry, radiance, BRDF parameters, deformation or warping parameters, and classification weights.The appendix focuses specifically on implicit surface representations while noting the broader range of output types.
  • Implicit surfaces: Distance functions encode distance to the nearest surface, while signed distance functions use the sign to indicate whether a point is inside or outside.These representations support applications including path planning, fabrication, occlusion approximation, and sphere-tracing visualization.
  • Implicit surfaces: Occupancy fields represent inside-versus-outside status, typically approximating binary values continuously and extracting an isosurface at b ∈ [0,1].Occupancy fields can be visualized using raymarching.
  • Volumetric representations: Volume density represents continuous particle density and is suited to volumetric scenes such as clouds, fog, and hair rather than only hard surfaces.Unlike occupancy, volume density is not binary and has no upper bound.
  • Geometric quantities: Medial fields represent local geometric thickness derived from the medial axis and can support rendering and ambient-occlusion approximation.Their use is described as similar to signed distance functions for these purposes.
Loading 2111.11426v4…