PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLU Fields: The Little Non-linearity That Could

Animesh Karnewar, Tobias Ritschel, Oliver Wang, Niloy J. Mitra

arXiv:2205.10824v2cs.CVcs.GR

TL;DR

MLPs model complex signals with high fidelity and low memory use but are slow to train and evaluate, while regular grids are fast but need substantial memory for high-frequency content. ReLU Fields add a fixed ReLU after grid interpolation, without learned parameters, and perform comparably with state-of-the-art approaches in a fraction of the time. The method has a per-cell boundary: it cannot model more than one discontinuity per grid cell.

  • Problem

    MLPs are slow to train and evaluate, while regular grids require significant memory to represent high-frequency content.

    Source: §1 · p. 1

  • Method

    ReLU Fields store unbounded grid-vertex values, interpolate them, and apply a single ReLU without a neural-network decoder.

    Source: It’s Just a Little ReLU · p. 3

  • Results

    ReLU Fields perform comparably with state-of-the-art approaches in a fraction of the time.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    A single ReLU after interpolation can strictly improve grid-based representational capability with virtually no added computational cost or complexity.

    Source: §5.2 · p. 7

    ReLU Fields cannot model more than one discontinuity per grid cell.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

In many recent works, multi-layer perceptions (MLPs) have been shown to be suitable for modeling complex spatially-varying functions including images and 3D scenes. Although the MLPs are able to represent complex scenes with unprecedented quality and memory footprint, this expressive power of the MLPs, however, comes at the cost of long training and inference times. On the other hand, bilinear/trilinear interpolation on regular grid based representations can give fast training and inference times, but cannot match the quality of MLPs without requiring significant additional memory. Hence, in this work, we investigate what is the smallest change to grid-based representations that allows for retaining the high fidelity result of MLPs while enabling fast reconstruction and rendering times. We introduce a surprisingly simple change that achieves this task -- simply allowing a fixed non-linearity (ReLU) on interpolated grid values. When combined with coarse to-fine optimization, we show that such an approach becomes competitive with the state-of-the-art. We report results on radiance fields, and occupancy fields, and compare against multiple existing alternatives. Code and data for the paper are available at https://geometry.cs.ucl.ac.uk/projects/2022/relu_fields.

1 Introduction

MLPs represent complex signals with high fidelity and low memory use, but are slow to train and evaluate, while grids are fast but memory-intensive for high-frequency content. ReLU Fields add a ReLU after grid interpolation, without learned parameters, and are applied to image-based scene reconstruction and implicit geometry modeling.

1 Introduction

  • MLPs represent complex signals with high fidelity and low memory use, but are slow to train and evaluate; regular grids are fast but memory-intensive for high-frequency content.Hybrid representations combine fast spatial data structures with coordinate-based MLP decoders.Source: §1 · p. 1
  • ReLU Fields add a fixed ReLU after interpolating grid values, without additional learned parameters, and optimize the representation progressively.The paper presents this as a minimal change to regular grid-based models.Source: §1 · p. 2
  • The paper validates ReLU Fields in image-based 3D scene reconstruction and implicit modeling of 3D geometries.The representation is described as directly differentiable and fast to optimize and evaluate, without neural networks.Source: §1 · p. 2

2 Related Work

Related work spans grid-based signal representations, learned neural representations, and hybrids that pair spatial data structures with MLP decoders. ReLU Fields are positioned as a minimal way to increase grid capacity for signals with sharp discontinuities, with compatibility with existing sparse structures and legacy software noted as distinctions.

2 Related Work

  • Image representations have added explicit edges, curves, multiple colors, or discontinuity information to pixels; ReLU Fields instead encode discontinuities implicitly through grid values and a ReLU.The cited approaches include zero-crossing boundaries, curves, and deformed grids.Source: §2 · p. 2
  • Coordinate-based MLPs improve representation or reconstruction quality, but incur high evaluation costs and are poorly suited to post-training manipulation.Post-hoc grid fitting can speed execution but cannot reconstruct scenes from scratch [Garbin et al. 2021; Hedman et al. 2021; Reiser et al. 2021; Yu et al. 2021b].Source: §2 · p. 2
  • Hybrid methods store learned features in regular grids, sparse voxels, point clouds, local grid crops, or triplanes, then decode interpolated features with an MLP.Examples include regular grids [Nguyen-Phuoc et al. 2020], sparse voxels, point clouds, local crops, and triplanes.Source: §2 · pp. 2–3
  • Concurrent approaches include DVGo’s post-activated density grid, Plenoxels’ sparse ReLU grid, and NGP’s hashed learned features with an MLP decoder.DVGo also uses a learned feature grid and MLP for view-dependent appearance; NGP’s reconstruction speed is attributed to its engineered GPU implementation.Source: §2 · p. 3
  • ReLU Fields target increased grid capacity for sharp discontinuities, remain compatible with Plenoxels and NGP sparse structures, and can load in legacy software that supports signed data and transfer functions.The authors distinguish their focus from concurrent efforts centered on 3D scene reconstruction.Source: §2 · p. 3

3 Method

ReLU Fields store unbounded feature vectors at grid vertices, interpolate them, and apply a single ReLU rather than a neural-network decoder. The vertex values can shift the ReLU clipping plane to align with discontinuities, although the image example indicates that the representation is suited to sparse discontinuities rather than natural images.

3 Method

  • A regular 3D grid stores an n-dimensional vector at each vertex, and these unbounded values are features rather than physically interpreted signal values.Signal values may have explicit ranges such as [0, 1] or [0, +∞).Source: It’s Just a Little ReLU · p. 3
  • Within a grid cell, a ReLU Field can match a sharp c1-discontinuity that a linear basis cannot capture.The illustrated construction chooses two values whose interpolation, after clamping, matches the discontinuity.Source: Figure 3 · p. 3, It’s Just a Little ReLU · p. 3
  • After m-linear interpolation, a single ReLU clips negative feature values to zero; no MLP or other neural network interprets the features.During optimization, vertex values can shift so the clipping plane aligns with ground-truth c1-discontinuities.Source: It’s Just a Little ReLU · p. 3
  • For images, ReLU Fields represent sharp edges at higher fidelity than bilinear interpolation at the same grid resolution, but natural images do not benefit from this representation.The stated useful scope is signals with sparse c1-discontinuities, such as flat-shaded images and volumetric density.Source: It’s Just a Little ReLU · pp. 3–4

4 Applications

The applications test ReLU Fields for radiance-field scene reconstruction and occupancy-based 3D geometry reconstruction. They describe each task’s representation and report qualitative or comparative results.

4.1 Radiance Fields

  • Radiance reconstruction takes posed images as input and optimizes a grid scene representation to match rendered images to the inputs.The grid stores pre-ReLU density values for geometry and second-degree spherical-harmonic coefficients for view-dependent appearance; ReLU applies only to density.Source: §4.1 · p. 4
  • ReLU Fields achieve performance comparable to NeRF-PT while reconstructing and rendering much faster; traditional Grid and GridL perform worst.Table 1 reports PSNR and LPIPS on synthetic scenes.Source: §4.1 · p. 4, Table 1 · p. 6
  • Progressive growing doubles grid resolution in stages, and ablations indicate it is critical for good reconstruction; longer optimization gives only minor improvement.ReLUFieldL outperforms RFLong, suggesting grid resolution matters more than optimization time in this comparison.Source: §4.1 · p. 4
  • On the real-captured Flowers scene, spherical background shells accompany a foreground ReLU Field, and after one hour the field represents flower details while the background is modeled reasonably well.The background grid is bilinear without ReLU, and the extension uses ERP rather than ODS for mapping the image plane to background shells.Source: §4.1 · p. 4, Figure 5 · p. 6

4.2 Occupancy Fields

  • For occupancy reconstruction, the grid predicts occupancy probabilities from 3D points supervised with ground-truth inside/outside labels and binary cross-entropy.The grid is bounded with tanh, though other bounding functions or tensor normalizations can be used.Source: §4.2 · p. 6
  • For occupancy fields, ReLUField yields higher-quality reconstructions than Grid and coordinate-based MLP, while training takes 15 minutes versus 1.5 hours for MLP.The comparison covers ThaiStatue, Lucy, Bimba, Grog, Lion, Ramses, and Dragon; Table 2 reports Volumetric-IoU.Source: §4.2 · pp. 6–7, Table 2 · p. 6

5 Discussion

ReLU Fields remain memory-intensive, and each grid cell can model only one discontinuity. The authors conclude that a single ReLU after interpolation yields a simple representation that performs comparably with state-of-the-art approaches in a fraction of the time.

5.1 Limitations

  • 128^3 radiance-field ReLU Fields use 260Mb, while the 256^3 version requires 2.0 Gb of storage.Source: §5.1 · p. 7
  • Each grid cell can model only one crease; learned features represent more complex signals but incur high compute costs.Source: §5.1 · p. 7
  • For a given task, the optimal representation may combine learned features, neural networks, and discrete signal representations.Source: §5.1 · p. 7

5.2 Conclusion

  • A single ReLU after linear interpolation gives a simple grid representation that performs comparably with state-of-the-art approaches in a fraction of the time.Source: §5.2 · p. 7

Related papers

Loading 2205.10824v2…