PaperlayerView exact revision on arXiv

Source-linked AI summary

CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture

Ruofan Liang, Hongyi Sun, Nandita Vijaykumar

arXiv:2201.12425v1cs.CVcs.LG

TL;DR

Coordinate-based MLPs require substantial computation for training and inference over large sets of signal coordinates. CoordX splits early MLP processing across coordinate dimensions and fuses the features; the paper reports faster training and inference across signal-fitting tasks while retaining similar accuracy to baseline models.

  • Problem

    Training and inference with coordinate-based MLPs are computationally demanding because they process large numbers of coordinates.

    Source: §1 · p. 2

  • Method

    CoordX processes input-coordinate dimensions in separate branches, shares later layers, and fuses branch features to predict the signal.

    Source: §1 · p. 2

  • Results

    ∼2.5x speedup is reported for 5-layer CoordX models across signal-fitting tasks, while retaining accuracy close to baseline architectures.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The dimension-splitting approach is presented for a wide range of coordinate-based MLP tasks and can be combined with sparsity-based optimization without much additional memory overhead.

    Source: §1 · p. 2

    CoordX does not improve performance when fitting unidimensional signals such as audio.

    Source: §1 · p. 2

Abstract

from arXiv · show

Implicit neural representations with multi-layer perceptrons (MLPs) have recently gained prominence for a wide variety of tasks such as novel view synthesis and 3D object representation and rendering. However, a significant challenge with these representations is that both training and inference with an MLP over a large number of input coordinates to learn and represent an image, video, or 3D object, require large amounts of computation and incur long processing times. In this work, we aim to accelerate inference and training of coordinate-based MLPs for implicit neural representations by proposing a new split MLP architecture, CoordX. With CoordX, the initial layers are split to learn each dimension of the input coordinates separately. The intermediate features are then fused by the last layers to generate the learned signal at the corresponding coordinate point. This significantly reduces the amount of computation required and leads to large speedups in training and inference, while achieving similar accuracy as the baseline MLP. This approach thus aims at first learning functions that are a decomposition of the original signal and then fusing them to generate the learned signal. Our proposed architecture can be generally used for many implicit neural representation tasks with no additional memory overheads. We demonstrate a speedup of up to 2.92x compared to the baseline model for image, video, and 3D shape representation and rendering tasks.

1 INTRODUCTION

Coordinate-based MLPs represent signals by mapping input coordinates to predicted values, but processing large numbers of coordinates makes training and inference computationally demanding. CoordX splits processing across coordinate dimensions and fuses the resulting features, aiming to speed both operations while retaining similar accuracy.

1 INTRODUCTION

  • CoordX targets the high computation demand of training and inference, which arises because coordinate-based MLPs process many input points.The introduction cites millions of MLP queries for rendering tasks and extensive training passes over signal samples.Source: §1 · pp. 1–2
  • Prior acceleration approaches exploit spatial sparsity, but may be task-specific, add memory overhead, or work less effectively on dense, feature-rich signals.The paper gives voxel-based representations as one example of reducing computation by operating on a discrete set of points.Source: §1 · p. 2
  • ∼2.5x speedup is reported for 5-layer CoordX models across signal-fitting tasks, with 2.35x inference speedup for a 1k-resolution image and 2.78x for a 1k-resolution video clip.The reported speedups are compared with baseline coordMLPs.Source: §1 · p. 2
  • CoordX splits initial processing across input dimensions, shares subsequent fully connected layers, and fuses branch features with an outer product before final prediction.The approach reduces the image inputs processed by the MLP from H × W to H + W while leveraging locality between points with similar coordinate values.Source: §1 · p. 2
  • CoordX does not improve performance when fitting unidimensional signals such as audio.Source: §1 · p. 2
  • The architecture is presented for image, video, and 3D shape fitting, as well as applications such as image super-resolution, without much additional memory overhead.The authors also describe the splitting technique as orthogonal to sparsity-based optimization methods.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work both expands the representational capacity of coordinate-based MLPs and accelerates them by reducing queried points or per-point processing. CoordX uses a different coordinate-splitting strategy that the paper describes as orthogonal to ACORN’s block-coordinate approach.

2 RELATED WORK

  • Coordinate-based MLPs have been used for images, occupancy, signed distance, texture, and radiance fields; positional encoding and sinusoidal or Fourier features address high-frequency variation.Source: §2 · p. 3
  • Existing acceleration methods either reduce queried points, often by exploiting spatial sparsity, or reduce per-point processing through smaller MLPs or precomputed results.Examples include depth-guided sampling, octrees, regional MLPs, and stored rendering results.Source: §2 · p. 3
  • ACORN hierarchically decomposes coordinate grids into patches, whereas CoordX splits coordinates by dimension; the paper says the approaches are orthogonal and can be combined.Source: §2 · p. 3
  • Figure 1 contrasts the baseline, which independently feeds N = HW image coordinates to an MLP, with CoordX branches that process decomposed coordinate dimensions before feature fusion.Source: Figure 1 · p. 4

3 METHOD

CoordX splits coordinate inputs into branches, shares layers before fusion, and combines branch features to predict the signal. For training, it samples along decomposed dimensions to construct complete coordinate combinations; a fully split variant represents an image through learned matrices.

3.1 NEURAL NETWORK ARCHITECTURE

  • CoordX splits the first layer across coordinate branches, shares subsequent pre-fusion layers, and fuses intermediate features before the final Df layers generate predictions.The shared layers avoid adding parameters, while the outer product is the fusion operation.Source: §3.1 · pp. 4–5
  • 2.5x is the theoretical speedup upper bound for a 5-layer CoordX MLP with Ds = 3 and Df = 2 under the stated computation assumptions.The bound is approached when B is large enough that the specified term becomes negligible.Source: §3.1 · p. 5

3.2 ACCELERATING TRAINING WITH COORDX

  • Training samples positions along each decomposed dimension, then combines them so the signal values needed for fusion are available across coordinate combinations.Randomly sampled points may omit combinations required to split coordinates, so the method constructs combinations from sampled positions.Source: §3.2 · p. 5

3.3 THE SIGNAL DECOMPOSITION OF COORDX.

  • The fully split variant moves fusion to the end, where outer products reconstruct the signal; for images, two learned feature matrices can represent the image through matrix multiplication.The general fusion reshapes branch features to R × S; the fully split image example uses R = 256 and S = 1.Source: §3.3 · pp. 5–6, Figure 2 · p. 6

4 EXPERIMENTAL EVALUATION

The evaluation tests CoordX across image, video, 3D shape, NeRF, and latent-code LIIF workloads against baseline coordinate MLPs. Results generally show faster processing with retained or improved representation quality, alongside task-specific accuracy tradeoffs.

4 EXPERIMENTAL EVALUATION

  • Experiments generally use 5-layer MLPs, with CoordX models having two fully connected layers after fusion and corresponding baseline hyperparameters.Models were implemented in PyTorch and evaluated on an NVIDIA RTX3090 GPU.Source: §4 · p. 6

4.1 SIGNAL FITTING WITH COORDX

  • Image inference is up to 2.83x faster at 2K despite a ∼3 dB PSNR drop; XY+T-A video training is 2.02X faster and inference 2.78x faster with high quality.Video inference speedups are higher at higher resolutions.Source: Figure 3 · p. 7, §4.1 · p. 7
  • 3D shape fitting retains similar IoU, with training up to 1.7x faster and inference up to 2.81x faster at 256^3 (2.55x average).Marching-cubes inference was tested at different grid resolutions.Source: Figure 3 · p. 7, §4.1 · p. 7
  • Grid-interpolation CoordX accelerates baseline NeRF training 1.81x and rendering 4.41x, with under 0.5 dB quality drop on low-resolution Blender scenes.Because NeRF has an irregular input coordinate grid, the method uses precomputed dense coordinates and grid interpolation over MLP predictions.Source: §4.1 · p. 8

4.2 COORDX WITH LATENT CODES

  • CoordX-LIIF maintains almost the same PSNR as baseline LIIF and reaches up to 2.88x inference speedup for target 2K super-resolution, but gives no significant training speedup.The MLP is only part of the overall training pipeline.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 THE DEGREE OF SPLITTING FOR MLP

  • Lower Df, meaning more split layers, slightly reduces IoU and trades representation accuracy for speed; simply using fewer baseline MLP layers did not yield similar accuracy.At Df = 5, CoordX is the baseline coordMLP.Source: §4.3 · p. 8, Table 4 · p. 9

4.4 IMPROVING REPRESENTATION QUALITY

  • Increasing R improves image PSNR, with augmented CoordX exceeding baseline PSNR while remaining faster; R++ helps more at fixed R, but raising R helps more than adding R++ parameters.The compared configurations keep fused feature size S at 256 for image models.Source: §4.4 · p. 9

5 CONCLUSION AND FUTURE WORK

CoordX splits positional coordinates across MLP branches, then fuses branch features to produce the learned signal. The conclusion also identifies further applications as directions for future exploration.

5 CONCLUSION AND FUTURE WORK

  • CoordX decomposes positional or temporal coordinates by input dimension and uses a multi-branch MLP whose features are fused to produce the signal.The authors report faster inference and training while retaining similar representation accuracy to baseline models.Source: §5 · p. 9
  • Potential extensions include accelerating dynamic- and deformable-scene view synthesis, implicit-representation compression, and multidimensional or multivariate function decomposition.The authors leave these applications for future work.Source: §5 · p. 9

A INPUT POINT SAMPLING FOR COORDX TRAINING

CoordX training requires points that can be decomposed along each coordinate dimension, so the paper introduces a sampling strategy that selects combinations of split inputs. The strategy preserves point or region selection probabilities while adding randomness to the sampled coordinates.

A INPUT POINT SAMPLING FOR COORDX TRAINING

  • Randomly sampled training coordinates cannot be decomposed along each dimension, so they do not provide CoordX speedups.The paper therefore requires a sampling strategy that selects combinations of the split input vectors.Source: A INPUT POINT SAMPLING FOR COORDX… · p. 13
  • The strategy assumes approximately μS_i sampled positions along dimension i, with μ chosen so the new and original methods use approximately the same number of points.The number of sampled positions is assumed proportional to each dimension's length S_i.Source: A INPUT POINT SAMPLING FOR COORDX… · p. 13
  • The sampling method preserves the probability of choosing a point or region and adds noise to μS_i to introduce more randomness.It applies to both continuous and discrete coordinate spaces.Source: A INPUT POINT SAMPLING FOR COORDX… · p. 13
  • Selecting combinations of split coordinate inputs ensures that the fused input points are included for training; Figure 4 illustrates the strategy in 2-D.Its samples have matching proportions along the coordinate dimensions.Source: A INPUT POINT SAMPLING FOR COORDX… · p. 13, Figure 4 · p. 13

B ADDITIONAL TASK DETAILS

The appendix gives additional signal-fitting details, including the image-fitting setup. In that task, a coordinate-based model predicts pixel color, using a continuous image representation that can be queried at arbitrary resolutions.

B ADDITIONAL TASK DETAILS

B.1 IMAGE FITTING

  • Image fitting maps 2-D pixel coordinates to color values, allowing the continuous representation to describe images at arbitrary resolutions.The output dimension is three for RGB images and one for grayscale images.Source: §B.1 · pp. 13–14
  • The image experiment fits twelve center-cropped DIV2K images at 512 × 512 resolution for 20,000 epochs.Source: §B.1 · p. 14
  • With positional encoding, normalized coordinates are mapped using sinusoidal functions and concatenated with the original input; training samples 512 coordinates from each image axis per batch.The encoding frequency follows the image-fitting settings of Tancik et al. (2020).Source: §B.1 · p. 14
  • Figure 5 presents qualitative fitting results for four DIV2K images of varying complexity.Source: §B.1 · p. 14

B.2 VIDEO FITTING

The appendix describes video fitting with spatial and temporal coordinate inputs, then details 3-D shape fitting with occupancy labels. It specifies the splitting strategies, training setups, and qualitative comparisons for both tasks.

B.2 VIDEO FITTING

  • Video fitting uses (x, y, t) inputs and compares splitting all three coordinates (X+Y+T) with splitting spatial coordinates from time (XY+T).The model predicts pixel color and is supervised with mean squared error.Source: §B.2 · p. 14
  • The video experiment fits two 10-second clips using five-layer coordMLPs with 1,024 hidden units for 100,000 epochs.For XY+T sampling, each batch samples 3.8% of total video points after choosing frame values; Figure 6 and Figure 7 show frames for comparison.Source: §B.2 · p. 14, Figure 6 · p. 16, Figure 7 · p. 16
  • The video results include a cat clip and a bike clip, with several frames shown for qualitative comparison.Source: §B.2 · p. 14

B.3 3D MODEL FITTING

  • 3-D shapes are represented by occupancy probabilities at queried coordinates; the experiment splits all three input dimensions and trains on four scanned objects for 10,000 epochs.The model uses five-layer coordMLPs with 256 hidden units and cross-entropy loss for occupancy classification.Source: §B.3 · p. 15
  • For the SIREN test, mesh points are recentered to [−1, 1], and sampled points with their occupancy labels are stored to skip a time-consuming sampling step.Source: §B.3 · p. 15

B.4 FEATURE FUSION

Multiplicative modulation performs best among the compared fusion strategies. For volume rendering, interpolation addresses the non-axis-aligned sampled coordinates used along camera rays.

B.4 FEATURE FUSION

  • Multiplicative modulation performs best; additive and multiplicative use the same parameter count, while concatenation needs more parameters and takes ∼8% longer to train than multiplicative modulation.The comparison uses image-fitting tasks trained for 10,000 epochs.Source: Table 6 · p. 17, §B.4 · p. 17

C VOLUME RENDERING WITH COORDX

  • Interpolation transforms sampled points that are not axis-aligned into axis-aligned grids, enabling CoordX to be used for volume rendering.Ray samples generally do not share coordinate values because camera rays are not usually aligned with the x, y, or z axes.Source: C VOLUME RENDERING WITH COORDX · p. 17

C.1 VOLUME RENDERING

  • NeRF estimates each ray’s color by densely sampling 3D points along the camera ray, predicting color and volume density at each point, and accumulating them to render the image.Hierarchical sampling first estimates rough depth with coarse samples, then samples more finely around the object surface.Source: §C.1 · p. 18

C.2 RAY SAMPLING WITH A SPLIT MLP ARCHITECTURE

CoordX uses interpolation to handle ray samples that cannot be decomposed along coordinate dimensions. The method accelerates grid pre-computation and also adapts feature interpolation for training.

C.2 RAY SAMPLING WITH A SPLIT MLP ARCHITECTURE

  • Because ray samples are not axis-aligned, the method pre-computes a 3D occupancy grid and trilinearly interpolates each sample from its voxel’s eight endpoints before volumetric rendering.Accumulated color and depth are then obtained using the original volumetric rendering method.Source: §C.2 · p. 19
  • 2.8x speedup is demonstrated with CoordX for the grid pre-computation step, the most time-consuming step in the processing pipeline.The pre-computed grid’s points can be decomposed into three vectors for querying CoordX.Source: §C.2 · p. 19
  • For training, CoordX interpolates split features before fusion rather than training on every grid point, balancing grid reuse with computation efficiency.Bilinear interpolation is performed directly on the split features for each query point.Source: §C.2 · p. 19

C.3 VISUALIZATION OF LEARNED 3D OCCUPANCY.

Grid interpolation rendering accelerates rendering compared with baseline volumetric rendering, and CoordX is faster than the baseline MLP under grid interpolation. The comparison measures time to render one image, including grid pre-computation for the grid method.

C.3 VISUALIZATION OF LEARNED 3D OCCUPANCY.

  • The comparison renders 512 × 512 images, samples 256 points per ray at each of two hierarchical granularity levels, and uses a grid resolution of 384 along the object’s longest bounding-box dimension.Rendering time includes grid pre-computation for the grid interpolation method.Source: §C.3 · p. 19
  • 2.15x speedup over the baseline MLP is achieved with grid interpolation rendering; versus baseline volumetric rendering, grid interpolation is 2.70x for baseline and 14.83x for CoordX.The rendered images are described as high-quality for different MLP models.Source: Table 7 · p. 19, §C.3 · p. 19

C.4 NOVEL VIEW SYNTHESIS

For novel-view synthesis, CoordX splits the NeRF model’s positional coordinates into three branches. It achieves faster training while retaining similar reported quality to baseline NeRF.

C.4 NOVEL VIEW SYNTHESIS

  • The CoordX NeRF model splits positional coordinates (x, y, z) into three branches and keeps two fully connected layers after fusion.The model uses a feature grid with resolution 384; the study trains simplified eight-layer NeRF models for Lego, ship, and drums scenes.Source: §C.4 · pp. 19–20
  • 1.81x training speedup over baseline NeRF accompanies similar PSRN scores with less than 0.5 dB drop; grid interpolation rendering has less than 0.5 dB drop for both models versus original rendering.The speeds reported in Table 8 are tested on the Lego scene.Source: Table 8 · p. 20, §C.4 · p. 20
  • The authors conclude that grid interpolation renders images with comparable quality and significant speedups, while enabling CoordX training and inference acceleration over the baseline.Source: §C.4 · p. 20
Loading 2201.12425v1…