PaperlayerView exact revision on arXiv

Source-linked AI summary

Random Weight Factorization Improves the Training of Continuous Neural Representations

Sifan Wang, Hanwen Wang, Jacob H. Seidman, Paris Perdikaris

arXiv:2210.01274v2cs.LG

TL;DR

Coordinate-based MLPs can struggle to learn high-frequency detail and fine features in continuous neural representations. This paper introduces random weight factorization, a drop-in weight reparameterization with a self-adaptive learning-rate effect; it reports consistent improvements across six benchmarks spanning computer vision, graphics, and scientific computing.

  • Problem

    Coordinate-based MLPs suffer from spectral bias, making high-frequency components and fine details difficult to learn, while some approaches require carefully tuned hyperparameters.

    Source: §1 · pp. 1–2

  • Method

    Random weight factorization reparameterizes each neuron’s weights with a trainable scale and vector, altering the loss landscape and effectively assigning each neuron a self-adaptive learning rate.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Across six benchmarks from image regression to learning operators, random weight factorization shows consistent, robust improvements across computer vision, graphics, and scientific computing.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors present random weight factorization as a simple reparameterization that can mitigate spectral bias and help networks reach better local optima farther from initialization.

    Source: §5 · p. 9

    The reported cavity-flow setup assumes Reynolds number Re = 1,000, limiting this evidence to that stated setting.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Continuous neural representations have recently emerged as a powerful and flexible alternative to classical discretized representations of signals. However, training them to capture fine details in multi-scale signals is difficult and computationally expensive. Here we propose random weight factorization as a simple drop-in replacement for parameterizing and initializing conventional linear layers in coordinate-based multi-layer perceptrons (MLPs) that significantly accelerates and improves their training. We show how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate. This not only helps with mitigating spectral bias, but also allows networks to quickly recover from poor initializations and reach better local minima. We demonstrate how random weight factorization can be leveraged to improve the training of neural representations on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces.

1 INTRODUCTION

Coordinate-based MLPs offer flexible, differentiable, memory-efficient signal representations, but struggle to learn high-frequency detail and often require carefully tuned hyperparameters. The paper proposes random weight factorization as a drop-in enhancement, with theoretical and empirical benefits across computer vision, graphics, and scientific computing tasks.

1 INTRODUCTION

  • Coordinate-based MLPs parameterize discrete signals continuously, offering differentiability, lower memory use than grid-sampled representations, and smooth interpolation to unseen coordinates.The paper situates their use across computer vision, graphics, and scientific computing.Source: §1 · p. 1
  • Coordinate-based MLPs exhibit spectral bias, making high-frequency components and fine details difficult to learn.Fourier features and SIREN networks address this issue, but their associated hyperparameters require careful tuning to avoid catastrophic generalization or interpolation errors.Source: §1 · p. 1
  • Appropriate hyperparameter selection can depend on prior knowledge of the target signals, which may be unavailable in some applications.This concern accompanies methods such as Fourier-feature embeddings and periodic-activation SIREN networks.Source: §1 · pp. 1–2
  • The paper frames random weight factorization as a reparameterization motivated by weight normalization and reports significant performance gains.It presents the approach alongside broader normalization methods for improving MLP training and performance.Source: §1 · p. 2
  • Random weight factorization alters the loss landscape, assigns each neuron a self-adaptive learning rate, and can help networks escape poor initializations and reach better local minima.The paper presents it as a drop-in enhancement to conventional linear layers, with improvements across computer vision, graphics, and scientific computing tasks.Source: §1 · p. 2

2 WEIGHT FACTORIZATION

Random weight factorization represents each neuron’s weight vector as a trainable scale times a vector. The paper motivates this parameterization geometrically, derives its self-adaptive learning-rate effect, and distinguishes it from related approaches.

2 WEIGHT FACTORIZATION

  • Each neuron’s weight vector is factorized into a trainable scale and vector, yielding a matrix parameterization W^(l) = diag(s^(l)) · V^(l).The scale is assigned to each individual neuron; the approach reparameterizes the weights trained by gradient descent.Source: §2 · p. 2

2.1 A GEOMETRIC PERSPECTIVE

  • In the factorized sv-plane, distinct parameter configurations can have arbitrarily small distance for sufficiently large s, even when they are far apart in the original parameter space.For a scalar weight, configurations with the same product w = s · v lie on hyperbolas of constant loss; Figure 1 illustrates the reduced distance between initialization and a global minimum.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 SELF-ADAPTIVE LEARNING RATE FOR EACH NEURON

  • Because the trainable scale and vector rescale the original weight’s update, factorization effectively assigns a self-adaptive learning rate to each neuron.The update rescaling depends on the scale and vector parameters, which can change during training.Source: §2.2 · p. 3

2.3 RELATION TO EXISTING WORKS

  • Weight normalization is a special case of the factorization, while random scale initialization distinguishes the approach from adaptive activations and supports reported gains over weight normalization, SIREN, and adaptive activations.Weight normalization sets the scale to the weight norm; adaptive activations are generally initialized with scale a = 1, unlike the random scale initialization described here.Source: §2.3 · p. 4

3 RANDOM WEIGHT FACTORIZATION IN PRACTICE

In a one-dimensional regression comparison, random weight factorization (RWF) yields the best predictive accuracy and loss convergence among the tested parameterizations. Its initialization matters: the paper reports that μ = 1 and σ = 0.1 improves convergence and accuracy for most tasks considered.

3 RANDOM WEIGHT FACTORIZATION IN PRACTICE

  • The regression experiment initializes the MLP with Glorot weights, samples scale parameters s from a multivariate normal distribution, and trains s and V directly.Each weight matrix is factorized at initialization as W = diag(exp(s)) · V.Source: §3 · p. 4
  • RWF yields the best predictive accuracy and loss convergence among the conventional MLP, adaptive activations, and weight normalization in 1D regression.The figure compares predictions, training MSE, original-space relative weight changes, and final empirical NTK eigenvalues; RWF also shows the largest weight changes and a flatter NTK spectrum with slower eigenvalue decay.Source: §3 · pp. 4–5, Figure 2 · p. 5
  • μ = 1 and σ = 0.1 consistently improve loss convergence and model accuracy for the vast majority of tasks considered.Too-small μ and σ can yield performance similar to a conventional MLP, while too-large values can make training unstable.Source: §3 · p. 5

4 EXPERIMENTS

The experiments compare plain MLPs, adaptive activations, weight normalization, and random weight factorization across several tasks and architectures. Random weight factorization consistently achieves the best performance across tasks and architectures, with marginal computational overhead.

4 EXPERIMENTS

  • Random weight factorization consistently outperforms plain MLPs, adaptive activations, and weight normalization across benchmarks under identical hyperparameter settings.The comparison covers multiple architectures, and the method adds only marginal computational overhead.Source: §4 · p. 5, Table 1 · p. 6

4.1 2D IMAGE REGRESSION

  • The 2D image-regression experiments compare full-resolution PSNR after training on images downsampled by a factor of two.The Natural and Text datasets each contain 16 images.Source: §4.1 · pp. 5–6

4.2 3D SHAPE REPRESENTATION

  • The 3D shape task evaluates occupancy-network representations of Dragon and Armadillo using IoU on points sampled near the mesh surface.The near-surface sampling is intended to highlight differences in resolving fine details.Source: §4.2 · p. 6

4.3 2D COMPUTED TOMOGRAPHY (CT)

  • The CT task trains an MLP to infer 2D density fields from sparse integral projections, evaluating reconstructions with PSNR.The benchmarks are Shepp-Logan phantoms and ATLAS brain images.Source: §4.3 · p. 6

4.4 3D INVERSE RENDERING FOR VIEW SYNTHESIS

  • The inverse-rendering task uses a simplified NeRF to learn a 3D scene from 2D photographs and compares trained models by PSNR.The experiments use a down-sampled Lego dataset and remove hierarchical sampling and view dependence.Source: §4.4 · pp. 6–7, Figure 3 · p. 7

4.5 SOLVING PARTIAL DIFFERENTIAL EQUATIONS (PDES)

  • For advection and Navier–Stokes PINNs, random weight factorization yields predictions in excellent agreement with ground truth, while the other parameterizations produce poor or non-physical approximations.The authors attribute the improvement to reaching better local minima beyond the initialization neighborhood.Source: §4.5 · p. 7, Figure 4 · p. 8

4.6 LEARNING OPERATORS

  • Across diffusion-reaction, Darcy flow, and Burgers’ equation, random weight factorization yields the best DeepONet loss convergence in every example.The authors report faster stochastic-gradient convergence and better local minima.Source: §4.6 · p. 8, Figure 5 · p. 9

5 CONCLUSIONS

The paper concludes that random weight factorization improves continuous neural representation training across six benchmarks, while its proof examines equivalent factorizations of the same weights. The proof also establishes that sufficiently large scale factors can correspond to arbitrarily small factor vectors.

5 CONCLUSIONS

  • Random weight factorization improves training across six benchmarks spanning computer vision, graphics, and scientific computing.The authors report consistent and robust improvements across tasks.Source: §5 · p. 9

A.1 PROOF OF THEOREM 1

  • The proof considers pairs of scale vectors and factor matrices whose product equals the original weight matrix.It states the relationship as diag(s(l)) · V(l) = W(l).Source: §A.1 · p. 14
  • For any ε > 0, sufficiently large scale factors yield factor vectors with norm below ε for the equivalent weight representations.The proof states that there exists an M such that when the scale norm exceeds M, the corresponding factor-vector norm is below ε.Source: §A.1 · p. 14

A.2 PROOF OF THEOREM 2

The proof derives gradients and update rules for the factorized weights and scales. The accompanying algorithm specifies initialization and training, and the appendix includes a JAX Flax implementation of a factorized linear layer.

A.2 PROOF OF THEOREM 2

  • The proof differentiates the loss with respect to each factorized weight vector and its scale, then rewrites the updates for both parameters and the resulting weight.The passage introduces the gradient derivation and update-rule transformations without giving their expressions in the supplied text.Source: §A.2 · p. 14

B ALGORITHM OF RANDOM WEIGHT FACTORIZATION

  • The algorithm initializes a network, samples scale factors for each layer, constructs factorized weights, and trains the factorized parameters by gradient descent.It recommends μ = 1.0 and σ = 0.1.Source: B ALGORITHM OF RANDOM WEIGHT… · p. 15

C A DROP-IN ENHANCEMENT FOR LINEAR LAYERS

  • The appendix provides a JAX Flax implementation of a linear layer with random weight factorization.It presents the factorized layer alongside a conventional linear-layer listing.Source: C A DROP-IN ENHANCEMENT FOR LINEAR… · p. 15

D 1D REGRESSION ABLATION STUDY

The 1D regression ablation finds that μ = 1 and σ = 0.1 performs best, while large initialization values can destabilize training. The appendix also summarizes benchmark hyperparameters and reports marginal computational overhead.

D 1D REGRESSION ABLATION STUDY

E HYPER-PARAMETERS

F COMPUTATIONAL COSTS

  • Random weight factorization has marginal computational overhead, with training speed reported in iterations per second on one NVIDIA RTX A6000 GPU.The appendix identifies the benchmark timings as the basis for the overhead comparison.Source: Table 5 · p. 18, F COMPUTATIONAL COSTS · p. 18

G 2D IMAGE REGRESSION

In 2D image regression, weight factorization with Gaussian Fourier features achieves the best reported PSNR among the tested cases. The SIREN comparison finds that its scale factor matters and that its best PSNR remains below the proposed method with positional encodings or Gaussian Fourier features.

G 2D IMAGE REGRESSION

  • The image-regression experiments compare weight parameterizations and input embeddings, training four-layer, 256-channel ReLU models for 2,000 iterations.The models use full-batch gradient descent with Adam; test PSNR is reported in Tables 6 and 7.Source: G 2D IMAGE REGRESSION · p. 19
  • Weight factorization with Gaussian Fourier features achieves the best PSNR among the tested image-regression cases.The reported comparisons cover the Natural and Text datasets.Source: Table 6 · p. 19, Table 7 · p. 19, G 2D IMAGE REGRESSION · p. 19
  • SIREN’s scale factor strongly affects performance; at w0 = 1, it performs similarly to the plain MLP baseline, and its best PSNR remains below the proposed method with positional encodings or Gaussian Fourier features.Figure 9 reports PSNR across scale factors for the Natural and Text datasets.Source: Figure 9 · p. 20, G 2D IMAGE REGRESSION · p. 20

H 2D COMPUTED TOMOGRAPHY

The computed-tomography experiments compare MLP input mappings and weight parameterizations on Shepp-Logan and ATLAS images. Random weight factorization gives the best PSNR across input mappings; SIREN remains below positional encodings and random Fourier features.

H 2D COMPUTED TOMOGRAPHY

  • The evaluation uses 20 Shepp-Logan phantoms and 20 ATLAS brain images, each at 512 × 512 resolution, with 20 and 40 projections per image, respectively.Models use five layers, 256 ReLU channels, and are trained for 2,000 iterations.Source: H 2D COMPUTED TOMOGRAPHY · pp. 20–21
  • Random weight factorization yields the best test PSNR across input mappings on both Shepp and ATLAS datasets.Table 8 reports mean and standard deviation of PSNR for the two datasets.Source: H 2D COMPUTED TOMOGRAPHY · p. 21, Table 8 · p. 21
  • SIREN with its optimal scale factor is slightly better than a plain MLP but worse than positional encodings and random Fourier features.Figure 12 plots PSNR across SIREN scale factors for Shepp and ATLAS, with a plain MLP as baseline.Source: H 2D COMPUTED TOMOGRAPHY · p. 22, Figure 12 · p. 22

I 3D SHAPE REPRESENTATION

The shape-representation experiments evaluate MLPs on Dragon and Armadillo meshes, with test error measured near the surface. Random weight factorization consistently improves IoU across input mappings and datasets.

I 3D SHAPE REPRESENTATION

  • The two test meshes, Dragon and Armadillo, each contain hundreds of thousands of vertices and are rescaled to the unit cube.The mesh centroid is set to (0.5, 0.5, 0.5).Source: I 3D SHAPE REPRESENTATION · p. 22
  • Test error is measured on points near the mesh surface, emphasizing the learning of fine surface details.The points are perturbed mesh vertices, using Gaussian noise with standard deviation 0.01.Source: I 3D SHAPE REPRESENTATION · p. 22
  • Random weight factorization consistently improves IoU across input mappings and both shape datasets, outperforming the other parameterizations.Table 9 reports IoU for Dragon and Armadillo.Source: I 3D SHAPE REPRESENTATION · p. 22, Table 9 · p. 22

J 3D INVERSE RENDERING FOR VIEW SYNTHESIS

For view synthesis, the experiments train MLPs on the NeRF Lego dataset using Gaussian Fourier features. Random weight factorization achieves the best test PSNR among the four parameterizations during training.

J 3D INVERSE RENDERING FOR VIEW SYNTHESIS

K SOLVING PDES

  • The following PDE section provides implementation details for PINNs solving advection and Navier–Stokes equations.The supplied section passage states the scope of those implementation details.Source: K SOLVING PDES · p. 24

K.1 ADVECTION EQUATION

For advection, random weight factorization is the only tested parameterization that enables stable, reasonable PINN accuracy, with further improvement from curriculum or causal training. The experiments also report results for Navier–Stokes and three operator-learning PDE benchmarks.

K.1 ADVECTION EQUATION

  • For the advection equation, random weight factorization alone enables reasonable, stable predictive accuracy; curriculum or causal training can improve it further.The PINNs use a special Fourier embedding for exact periodic boundary conditions; curriculum training raises the advection coefficient from 10 to 50, while causal training reweights temporal residuals.Source: §K.1 · pp. 24–25, Table 10 · p. 25

K.2 NAVIER-STOKES EQUATION

  • For Navier–Stokes, random weight factorization performs best among the tested parameterizations by a large margin.The study uses a modified MLP and curriculum training, increasing Reynolds number from 100 to 500 and then 1,000.Source: §K.2 · pp. 27–28, Table 11 · p. 27

L LEARNING OPERATORS

  • The learning-operators experiments use DeepONets, with separate branch and trunk networks whose embeddings are merged by a dot product.Table 12 reports relative L2 errors for the operator-learning examples.Source: L LEARNING OPERATORS · p. 28, Table 12 · p. 29

L.1 DIFFUSION-REACTION

  • For diffusion-reaction, a DeepONet is trained on sampled input functions and sparse solution measurements, with test relative L2 errors reported.The training data uses 5,000 input functions and 100 measurements from each solution; the test set contains 100 solutions.Source: §L.1 · p. 29, Table 12 · p. 29, Figure 19 · p. 30

L.2 DARCY FLOW

  • For Darcy flow, a DeepONet with a CNN branch and an MLP trunk is evaluated using relative L2 errors and predicted solutions.Training data contains 2,000 solutions and the test set contains 100 solutions.Source: Table 12 · p. 29, §L.2 · p. 30, Figure 20 · p. 31

L.3 BURGERS’ EQUATION

  • For Burgers’ equation, a physics-informed DeepONet maps initial conditions to full spatiotemporal solutions, with test errors and predictions reported.The model uses Fourier features for exact periodic boundary conditions and trains with sampled input functions and collocation points.Source: Table 12 · p. 29, §L.3 · pp. 31–32, Figure 21 · p. 33
Loading 2210.01274v2…