PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs

Sameera Ramasinghe, Simon Lucey

arXiv:2111.15135v2cs.LG

TL;DR

Coordinate-MLPs need better activation choices because ReLU networks poorly represent high-frequency signals, while sinusoidal alternatives remain sensitive to initialization and insufficiently understood. The paper develops a broader activation framework based on Lipschitz smoothness and singular-value distributions, then proposes non-periodic functions including Gaussian activations. These activations support embedding-free coordinate-MLPs and are reported to provide stronger convergence, initialization robustness, and performance than the compared alternatives.

  • Problem

    ReLU-based coordinate-MLPs poorly encode high-frequency signals, while sinusoidal activations' effectiveness and sensitivity to initialization remain insufficiently understood.

  • Method

    The paper relates activation suitability to Lipschitz smoothness and hidden-layer singular-value distributions, derives activation-design guidelines, and proposes non-periodic functions.

  • Results

    The proposed non-periodic activations support positional-embedding-free coordinate-MLPs and show better convergence properties and initialization robustness than sinusoidal activations; Gaussian activations also outperform ReLU with ∼50% less parameters.

  • Takeaways & Limitations

    Sinusoidal periodicity is not crucial: coordinate-MLPs can use broader non-periodic activation families for high-fidelity signal encoding.

  • Takeaways & Limitations

    Generalization depends on priors about the ground-truth signal, with smooth interpolation serving as an engineering choice when such priors are unavailable.

Abstract

from arXiv · show

Coordinate-MLPs are emerging as an effective tool for modeling multidimensional continuous signals, overcoming many drawbacks associated with discrete grid-based approximations. However, coordinate-MLPs with ReLU activations, in their rudimentary form, demonstrate poor performance in representing signals with high fidelity, promoting the need for positional embedding layers. Recently, Sitzmann et al. proposed a sinusoidal activation function that has the capacity to omit positional embedding from coordinate-MLPs while still preserving high signal fidelity. Despite its potential, ReLUs are still dominating the space of coordinate-MLPs; we speculate that this is due to the hyper-sensitivity of networks -- that employ such sinusoidal activations -- to the initialization schemes. In this paper, we attempt to broaden the current understanding of the effect of activations in coordinate-MLPs, and show that there exists a broader class of activations that are suitable for encoding signals. We affirm that sinusoidal activations are only a single example in this class, and propose several non-periodic functions that empirically demonstrate more robust performance against random initializations than sinusoids. Finally, we advocate for a shift towards coordinate-MLPs that employ these non-traditional activation functions due to their high performance and simplicity.

1 Introduction

Coordinate-MLPs provide compact, continuously differentiable representations of multidimensional signals, but ReLU networks struggle with high-frequency content. The paper broadens activation-function design beyond sinusoidal periodicity, proposing non-periodic activations such as Gaussian that support embedding-free architectures and greater initialization robustness.

  • 1 Introduction: Coordinate-MLPs encode continuous signals from low-dimensional coordinates as network weights and can be queried at extremely high resolutions.They offer a more compact relationship between memory, dimension, and resolution than grid-based representations.
  • 1 Introduction: ReLU-based coordinate-MLPs poorly encode high-frequency signals because MLP spectral bias produces high-frequency fall-offs in their neural tangent kernels.Positional embeddings using Fourier features are the prevalent workaround.
  • 1 Introduction: Sinusoidal activations can encode high-frequency signals without positional embeddings, but their effectiveness and sensitivity to initialization remain insufficiently understood.The paper notes that prior work emphasized periodicity while evidence for that relationship remains scant.
  • 1 Introduction: The paper links coordinate-MLP efficacy to Lipschitz smoothness and hidden-layer singular-value distributions, whose optimal values depend on the encoded signal.It derives connections between these properties and activation-function characteristics to guide activation tuning and prediction before implementation.
  • 1 Introduction: Non-periodic activations are presented as alternatives to sinusoids, with Gaussian activations achieving better results using ∼50% less parameters and supporting embedding-free architectures.The proposed activations are also described as more robust to random initialization than sinusoidal activations.

2 Related works

The related-work discussion situates the paper within research on activation functions and coordinate-MLPs, which parameterize signals with neural networks rather than discrete structures.

  • 2 Related works: Activation functions are essential neural-network components for modeling relationships between complex functions and their coordinates.The discussion notes that nonlinear activations broaden the class of signals and functions that can be approximated.
  • 2 Related works: Coordinate-MLPs, also called implicit neural functions, parameterize signals using neural networks and differ from conventional MLPs in inputs and intended use.The passage contrasts coordinate-MLPs with conventional networks used on high-dimensional inputs for classification.

3 Methodology

The methodology analyzes coordinate-MLPs through memorization, generalization, representation rank, Lipschitz smoothness, and activation derivatives. It uses these relationships to motivate parameterized activations satisfying two derivative-based requirements for high-fidelity signal encoding.

  • MLP formulation: Coordinate-MLPs encode continuous signals as network weights from low-dimensional coordinates, using fully connected architectures with nonlinear hidden layers and a linear final layer.The hidden representation is organized as an embedding matrix of training examples before the final affine projection.
  • 3.1 Rank and memorization: Rank(X) = N guarantees perfect reconstruction of arbitrary training outputs when the embedding dimension D is at least the number of training examples N.For redundant natural signals, lower rank can still suffice, whereas limited-redundancy noise requires greater width.
  • 3.1 Rank and memorization: High embedding rank improves memorization but does not ensure generalization, because sharply localized embeddings can map unseen points to zero.The analysis therefore distinguishes fitting observed coordinates from predicting values at unseen coordinates.
  • 3.3 Singular value distribution as a proxy for smoothness: Larger stable rank tends to indicate larger local Lipschitz constants, making stable rank a practical proxy for layer smoothness despite the absence of a rigorous proof.Networks empirically increase Lipschitz constants primarily by increasing angles between output vectors; approximately equal singular-value distributions violate the desired smoothness.
  • 3.4 Local Lipschitz smoothness and the activation function: High-frequency signals require activations with large first-order derivative magnitudes, while signals with varying fluctuations additionally require nonnegligible second-order derivatives.R1 controls derivative magnitude through hyperparameters; R2 supplies varying local smoothness so affine transformations can project points into different activation regions.
  • 3.4 Local Lipschitz smoothness and the activation function: Sine and the proposed non-periodic activations satisfy R1 and R2, showing that periodicity is not required for coordinate-MLPs to encode high-frequency signals.The proposed activation family includes parameterized infinitely differentiable functions, broadening the class beyond sinusoidal activations.

4 Experiments

Experiments compare coordinate-MLPs using different activations and show that proposed non-periodic functions encode signals with higher fidelity, while Gaussian activations converge more robustly under varied initialization. The results also link reconstruction quality to suitably tuned Lipschitz smoothness and stable rank.

  • Gaussian, Laplacian, ExpSin, and Quadratic activations encode images with sharper gradients and higher fidelity than ReLU, Tanh, SoftPlus, and SiLU.The comparison concerns 2D image encoding and associates sharper gradients with higher Lipschitz constants.
  • Without positional embeddings, Gaussian activations capture high-frequency details with higher fidelity than ReLU MLPs.
  • 4.3 Convergence: Extremely high or low stable rank or Lipschitz constants hamper fine-detail encoding, so activation hyperparameters must be tuned to a suitable range.
  • 4.3 Convergence: Sine-activated MLPs may fail to converge after 3000 epochs when initialization does not strictly follow Sitzmann et al.'s method, whereas Gaussian activations converge faster.
  • 4.4 Local Lipschitz smoothness: A low Lipschitz constant produces blurry edges, while an extremely large one permits unwanted fluctuations; suitable parameters support better performance.
  • 4.4 Local Lipschitz smoothness: Gaussian reconstructions with properly chosen parameters exhibit a distribution of local Lipschitz constants after signal encoding.

5 Conclusion

The conclusion presents sinusoidal activation as one member of a broader class of activations for high-fidelity coordinate-MLP encoding. Proposed non-periodic functions support positional-embedding-free architectures and show stronger robustness to initialization, while Gaussian activations perform well across signal-encoding tasks.

  • Gaussian activations are significantly more robust than sine activations across initialization methods, although sine converges similarly when initialized strictly according to Sitzmann et al.'s guidelines.
  • Sinusoidal activation is one example of a broader class of activations that encode high-frequency signals.
  • The proposed non-periodic activations enable positional-embedding-free coordinate-MLPs and converge more robustly under varied initialization schemes than sinusoidal activations.
  • Gaussian activations demonstrate compelling results across various signal-encoding tasks.

A Encoding signals

This section presents qualitative and quantitative evaluations of proposed activations for signal encoding, including 2D images, 3D view synthesis, convergence, and 1D signals. The supplied table caption states that proposed activations achieve high PSNRs, while noise signals remain difficult because of limited redundancy.

  • Table 2 compares activations for 2D signal encoding after 3000 epochs using PSNR, with proposed activations yielding high values.Noise signals are difficult to encode with high fidelity because of limited redundancy.
  • Figure 10 shows qualitative examples of 2D signal encoding with the proposed activations on natural images.
  • Figures 11 and 12 qualitatively compare ReLU and Gaussian activations without positional embedding for 3D view synthesis.
  • Figure 13 tracks convergence of the Frobenius norm of the Jacobian during training.
  • Figure 14 presents 1D signal encoding using Gaussian-activated MLPs.

B Norms of the layer outputs.

The section examines how local Lipschitz smoothness relates to Jacobian norms, activation derivatives, and layer-output norms. It argues that suitable derivative ranges and approximately locally constant output norms support the proposed analysis.

  • Layer-output norms remain approximately locally constant within image patches measured using overlapping sliding windows.
  • The analysis is initially derived assuming approximately constant layer-output norms, but its conclusions are stated to hold even when those norms are not approximately constant.
  • A hidden layer composes an affine map with a nonlinearity, and its local Lipschitz constant is upper-bounded by the product of their local constants.
  • Taylor expansion expresses the activation near x0 as its value at x0 plus a Jacobian term and a rapidly decaying remainder.
  • The local Lipschitz constant is bounded by the Frobenius norm of the Jacobian at the corresponding point.
  • A larger Lipschitz constant over an interval requires a higher maximum first-order activation derivative there, while varying local smoothness requires derivative maxima that vary across the domain.
Loading 2111.15135v2…