PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Neural Representations with Periodic Activation Functions

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein

arXiv:2006.09661v1cs.CVcs.LGeess.IV

TL;DR

Existing implicit neural representations struggle with fine detail and reliable signal derivatives, limiting their use for physical problems. This paper introduces periodic-activation SIRENs and shows they represent complex signals and derivatives while robustly handling several boundary value problems.

  • Problem

    ReLU-based implicit neural representations struggle to model fine signal detail and typically represent target-signal derivatives poorly, despite derivatives being important for physical problems.

  • Method

    The paper uses periodic-activation MLPs, develops an initialization scheme, and combines SIRENs with hypernetworks to model implicit signals and their derivatives.

  • Results

    SIRENs robustly represent complex natural signals and derivatives and demonstrate applications spanning images, video, audio, shapes, and first- and second-order differential equations.

  • Takeaways & Limitations

    Periodic activations provide an implicit neural representation framework suited to complex signals, their derivatives, and several boundary value problems.

  • Takeaways & Limitations

    Formalizing the distribution of output frequencies throughout SIRENs remains unresolved and is outside the paper’s scope.

Abstract

from arXiv · show

Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and demonstrate the representation of images, wavefields, video, sound, and their derivatives. Further, we show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations (yielding signed distance functions), the Poisson equation, and the Helmholtz and wave equations. Lastly, we combine Sirens with hypernetworks to learn priors over the space of Siren functions.

1 Introduction · 2 Related Work · 3 Formulation

The paper formulates implicit neural representations as continuous neural parameterizations of functions constrained by differential equations, then introduces periodic-activation SIRENs to model complex signals and their derivatives. It develops the feasibility-loss formulation underlying these representations and summarizes applications, initialization, and related work.

  • 1 Introduction: Implicit neural representations parameterize a function Φ from spatial or spatio-temporal coordinates, optionally using derivatives, while satisfying constraints defined by F.This formulation treats Φ as implicitly defined by the relation encoded in F.
  • 1 Introduction: Continuous parameterizations can use less memory than discrete grids, represent detail beyond grid resolution, and provide gradients and higher-order derivatives.These benefits arise because Φ is defined over a continuous domain and is differentiable.
  • 1 Introduction: ReLU-based implicit architectures often lack fine-detail capacity and poorly represent derivatives because ReLU networks are piecewise linear.The paper motivates periodic activations as a response to these limitations and reports better signal and derivative representation than ReLU-MLPs or positional encoding.
  • 1 Introduction: The paper contributes periodic implicit representations for complex signals and derivatives, a principled initialization scheme, hypernetwork-learned representation distributions, and applications spanning media, shapes, and differential equations.Applications include image, video, audio, 3D shape reconstruction, gradient-supervised first-order equations, and second-order equations.
  • 2 Related Work: Prior implicit neural representations use fully connected networks as continuous, memory-efficient representations for shape parts, objects, and scenes, commonly trained as signed distance functions or occupancy networks.Related work also extends some models to encode object appearance.
  • 2 Related Work: Earlier periodic nonlinearities included Fourier neural networks, periodic-activation classifiers, and recurrent networks, but had not robustly outperformed alternative activation functions.The paper positions SIREN as a periodic-activation approach for implicit representations rather than those earlier tasks and architectures.
  • 3 Formulation: The formulation seeks Φ satisfying M constraints Cm(a(x), Φ(x), ∇Φ(x), ...) = 0 over domains Ωm, casting the task as a feasibility problem.The constraints relate Φ and/or its derivatives to quantities a(x).
  • 3 Formulation: Training penalizes constraint violations on sampled coordinates, with dynamically sampled datasets approximating the domain loss through Monte Carlo integration.Functions Φθ are parameterized by fully connected networks and optimized using gradient descent.

4 Experiments

Experiments show that SIRENs can be trained from derivative supervision to reconstruct signals, represent detailed signed distance functions, and solve challenging wave-related boundary value problems. Combining SIRENs with hypernetworks also enables learned priors over implicit function spaces and improves CelebA inpainting performance.

  • Eikonal equation and shape representation: For signed distance functions, periodic activations improve object detail and scene complexity over ReLU implicit representations.SIRENs are fitted directly to oriented point clouds while enforcing zero values on the surface, matching normals, and unit gradient norms almost everywhere.
  • Derivative supervision: SIRENs accurately represent functions and derivatives while also supporting supervision using only first- or higher-order derivatives.This capability is demonstrated through Poisson reconstruction, where gradient or Laplacian supervision successfully reconstructs images despite ill-posed intensity variations.
  • Poisson equation: Gradient- and Laplacian-supervised SIRENs successfully reconstruct images, while gradient-domain supervision enables seamless image fusion.The fusion uses a composite gradient formed from two images with mixing coefficient α ∈ [0, 1].
  • Helmholtz and wave equations: SIRENs solve two-dimensional Helmholtz problems with complex-valued outputs, closely matching a principled grid solver while other neural architectures fail to find the correct solution.The network is trained by sampling points in the domain and enforcing the Helmholtz equation with modified perfectly matched boundary conditions.
  • Helmholtz and wave equations: SIRENs directly recover accurate wavefields for known velocity perturbations and jointly recover wavefields and velocity in neural full-waveform inversion.The inversion experiment uses 5 sources, 30 receivers, and measurements at a single frequency of 3.2 Hz.
  • Priors over implicit functions: A ReLU hypernetwork maps latent codes to SIREN weights, and a convolutional encoder improves quantitative and qualitative CelebA inpainting performance.This demonstrates that SIREN function spaces can support learned priors over families of implicit functions.

5 Discussion and Conclusion

Periodic activation functions enable implicit neural representations to model complex natural signals and their derivatives, while also supporting prototypes for several boundary value problems.

  • 5 Discussion and Conclusion: Periodic activation functions are well suited to representing complex natural signals and their derivatives with implicit neural representations.The work positions implicit neural representations as a potential tool across science and engineering, offering benefits over conventional continuous and discrete representations.
  • 5 Discussion and Conclusion: The authors also prototype several boundary value problems using implicit neural representations.

Broader Impact · Activation Functions · 1 Initialization and Distribution of Activations

SIREN uses periodic sine activations to represent natural signals and derives an initialization scheme that preserves activation distributions across depth. The paper also discusses potential downstream uses and misuse of accurate signal representations.

  • Broader Impact: SIREN accurately represents natural signals including images, audio, and video, potentially enabling downstream classification, speech-to-text, and signal-generation applications.The paper notes that such applications may serve positive or negative ends, including impersonation misuse.
  • 1.1 Informal statement: The distribution-preserving argument recursively combines arcsine outputs, their variances, and a central-limit approximation for subsequent layer inputs.The proof overview states that matching input and output distributions preserves them throughout the network.
  • Activation Functions: The sine activation is periodic with period 2π and odd, making it suitable for mapping normalized coordinates through SIREN.The analysis considers inputs uniformly distributed in [-1,1] and studies a frequency-scaled SIREN.
  • 1.2 Preliminary results: A standard normal input passed through a sine nonlinearity produces an approximately arcsine-distributed output, as shown by the theoretical CDF comparison.The approximation is evaluated over [-3,3], containing 99.7% of the standard normal probability mass.
  • 1.3 Formal statement and proof of the initialization scheme: The proposed initialization draws weights uniformly so activations remain standard normal before each sine and arcsine-distributed after each sine throughout the network.The scheme is designed to preserve activation distributions and support deep architectures.
  • 1.4 Empirical evaluation: A 6-layer SIREN with 2048 hidden units and 28 inputs was used to empirically validate activation histograms, Fourier transforms, final-layer sums, and gradients.The experiment initialized the network according to the proposed scheme and evaluated activations after each linear and sine layer.
  • 1.5 About ω0: Using ω0 in every layer keeps activation distributions constant while boosting gradients to the factored weight matrix by ω0 and leaving input gradients unchanged.The paper presents this factorization as a way to accelerate SIREN training.

2 Evaluating the Gradient of a SIREN is Evaluating another SIREN

At initialization, a six-layer SIREN’s activation and gradient statistics closely match analytical predictions. Activations and gradients remain consistent across layers, while maximum frequency grows only slowly.

  • Initialization statistics: The six-layer SIREN’s activation and gradient statistics at initialization closely match analytically predicted distributions.The figure compares experimental statistics with orange dotted analytical predictions.
  • Layer-wise behavior: Activation distributions stay consistent across layers, maximum frequency grows only slowly, and gradient statistics similarly remain consistent from layer to layer.These observations are reported across the six-layer network as depth increases.

ReLU P.E. Tanh

The section defines SIREN sine layers and shows that their input gradients can be evaluated through related SIREN networks. In Poisson image reconstruction, ReLU positional-encoding and tanh architectures recover gradients less accurately than SIREN and fail to reconstruct the Laplacian accurately.

  • Poisson reconstruction: ReLU P.E. and tanh networks reconstruct images from gradients with lower quality than SIREN, while Laplacian reconstruction is not at all accurate for either architecture.The comparison concerns Poisson image reconstruction using the ReLU positional-encoding and tanh architectures.
  • Sine layers: A sine layer applies a sine nonlinearity to an affine transformation, φ_i(x) = (sin ◦ T_i)(x), with T_i(x) = W_i x + b_i.The affine map is represented using augmented weights and inputs, Ẇ = [W, b] and x̂ = [x, 1].
  • Derivative evaluation: The gradient of a SIREN can be evaluated by another SIREN whose biases are shifted by π/2.This follows by rewriting sine derivatives as phase-shifted sine activations and evaluating a modified forward pass.

3 Solving the Poisson Equation

The Poisson image reconstruction experiments compare neural architectures by supervising reconstructed images and their derivatives. SIREN outperforms alternatives in representing gradient and Laplace images, whose high-frequency features challenge other architectures.

  • Evaluation: The evaluation compares reconstructed image, gradient image, and Laplace image quality using PSNR on the starfish Poisson reconstruction task.Images are colorized and normalized before PSNR is computed.
  • Architecture comparison: SIREN is more accurate than ReLU P.E. for supervising image derivatives in the Poisson reconstruction task.ReLU P.E., despite performing best among non-SIREN models on single-image fitting, is less accurate when supervising derivatives.
  • Architecture comparison: ReLU P.E. struggles to fit gradient and Laplace images even when directly supervised, despite fitting images relatively accurately.The passage attributes this difficulty to the many high-frequency features in the ground-truth derivatives.
  • Experimental setup: The starfish image comes from BSDS500, is center-cropped to 321 × 321, and resized to 256 × 256.Ground-truth gradients use a Sobel filter and ground-truth Laplacians use a Laplace filter; both are scaled for training.
  • Experimental setup: All image and gradient-fitting experiments use the same 5-layer SIREN MLP and train for 10,000 iterations with Adam at a learning rate of 1 × 10−4.Each iteration fits every pixel in the gradient or Laplacian image.

4 Representing Shapes with Signed Distance Functions

SIREN fits signed distance functions from oriented point clouds while preserving high-frequency shape detail better than a ReLU positional-encoding baseline. Experiments use a five-layer SIREN MLP on scanned meshes and a 3D room model.

  • SDF fitting: SIREN’s SDF zero-level sets preserve high-frequency shape features, whereas ReLU positional encoding introduces high frequencies absent from the shape while retaining low detail.The comparison uses an MLP with the ReLU positional encoding proposed in.
  • Data: The dataset comprises a Thai statue and a Free 3D room model, each sampled by subdivision to 10 million points with normals.The meshes and normals are converted to .xyz format for loading.
  • Architecture: All SDF experiments use a five-layer SIREN MLP with 256 units per layer for the statue and 1024 units per layer for the room.The same five-layer architecture is used across the SDF experiments.
  • Training: 50,000 iterations are trained with Adam at a learning rate of 1 × 10^-4, fitting every voxel of the volume at each iteration.The objective includes Eikonal, signed-distance, off-surface, and oriented surface/normal constraints with specified weights.
  • Runtime: Approximately 6 hours are required to fit and evaluate a SIREN, although it converges well after around 5,000-7,000 iterations.The longer 50,000-iteration schedule is used because baselines require substantially more iterations.

5 Solving the Helmholtz and Wave Equations

The section applies SIREN to Helmholtz and wave-equation problems, using boundary attenuation, pretraining, and physics-based losses to represent wavefields. For the wave initial-value problem, SIREN closely matches a principled solver, whereas tanh fails to converge meaningfully.

  • Helmholtz equation: A perfectly matched layer attenuates Helmholtz waves at the finite-domain boundary, while the equation remains unchanged elsewhere.The experiments use a0 = 5 and apply the PML on ∂Ω = {x | 0.5 < ∥x∥∞ < 1}.
  • Full-waveform inversion: SIREN solves full-waveform inversion by first fitting wavefields under a uniform velocity model, then jointly recovering wavefields and squared slowness from receiver data.Pretraining uses m0 = 1 and a slowness regularization term before the subsequent penalty-method inversion.
  • Wave equation: The wave equation is formulated as a time-dependent real-valued wavefield Φ(t, x) with initial condition Φ(0, x) = f(x).For the acoustic equation, f(x) is an initial pressure distribution and the initial particle velocity is zero.
  • Wave equation: Training propagates the initial condition to later times by increasing sampled t linearly during optimization.The time interval grows from 0.0 to 0.4 over 100,000 iterations, with an ADAM learning rate of 2 × 10^-5.
  • Wave equation: SIREN recovers a wavefield close to a principled solver for a Gaussian initial pulse, while the tanh baseline fails to converge to a meaningful result.The Gaussian is centered at the origin with variance 5 × 10^-4.

6 Application to Image Processing

SIRENs represent continuous natural images and enable image-processing formulations that combine sampled-image data fidelity with priors over SIREN functions. The section demonstrates this approach for image fitting and single-image inpainting, including derivative representation and prior-based reconstruction.

  • Image processing formulation: Continuous SIREN image fitting minimizes sampled-image data fidelity together with a regularization prior defined over the continuous representation.The sampling operator and downsampling kernel connect the continuous image Φθ(x, y) to the observed discrete image b.
  • Image fitting: SIRENs can fit natural images while uniquely enabling accurate fitting of higher-order derivatives among the compared architectures.The comparison includes ReLU, tanh, ReLU P.E., RBF-input, Softplus, ELU, and SELU architectures.
  • Single-image inpainting: SIRENs solve single-image inpainting by fitting the observed pixels while enforcing priors, with comparisons against Deep Image Prior, diffusion, no-prior SIRENs, and regularized SIRENs.The evaluated priors include total variation and Frobenius-norm Hessian regularization.
  • Single-image inpainting: 10% of pixels are sampled for the inpainting experiments, and including the TV prior improves performance when the image is known to match that prior.Experiments use texture images, including art at 513 × 513 resolution and tiles at 355 × 533 × 3 resolution.
  • Single-image inpainting: SIREN inpainting is highly stable and insensitive to the specific sampled pixel mask across the reported independent runs.The quantitative evaluation reports mean and standard deviation PSNR scores for the texture-image inpainting experiments.

7 Representing Video

The section fits videos with SIREN and ReLU networks, finding that SIREN better represents fine scene details. Experiments use cat and vehicle/traffic video datasets with matched five-layer architectures.

  • Video representation: SIREN improves representation of fine details in videos compared with a baseline ReLU architecture, including scenes of vehicles moving in traffic and outdoor environments.The comparison includes a cat video and a second video shown in Fig. 8.
  • Datasets: The first dataset is a cat video cropped and downsampled to 300 frames at 512×512 resolution.The source is permissively licensed and available from Pexels.
  • Datasets: The second dataset is the “bikes sequence” from the scikit-video Python package, while the experiments also use a video of vehicles in traffic and outdoor scenes.The supplied passage identifies the bikes sequence as the second dataset; the experiments describe its content as vehicles moving in traffic and outdoor scenes.
  • Architecture: The SIREN and ReLU architectures each use 5 layers with a hidden layer size of 1024.This provides the matched architecture configuration for the video comparison.
  • Training setup: Training uses Adam with a learning rate of 1 × 10−4, batches of roughly 160,000, 100,000 iterations, and approximately 15 hours on NVIDIA Titan X (Pascal) GPUs.The GPUs have 12 GB of memory.

8 Representing Audio Signals

SIRENs efficiently represent raw music and speech waveforms, whose periodic, multiscale structure suits periodic activations. They accurately fit audio while using fewer parameters than waveform samples, with stable convergence and recognizable reconstructions.

  • Audio waveform representation: SIRENs accurately represent raw music and speech waveforms, whereas other neural architectures fail to represent them.The evaluation uses audio clips of varying lengths, and Fig. 9 shows accurate SIREN fits for both music and speech.
  • Audio waveform representation: SIRENs use far fewer parameters than the number of samples in the ground-truth waveform, enabling signal compression.The paper suggests this compression capability could support lossy audio compression algorithms based on SIRENs.
  • Audio waveform representation: SIREN audio fitting achieves converged mean-squared error with highly stable convergence across 10 independent runs.Table 3 evaluates reconstruction MSE mean and variance after 5,000 fitting iterations; the supplied passage does not include the numerical values.
  • Experimental setup: The audio experiments use a 5-layer sine-activated MLP and scale the input domain to x ∈ [−100, 100] to account for the 44,100-samples-per-second sampling rate.The waveforms are normalized to [−1, 1] before fitting.

9 Learning a Space of Implicit Functions

The section learns a prior over SIREN functions for CelebA faces and uses it to reconstruct images from sparse observations through inpainting. A hypernetwork maps encoded context to SIREN weights, while regularization encourages unique, compact, lower-frequency representations.

  • Learning a Space of Implicit Functions: A learned prior over SIREN functions enables reconstruction from noisy or few observations and image inpainting on CelebA faces.The demonstrations use CelebA face images and reconstruct missing pixels from partial observations.
  • Learning a Space of Implicit Functions: The hypernetwork decoder maps a latent code from encoded partial observations to the weights of a five-layer SIREN with 256 hidden features.The decoder is a ReLU MLP with one hidden layer of 256 features.
  • Learning a Space of Implicit Functions: Regularization combines image fidelity, a Gaussian latent prior, and weight regularization that encourages unique, compact, lower-frequency SIREN representations.The weight term addresses the many possible SIREN representations of the same image.
  • Learning a Space of Implicit Functions: The convolutional encoder outperforms the partial convolutional encoder, while the learned SIREN prior enables inpainting comparably to image-based CNP methods.The authors attribute the convolutional encoder’s advantage to capturing complex spatial relationships and using information from masked pixels.
Loading 2006.09661v1…