PaperlayerView exact revision on arXiv

Source-linked AI summary

Modulated Periodic Activations for Generalizable Local Functional Representations

Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, Manmohan Chandraker

arXiv:2104.03960v1cs.CVcs.GR

TL;DR

Existing neural functional representations often require separate models per signal, while generalizing across instances can reduce reconstruction quality. This paper uses modulated periodic activations and local latent-coded functions to represent images, videos, and shapes with high fidelity while generalizing across instances.

  • Problem

    Existing approaches often train a separate MLP for each signal, while multi-instance generalization still causes significant quality degradation.

  • Method

    A dual-MLP architecture uses a modulation network to control periodic activation amplitude, phase, and frequency in a synthesis network, with latent codes assigned to tiled local regions.

  • Results

    The proposed representation generalizes across multiple image, video, and shape instances while achieving high-fidelity functional reconstructions.

  • Takeaways & Limitations

    Local tiling and modulated sine activations provide a general functional representation for large, complex signals across images, videos, and shapes.

  • Takeaways & Limitations

    Concatenating latent codes with input coordinates consistently fails when the synthesis network uses sine activations.

Abstract

from arXiv · show

Multi-Layer Perceptrons (MLPs) make powerful functional representations for sampling and reconstruction problems involving low-dimensional signals like images,shapes and light fields. Recent works have significantly improved their ability to represent high-frequency content by using periodic activations or positional encodings. This often came at the expense of generalization: modern methods are typically optimized for a single signal. We present a new representation that generalizes to multiple instances and achieves state-of-the-art fidelity. We use a dual-MLP architecture to encode the signals. A synthesis network creates a functional mapping from a low-dimensional input (e.g. pixel-position) to the output domain (e.g. RGB color). A modulation network maps a latent code corresponding to the target signal to parameters that modulate the periodic activations of the synthesis network. We also propose a local-functional representation which enables generalization. The signal's domain is partitioned into a regular grid,with each tile represented by a latent code. At test time, the signal is encoded with high-fidelity by inferring (or directly optimizing) the latent code-book. Our approach produces generalizable functional representations of images, videos and shapes, and achieves higher reconstruction quality than prior works that are optimized for a single signal.

1. Introduction

The paper introduces a dual-MLP neural-functional representation that combines high-fidelity periodic synthesis with latent-code modulation to generalize across multiple signals. A local tiled representation further makes high-resolution encoding tractable by modeling simpler recurring structures.

  • Problem and contribution: The method encodes multiple discrete signals with one model in a single feed-forward pass, addressing quality degradation in prior multi-instance generalization methods.Each signal is represented by a low-dimensional latent code conditioning the functional mapping.
  • Network architecture: A modulation network consumes latent codes and generates layer-wise parameters for a sine-activated synthesis network mapping coordinates to signal values.Periodic activations enable high-frequency reconstruction, while modulation provides the key conditioning mechanism for generalization.
  • Summary of contributions: Compared with previous methods, the approach produces qualitatively and quantitatively superior functional representations with improved generalization capabilities.The stated scope includes high-fidelity representations of images, shapes, and videos.
  • Local representation: The local representation partitions the input domain into tiles, assigns each tile a latent code, and combines local functions to reconstruct the target signal.Local structures are simpler and exhibit fewer variations than complete objects, making generalizable functional approximation more tractable.
  • Motivation: SIREN reconstruction accuracy drops sharply for high-resolution images, whereas the proposed method encodes multiple high-resolution images through feed-forward passes without gradient computation.The comparison is presented in Figure 2 and motivates the local, generalizable design.

2. Related Work

Prior work uses MLPs and periodic activations for continuous visual-signal representations, while instance-conditioned and local models address generalization and scalable processing limitations. This work combines locality with modulation to improve generalization as signal complexity increases.

  • Continuous Representations of Visual Signals: MLPs encode images [33], videos [37], shapes [31, 7], and 3D scenes [22] [4] [25] as continuously queryable functions supporting view synthesis and interpolation [22] [25] [26].These models provide continuous representations that can be queried at arbitrary locations in the signal’s input domain.
  • Periodic Activations: Periodic activations improve representation capability but become difficult to train in multilayer networks and may converge to undesired local minima.For small datasets, sine activations in the first layer with monotonic functions thereafter produce compelling results.
  • Periodic Activations: As resolution or video duration increases, the quality of existing networks degrades; this work addresses the limitation through locality and an effective modulation mechanism for generalization.The comparison concerns progressively higher-resolution images and longer videos.
  • Instance-conditioned Implicit Functions: Current implicit representations commonly require optimization for each test signal, motivating latent-variable conditioning mechanisms for forward-pass generalization.The passage contrasts individually optimized implicit models with established models trained on large datasets that require only a test-time forward pass.
  • Applications of Local Models: Local representations support complex-system processing and efficient rendering through KD-trees, regular grids, voxelized models, and applications including light-field compression and radiance transfer [5] [10] [38] [15].The cited applications span fluid simulation, photon mapping, ray tracing, surface light fields, pre-computed radiance transfer, and 3D representation.

3. Method

The method represents many signals with latent codes processed by a modulation network that conditionally modulates a synthesis-network template. It improves scalability and fidelity through local tile codes, which can be inferred with an encoder or optimized jointly with model parameters.

  • 3.1: A dual-MLP architecture maps each signal’s latent code through a modulation network that conditionally modulates a synthesis network representing the functional mapping.The approach simultaneously encodes a large number of functions, with each function summarized by a latent code.
  • 3.2: Sine-MLPs reconstruct a wider frequency spectrum, whereas ReLU-MLPs fail to reconstruct high-frequency components.The comparison uses a 6 × 6 grid of Perlin [32] texture patches whose horizontal and vertical frequencies increase from the top-left to bottom-right.
  • 3.3: Local tile latent codes enable functional representations of higher-resolution signals with higher fidelity than single-MLP approaches.The input domain is decomposed into local tiles, each represented by a latent code.
  • 3.4: Latent codes can be estimated from discrete inputs with a convolutional encoder or optimized jointly with model parameters, as in [5].These alternatives provide two ways to obtain the local representation at inference or optimization time.

3.1. Modulated Periodic Activations

The representation is a continuous conditional mapping built from synthesis and modulation MLPs. The synthesis network maps coordinates to signal values with periodic activations, while latent-code modulation controls those activations across layers.

  • 3.1. Modulated Periodic Activations: The model learns a continuous conditional mapping fθ(x; z) from sampled coordinates to signal values, encoding each target signal as a latent code z.For images, coordinates have dimension n = 2, RGB outputs have dimension m = 3, and the latent space has dimension d = 256.
  • 3.1. Modulated Periodic Activations: Its dual-MLP architecture combines a coordinate-based synthesis network with a ReLU modulation network that processes the target signal’s latent code.The model parameters are optimized using a domain-specific reconstruction loss over sampled coordinate–signal-value pairs.
  • 3.1.1. Synthesis Network: The synthesis network composes K hidden layers that recursively apply sine activations to spatial coordinates, producing a continuous mapping from a discrete signal’s domain to its output space.Sine activations are used because they are beneficial for modeling high-frequency signals [37].
  • 3.1.2. Modulation Network: The modulation network receives the latent code through skip connections at every layer and predicts per-layer modulation parameters αi for the synthesis network.These parameters modulate the amplitude of the synthesis network’s sine activations.
  • 3.1.2. Modulation Network: Because modulation parameters appear throughout the nested sine computation, latent codes also indirectly control the frequency and phase shift of subsequent-layer sinusoids.The paper illustrates the expressivity of this mechanism visually in Figure 4.

3.2. Expressivity of Modulation

Concatenating latent codes with input coordinates consistently fails with sine activations because the latent code only induces a first-layer phase shift. In contrast, the proposed modulator controls amplitude, frequency, and phase across all layers, yielding greater expressivity.

  • 3.2. Expressivity of Modulation: Concatenating [x, z] as input consistently fails with sine activations, despite being effective for ReLU-based signed-distance-field synthesis networks [31].The alternative rewrites the first layer using separate coordinate and latent-code weight submatrices.
  • 3.2. Expressivity of Modulation: The modulator controls amplitude, frequency, and phase at every synthesis layer, unlike concatenation-based conditioning, which restricts latent codes to a first-layer phase shift.Figure 4 visualizes the resulting difference in internal feature-map control across latent vectors.
  • 3.2. Expressivity of Modulation: Figure 4 shows that concatenation changes only second-layer signal phase, whereas the proposed modulator provides substantially greater control over internal feature maps.The comparison uses four different latent vectors with a randomly initialized synthesis network.

3.3. Local Functional Representations

The method decomposes each signal’s domain into a regular grid of tiles, representing each tile with a latent code and corresponding local continuous neural function. Overlapping tiles are blended with distance-based n-linear weighting to eliminate boundary discontinuities.

  • 3.3. Local Functional Representations: Each tile receives a latent code z_i, yielding a codebook {z_i} and corresponding local neural functions that represent the entire signal.
  • 3.3. Local Functional Representations: Overlapping tiles are used to eliminate visual discontinuities at tile boundaries.
  • 3.3. Local Functional Representations: During evaluation, tile contributions are blended using n-linear weighting based on distance to tile centers; Figure 5 illustrates the bilinear case.

3.4. Training procedure

The model supports auto-encoder and auto-decoder training modes. Auto-encoding is the default for continuous representations from discrete signals, while auto-decoding jointly optimizes network parameters and latent codes.

  • 3.4. Training procedure: The method provides two training modes: auto-encoding estimates latent codes with a discrete encoder, whereas auto-decoding randomly initializes and optimizes latent codes alongside network parameters as in [31].
  • 3.4. Training procedure: Unless otherwise specified, experiments use auto-encoding to construct continuous representations from discrete input signals through an auxiliary encoder network.This configuration supports applications including spatial super-resolution, video frame interpolation, and reconstruction from sparse lightfield or compression samples.
  • 3.4. Training procedure: Shape reconstruction uses auto-decoding, jointly optimizing network parameters θ and training-signal latent codes without the optional encoder, following [31].For unseen signals, a new latent code z is sampled and optimized with the training objective.

4. Experiments

Experiments evaluate the model’s global and local functional representations across images, videos, and 3D shapes. Results show strong generalization, continuity, high-frequency shape encoding, and high-resolution reconstruction across these domains.

  • Experimental scope: The experiments cover global representations for images, videos, and 3D shapes, plus local representations that partition signals into grid tiles with latent codes.Global experiments encode each signal with one latent code, while local experiments reconstruct tiled signals from estimated latent codes.
  • Global image experiments: On CelebA and CIFAR-10, the model is trained as an auto-encoder and evaluated using PSNR at native 1× resolution and finer 2× sampling for continuity.Images are resampled to 32 × 32 for training, with convolutional encoders estimating latent codes; CelebA and CIFAR-10 use separate train/test splits.
  • Global image experiments: FFN continuity depends critically on its scaling parameter σ, while SIREN+ and HyperNet-SIREN fail to generalize robustly on diverse CIFAR-10 images.The recommended σ = 10 causes a drop in 2× PSNR, and SIREN+ does not converge on CIFAR-10; HyperNet-SIREN shows similar problematic behavior.
  • 3D shape experiments: For ShapeNet cars, the model encodes signed distance fields more accurately than conditional ReLU-MLPs, recovering finer high-frequency geometric details.The auto-decoder experiment samples 500K points per shape and compares bidirectional Chamfer distance across 3514 car shapes.
  • Local functional representations: The local model reconstructs high-resolution images and videos from tiled latent codes, using overlapping 32 × 32 image tiles and 32 × 32 × 7 video tiles.Div2K images and high-resolution videos are encoded with pretrained local models; the video model is trained on Vimeo-90k and tested on collected 1920 × 1080 videos.

5. Conclusion

The proposed method represents signals with MLPs by partitioning their domains into tiles, enabling high-quality reconstructions of images, videos, and shapes. It uses sine activations for broader frequency coverage and latent-code conditioning because ReLU activations miss high frequencies and concatenation hinders expressivity.

  • The method partitions signal domains into tiles, simplifying signals locally and enabling high-quality MLP reconstructions of images, videos, and shapes.
  • Sine activations cover a wider frequency spectrum than ReLU activations, which fail to reconstruct high-frequency signal components.
  • Local models require latent-code conditioning, while concatenating latent codes with the input hinders expressivity.
Loading 2104.03960v1…