From data to functa: Your data point is a function and you can treat it like one

Emilien Dupont; Hyunjik Kim; S. M. Ali Eslami; Danilo Rezende; Dan Rosenbaum

Source-linked AI summary

From data to functa: Your data point is a function and you can treat it like one

Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, Dan Rosenbaum

arXiv:2201.12204v3Jan 28, 2022cs.LG

TL;DR

The paper asks how deep learning can operate on continuous neural representations rather than discrete arrays. It introduces functa—neural functions treated as data—and a framework using compact modulations across modalities and tasks. The approach shows desirable properties across generative modeling, imputation, novel view synthesis, and classification, while remaining less performant than conventional array-based deep learning and subject to representation and meta-learning limitations.

Problem
Discrete arrays impose grid-based representations on often-continuous signals, motivating a framework for treating neural functions themselves as data.
Method
The framework fits an INR to each datapoint, represents functa with latent modulations, and trains downstream models directly on these function representations.
Results
The framework supports generative modeling, data imputation, novel view synthesis, and classification across images, 3D shapes, NeRF scenes, and manifold data, with desirable functional properties.
Takeaways & Limitations
Functa provide a unified continuous data representation that can be used across modalities and downstream machine-learning tasks.
Takeaways & Limitations
The method does not yet outperform conventional array-based deep learning, and its shared modulation representation removes spatial structure while MAML adds memory and training constraints.

Abstract

from arXiv · show

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using an implicit neural representation, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as functa, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification. Code: https://github.com/deepmind/functa

1. Introduction

The paper replaces discrete array representations with continuous neural functions and treats those functions as data points for downstream learning. It introduces functa and a framework for creating and learning from them across modalities.

Discrete arrays represent measurements on grids, although the underlying signals are often continuous.
Implicit neural representations map spatial coordinates to measurements such as image RGB values using neural networks.
The framework fits an implicit neural representation to each datapoint, producing functions called functa.
Functa are then treated as data points for downstream tasks rather than as components of the task model.
The proposed framework addresses functaset creation and downstream learning across images, voxels, NeRF scenes, and manifold data.

2. Functa: Data points as INRs

Functa are neural functions fitted to individual datapoints and used as continuous, resolution-flexible dataset elements. This representation supports generic downstream processing across modalities while decoupling function fitting from task learning.

An INR maps coordinates x ∈ X to features f ∈ F, and is fitted by minimizing mean squared error over observed coordinate-feature pairs.
Each fitted function fθ can represent an individual datapoint such as an image, and is called a functum when treated as dataset data.
Scaling: Functa generally scale more gracefully with resolution than arrays, reducing memory and compute pressure for downstream neural-network training.
Moving away from fixed resolution: Functa can represent data at varying resolutions, including images with different shapes and resolutions, point clouds, and irregular meshes.
Signals that are inherently difficult to discretize: For NeRFs and manifold data, functa express signals continuously and bypass grid discretization during downstream learning.
Easing downstream task: Decoupling functa creation from downstream learning simplifies tasks such as generative modeling of NeRF scenes.

3. Functasets: Datasets of INRs

The framework represents functa with compact modulations applied to a shared SIREN base network, then uses meta-learning to create large functasets efficiently. Latent modulations offer a strong reconstruction–dimensionality tradeoff, while fitting requires only a few gradient steps.

3.1. Functa as MLP modulations: SIREN modulations replace large parameter vectors, using a shared base network for common structure and low-dimensional vectors for datapoint-specific variation.The paper uses shifts rather than shifts and scales because shifts perform similarly with half the representation size.
3.1. Functa as MLP modulations: Latent modulations are linearly mapped to shift modulations and provide the best tradeoff between reconstruction accuracy and modulation dimensionality.The paper typically uses dimensions 256 or 512, producing visually close reconstructions while remaining orders of magnitude smaller than array representations.
3.2. Meta-learning functa: Creating large functasets is important because individually fitting several thousand NeRF scenes would be prohibitively expensive.The method addresses both efficient representation and scalable conversion from datapoints into modulation datasets.
3.2. Meta-learning functa: Meta-learning optimizes a shared base-network initialization so each functa can be fitted by updating only its modulation in a few gradient steps.The outer loop updates base-network weights, while the inner loop updates modulations, corresponding to learning a subset of weights with MAML or CAVIA.
3.2. Meta-learning functa: After meta-learning, the paper fits modulation vectors for training and test datapoints, finding that three gradient steps work well for both meta-learning and modulation fitting.The procedure also makes the modulation space smooth because datapoints are reached through a handful of gradient steps from the shared initialization.

4. Deep learning on functa

The paper trains downstream models directly on modulation representations, decoupling functa fitting from task learning. This supports generative modeling and inference applications while reducing memory demands and enabling reuse across tasks.

4. Deep learning on functa: Deep learning models operate directly on modulation datasets, making the pipeline more memory-efficient than joint learning and allowing the same functa to support different tasks.Although modulations are arrays, they parameterize functions rather than representing the original measurements directly.
4. Deep learning on functa: Generative modeling is the primary focus because it is broad and supports inference applications including imputation and novel view synthesis.Classification is also evaluated to demonstrate coverage of both generative and discriminative tasks.
4. Deep learning on functa: Normalizing flows and diffusion are used for generative modeling, while VAEs and GANs are identified as sensible alternatives not explored in the paper.Transformers were explored as an autoregressive example but underperformed.
4. Deep learning on functa: Flows and diffusion are more efficient on modulations because their layer dimensionality, and thus typical model size, scales with input dimensionality.Modulations also provide a direct representation for NeRF scenes, for which an obvious array representation is unavailable.
4. Deep learning on functa: MAP inference combines a learned modulation prior with functa reconstruction likelihoods, enabling imputation from partial observations.For voxel-grid imputation, the optimized modulation is queried through its INR to fill the unobserved grid portion.
4. Deep learning on functa: Small MLP classifiers reach high test accuracy in a few thousand iterations, requiring minutes of wall-clock time on a single GPU.

5. Related Work

Prior work models distributions or manifolds of implicit neural representations and processes neural-network data, but differs in task coverage, modality support, or representation. The paper positions functa as a functional representation for broad downstream learning.

Generative models of functions: Earlier studies modeled functional representations of MNIST, conditional distributions of functions and images, and distributions of image INRs using adversarial methods.
Generative models of functions: GASP learns modality-agnostic INR distributions across images, 3D shapes, and manifold data, but its GAN training is unstable and does not apply to NeRF scenes.
Generative models of functions: GEM learns modality-agnostic INR manifolds across tasks, but its embeddings are not used to train generative models or classifiers.
Multimodal architectures: The Perceiver uses a shared architecture across modalities, but prior applications process array representations rather than functa.
Diffusion and flows in latent space: Other work applies diffusion or flow priors in VAE latent spaces, whereas this paper applies these models to functional representations.
Deep learning on neural networks: Related studies use neural networks as inputs to other neural networks, including predicting classifier accuracies or characteristics from vectorized weights.

6. Experiments

The framework is evaluated on images, voxels, NeRF scenes, and manifold data by fitting functas and applying generative modeling, imputation, novel-view synthesis, and classification.

Experimental setup: The experiments cover CelebA-HQ images, ShapeNet voxels, SRN Cars NeRF scenes, and ERA5 temperature data on a manifold.The framework is tested across four modalities using functaset representations.
Meta-learning: Three gradient steps fit modulations accurately across modalities, using 64–512 modulations with accuracy generally increasing as modulation size grows.SRN Cars is the exception, showing similar reconstruction quality across modulation sizes, possibly because of the basic rendering scheme.
Images: DDPM trained on 256-dimensional CelebA-HQ modulations produces realistic samples with FID 40.4, while its samples are more coherent but slightly blurrier than GASP.Nearest-neighbor checks are used to assess whether the model memorized the training set.
Voxels: The proposed voxel generative model produces coherent and realistic ShapeNet chair samples, whereas the compared baselines are less consistent.The comparison includes Occupancy Networks, a signed-distance GAN, and GASP.
NeRF scenes: DDPM generates plausible SRN Cars scenes with FID 80.3 versus 36.7 for π-GAN, while training directly on modulations is less expensive than backpropagating through volume rendering.π-GAN samples are sharper, but the authors suggest improved rendering could mitigate this difference.
NeRF scenes: Using an NSF prior enables smooth shape-and-texture interpolation and realistic novel-view synthesis from an occluded test view.Without the prior term, fitting only the occluded view produces an unrealistic scene.
Classification: The classification MLP performs similarly or better than the 3D CNN baseline while training in under 10 minutes on one GPU with batch size 1024.The CNN baseline is memory intensive and runs on eight devices.

7. Conclusion, limitations and future work

The paper concludes that functa can support multiple learning tasks across modalities, while identifying limitations from shared representations, MAML-based construction, and representation-choice criteria.

Conclusion: The framework creates functasets at scale and supports generative modeling, data imputation, novel-view synthesis, and classification across diverse modalities.The conclusion presents these capabilities as demonstrated within the proposed framework.
Limitations and future work: A shared functa framework cannot use modality-specific inductive biases, because modulation storage removes spatial structure and favors general MLPs.Spatial functional representations are proposed as a future direction for enabling locality and translation invariance.
Limitations and future work: The MAML-based functaset construction inherits large memory use and occasional instability from double-loop optimization.MAML can also constrain modulations to remain within a few gradient steps of the meta-learned initialization, potentially limiting reconstruction accuracy on complex datasets.
Limitations and future work: Choosing architectures and modulation dimensionality by compressibility may be suboptimal for downstream tasks.Task-specific metrics or end-to-end learning could improve task alignment but may reduce reconstruction quality, increase memory use, and require separate INR retraining.
Future work: Progress in INR architectures, activation functions, memory efficiency, and NeRF rendering is presented as directly applicable future improvement.The paper specifically mentions hybrid representations and more sophisticated NeRF techniques.

A.1. Datasets

The appendix describes the datasets, coordinate preprocessing, SIREN implementation, and latent-modulation architecture used to construct functasets.

Datasets: CelebA-HQ uses 64×64 images with a 27,000/3,000 train/test split and normalized pixel coordinates and intensities in [0, 1].Coordinates correspond to pixel centers in the unit square.
Datasets: ShapeNet starts from 128^3 voxels, downsamples to 64^3, and applies 50-fold independent axis rescaling for augmentation.The downsampling threshold balances shape preservation and smoothness, while augmentation helps prevent downstream overfitting.
Datasets: ERA5 temperature grids are converted from latitude-longitude coordinates to 3D Cartesian inputs and downsampled from 721×1440 to 181×360.The dataset spans observations from 1979–2020 and uses a 9676/2420 train/test split.
Datasets: SRN Cars contains 2458 training and 703 test scenes, with 50 training views and 251 test views per scene.Each view is 128×128, and camera pose and focal length determine sampled 3D ray coordinates supplied to SIREN.
SIREN and modulations: SIREN uses sinusoidal layers with ω0 fixed to 30, except ω0 = 5 for SRN Cars, and uses specialized weight initialization.The final layer is linear, with 0.5 added because targets are normalized to [0, 1].
SIREN and modulations: LatentModulatedSIREN stores a latent vector φ that is linearly mapped into shift modulations applied to a shared SIREN base network.The latent vector is initialized at zero and fitted with a few inner-loop gradient steps.
SIREN and modulations: Architectures are selected by sweeping depths {10, 15, 20} and widths {256, 384, 512} using reconstruction performance after meta-learning.The chosen metric is test PSNR or voxel accuracy, depending on the dataset.

A.3. Meta-learning Functa

This section describes meta-learning and generative modeling components for functa, including compact modulation representations, efficient fitting, and NSF/DDPM models.

Meta-learning Functa: Training can become unstable for larger SIRENs; reducing the outer-loop learning rate helps, whereas lowering the inner-loop rate has little effect.
Meta-learning Functa: Meta-learning uses per-parameter learning rates for modulations, with several inner-loop steps rather than the single step used in the original meta-SGD.The learning rates are initialized from U[0.005, 0.1] and clipped to (0, 1).
Meta-learning Functa: Meta-SGD noticeably improves performance for shift modulations but makes little difference for latent modulations.
Neural Spline Flows: A Neural Spline Flow models modulation data through invertible coupling and PLU linear layers, then samples by transforming Gaussian noise.The coupling transform uses an MLP conditioner to produce neural-spline parameters, while PLU layers preserve invertibility and permit efficient inversion.
Neural Spline Flows: PLU linear layers noticeably improve both training and test likelihood.
Denoising Diffusion Probabilistic Models: DDPMs add Gaussian noise through a fixed forward process and learn a neural reverse process that denoises noisy modulation data.The implementation predicts noise with ϵθ and samples by reversing the process from a Gaussian xT.
FID scores: FID was not a meaningful perceptual metric for CelebA-HQ in this model, because perceptually improved samples could receive worse later-stage scores.The functaset itself scored 28.4 for 256-dimensional modulations and 17.2 for 512-dimensional modulations despite being perceptually close to the original dataset.

B.1. Perturbation analysis

Perturbation analysis shows that earlier SIREN modulation layers have larger effects and encode frequency-structured, broadly shared image changes, while later layers produce smaller edge-aligned effects.

B.1. Perturbation analysis: Initial-layer units with high RMSE produce roughly diagonal periodic features with differing periodicities and directions.These units have RMSE greater than 5.
B.1. Perturbation analysis: Medium-RMSE units in the initial hidden layer remain periodic but have lower frequencies, indicating frequency diversity within that layer.The medium-RMSE range is [2, 3].
B.1. Perturbation analysis: Perturbations in later layers are non-periodic or smoother and tend to align with image edges, with smaller image-space magnitudes.Layer 1 and layer 2 perturbations are smaller than those from the initial hidden layer.
B.1. Perturbation analysis: Perturbation RMSE generally decreases with SIREN depth, suggesting earlier-layer modulations play a greater role in dataset-wide variation.
B.1. Perturbation analysis: A given modulation unit produces similar error patterns across images, resembling a coefficient of a shared but non-local basis function.The non-locality may challenge downstream architectures that rely on locality, such as CNNs and Transformers.

B.2. Alternative Modulation architectures

The paper compares modulation architectures and finds that shift-only and latent modulations offer stronger or more compact alternatives than several scale-inclusive and latent-concatenation designs.

B.2. Alternative Modulation architectures: Using both scale and shift is redundant because scale values are usually optimized to 1, while scale-only modulations lose about 5dB in image reconstruction PSNR.Shift-only modulations achieve similar performance to scale-plus-shift with half the representation size.
B.2. Alternative Modulation architectures: SubsetModulatedSiren uses shifts only in early layers but performs slightly worse than latent modulations.The authors expect combining subset and latent modulations could improve performance, though they have not tried it.
B.2. Alternative Modulation architectures: ModSine yields lower reconstruction accuracy than latent modulations at the same modulation dimension.
B.2. Alternative Modulation architectures: Concatenating the latent vector to the SIREN or positional-encoded ReLU MLP input performs worse than latent modulation.The best concatenated-latent model was a SIREN with test PSNR 23.0dB for latent dimension 256 on CelebA.

C. Negative Results

The appendix reports negative results that clarify trade-offs among architectures, fitting procedures, modulation dimensionality, and inner-loop computation. Several seemingly more powerful choices reduced reconstruction or sample quality, while random pixel subsampling remained preferable for SRN Cars.

Architecture choices: Transformers underperformed MLP-based conditioners in both normalizing flows and diffusion models.The authors also report that autoregressive Transformer samples were perceptually worse than diffusion samples.
Meta-learning choices: First-order MAML caused an approximately 10dB reduction in image reconstruction PSNR.The method was tested as a memory-saving alternative that avoids backpropagation through the inner loop.
Meta-learning choices: Using 50 fitting steps improved image reconstruction by around 2dB but produced less-smooth modulation spaces and worse generative sample quality.For chairs, performance deteriorated beyond three steps, likely because the fixed inner-loop learning rate was too high.
Meta-learning choices: Four or five inner-loop steps for scenes increased memory use and reduced subsampling without noticeably improving PSNR.The authors suggest minimal rendering was likely the bottleneck in this setup.
Generative modeling: Flow on 256-dimensional image modulations produced noticeably worse samples than DDPM, whereas flow outperformed diffusion on ShapeNet voxels.The authors conjecture that the preferred generative model depends on the dataset rather than the modality.
Generative modeling: For voxels, 512-dimensional modulations yielded much worse flow sample quality than 256-dimensional modulations.The authors conjecture that fewer modulations create a smoother modulation space despite lower reconstruction accuracy.
Data subsampling: Bounding-box pixel subsampling did not outperform random subsampling for SRN Cars, while including white background pixels improved PSNR.The improvement was attributed to more accurate recovery of the white background.
Meta-learning choices: The auto-decoder approach was unstable and produced notably worse image PSNR than meta-learning.The authors report that it remains unclear why auto-decoding works for signed-distance functions but not for other modalities.

D. Further Results

Further experiments examine reconstruction fidelity, sample diversity and resolution, temperature effects, classification, and additional modality-specific results. The figures and accompanying discussion extend the framework’s evaluation across images, shapes, and scenes.

Reconstruction: Reconstructions from 256-dimensional modulations are reasonably close to originals, though fine detail can be missed for complex shapes and scenes.The comparison covers the data modalities used to compute the reconstruction results in Table 1.
Image generation: CelebA-HQ diffusion samples are reasonably different from nearest training examples, indicating the model did not simply memorize the training set.Nearest neighbours are measured by L2 distance in modulation space.
Image generation: Directly rendering a 256-dimensional face modulation at 512×512 produces crisper results than upsampling a 64×64 rendering with bicubic interpolation.The difference is especially visible around the jawline and teeth.
Shape generation: Lower flow temperatures make ShapeNet chair samples more globally coherent but reduce diversity.The reported rows use temperatures 1.0, 0.9, 0.8, and 0.7.
Shape generation: The evaluation includes uncurated flow samples for ShapeNet Chairs and class-conditional samples for ShapeNet’s 10 classes.Additional figures show interpolations in modulation space and flow latent space.
Classification: Classification on VAE and AE latents reached best test accuracies of 91.9 ± 0.3% and 93.2 ± 0.2%, respectively.These baselines used 3D CNN encoders and decoders with 256-dimensional latents.
Classification: Larger classifiers improved performance for both functa-based MLPs and 3D CNNs, with bigger 3D CNNs outperforming bigger MLPs but requiring much more capacity.Training and per-class accuracy trajectories are shown in Figure 24.
Additional evaluations: Additional figures provide samples for CelebA-HQ, ShapeNet, and SRN Cars, plus chair imputation and novel-view synthesis results.The appendix also includes comparisons of modulation-space and flow-latent interpolations and direct high-resolution rendering.