Generative Models as Distributions of Functions

• GASP learns distributions over continuous functions using a hypernetwork and adversarial training with a discriminator on coordinate-feature sets.Source: §1 · p. 1
• Grid-based generative models can become intractable at high resolutions, scaling quadratically or cubically with resolution.Source: §1 · p. 1
• Its continuous-space formulation is independent of resolution and discretization, and can model functions on manifolds where discretization is difficult.Source: §1 · pp. 1–2
• Across images, 3D shapes, and climate data, the same model learns rich function distributions and generates sharp, realistic samples.Source: §1 · p. 2

• A function representation maps coordinates to features, allowing an image to be represented as fθ : R2 → R3.Source: §2 · p. 2
• Function representations scale with signal complexity rather than resolution, unlike grid storage whose memory grows quadratically for images and cubically for voxel grids.Source: §2 · p. 2
• Random Fourier features address the low-frequency bias of function representations by encoding coordinates before passing them through the MLP.Source: §2 · p. 2
• With RFF, the image representation reproduces high-frequency detail, whereas a ReLU MLP without RFF fails to capture it.Source: §2 · p. 2, Figure 2 · p. 2

3.1 Data Representation

• Data points are coordinate-feature sets, allowing function distributions to be learned without access to the ground-truth functions representing the data.For images, these sets comprise pixel locations and their corresponding RGB values.Source: §3.1 · p. 3
• Randomly sampling K coordinate-feature pairs removes direct dependence on data resolution, and manifold data is supported when a coordinate system can be defined.Subset training can reduce memory use for 3D shapes compared with passing an entire voxel grid.Source: §3.1 · p. 3

3.2 Function Generator

• A hypernetwork maps latent z to weights θ for a fixed-structure MLP, which is evaluated at encoded coordinates to generate features.The architecture diagram shows gφ producing the weights of fθ, which maps coordinates to features.Source: §3.2 · p. 3, Figure 3 · p. 3

3.3 Point Cloud Discriminator

• The discriminator must be permutation-invariant while accounting for distances between coordinates; PointNet and DeepSets were inadequate for complex function distributions.The authors argue that coordinate-space distance is crucial for learning complex distributions.Source: §3.3 · p. 4
• PointConv uses coordinate-MLP filters and metric-based nearest-neighbor sets to define continuous-space convolution.Its layers are described as translation equivariant and permutation invariant by construction.Source: §3.3 · p. 4, Figure 4 · p. 4

3.4 Training

• Each training step samples and probes a fake function, converts real data to a point cloud, then discriminates, with an R1-style penalty on feature gradients.The generator, discriminator, and loss operate directly on continuous point clouds.Source: §3.4 · p. 4, Figure 5 · p. 5

3.5 How Not to Learn Distributions of Functions

• PointNet/DeepSets were unsuitable for complex function distributions, and the tested approaches worked on MNIST but not CelebAHQ.The authors attribute the set discriminators’ limitations in part to their failure to use coordinate-space metrics and report that their Lipschitz constants can destabilize GAN training.Source: Set discriminators · p. 5
• The paper reports that the auto-decoder objective is not appropriate for generative modeling.The passage notes that auto-decoders have been used extensively in 3D computer vision and excel at a variety of tasks.Source: Set discriminators · p. 5

Implicit representations

• Implicit representations have been used for evolutionary algorithms, MNIST generation, 3D geometry, inverse rendering, dynamic and 3D scenes, and superresolution.The passages cite Ha (2016), Park et al. (2019), Sitzmann et al. (2019), Mildenhall et al. (2020), and other work across these applications.Source: §4 · p. 5

Continuous models of image distributions

• Neural processes model image distributions as functions but focus on uncertainty quantification and meta-learning; other continuous image models use gradient-based or adversarial approaches.Concurrent adversarial methods for high-frequency image representations use standard convolutional discriminators, restricting them to grid-based data.Source: §4 · p. 5

Continuous models of 3D shape distributions

• Prior 3D shape distribution methods include VAEs, GANs on autoencoder embeddings, auto-decoders, and set discriminators, which the paper says limit complex function distributions.The cited methods include Mescheder et al. (2019), Chen and Zhang (2019), Park et al. (2019), and Kleineberg et al. (2020).Source: §4 · pp. 5–6
• For NeRF scenes, prior methods include GRAF, GIRAFFE, and pi-GAN; the authors claim GASP is the first to demonstrate function distributions across images, 3D shapes, and manifold data.The passage describes differences in these models’ generator or discriminator designs and limits the first-work claim with “to the best of our knowledge.”Source: §4 · p. 6

5.1 Images

• GASP produces sharp, realistic CelebAHQ images at both 64×64 and 128×128, though artifacts and occasional poor samples remain, especially at 128×128.Figure 6 shows samples from five batches, with the best batch selected by visual inspection.Source: §5.1 · p. 6, Figure 6 · p. 6
• A function trained only on 64×64 images can be evaluated at 256×256, producing convincing four-times-higher-resolution images.Figure 7 compares the original resolution, 4× resolution, and bicubic upsampling.Source: §5.1 · p. 6, Figure 7 · p. 7
• On CelebAHQ 32×32, GASP outperforms the continuous baselines quantitatively and produces sharper, more diverse, and more plausible samples, though it trails specialized convolutional GANs.The ConvNP baseline was trained on CelebA rather than CelebAHQ and uses a different crop.Source: §5.1 · p. 7, Table 1 · p. 7, Figure 8 · p. 7

5.2 3D Scenes

• For 3D training, uniformly subsampling 4,096 of each chair’s 32,768 voxel points saves memory and permits larger batches on limited hardware.The paper says this is not possible with convolutional discriminators and lets training operate independently of data resolution.Source: §5.2 · p. 7
• Sampling the continuous 3D representation at higher resolutions produces clean, smooth meshes.Figure 10 illustrates evaluating the same function at different resolutions.Source: §5.2 · p. 7, Figure 10 · p. 7
• GASP produces coherent, fairly diverse ShapeNet chair samples comparable to occupancy-network VAEs and DeepSDFs trained with set discriminators.The comparison is shown in Figure 11.Source: §5.2 · p. 8, Figure 11 · p. 8

5.3 Climate Data

• Climate data are modeled as 8510 independent temperature grids, each represented by a function from the sphere to temperature; temporal correlations are not modeled.Latitude-longitude coordinates are converted to spherical coordinates before being passed to the function representation.Source: §5.3 · p. 8
• GASP generates plausible climate samples and smooth interpolations, remaining continuous across the sphere unlike the grid-trained baseline.The comparison is between GASP and a baseline trained on latitude-longitude grids.Source: §5.3 · p. 8, Figure 12 · p. 8

6 SCOPE, LIMITATIONS AND FUTURE WORK

• GASP does not outperform state-of-the-art specialized image and 3D models, and training can be unstable, particularly with point subsampling.On CelebAHQ, fewer points per example reduce image quality; the 3D model can collapse to simple chair shapes despite more complex shapes in the data.Source: §6 · p. 8
• Point-cloud neighborhood determination can be expensive with many points, while faster neighbor search may mitigate the cost.The authors conjecture that distant nearest neighbors under point subsampling may contribute to unstable training.Source: §6 · pp. 8–9
• The current model cannot scale to truly large datasets such as room-scale 3D scenes, although the coordinate-and-feature formulation may be a first step toward scaling generative models.The authors also identify geospatial, geological, meteorological, and molecular data as possible future applications.Source: §6 · p. 9

7 CONCLUSION

• GASP learns generative models in continuous spaces by learning distributions over functions, and experiments show rich function distributions for images, 3D shapes, and climate data.The authors hope this could eventually enable generative modeling where discretization is expensive, as in 3D, or difficult, as with non-Euclidean data.Source: §7 · p. 9

A EXPERIMENTAL DETAILS

• All experiments were implemented in PyTorch and trained on one 2080Ti GPU with 11GB of RAM; reproducibility code was provided.The supplement also specifies discriminator and optimizer settings for the GASP experiments.Source: A EXPERIMENTAL DETAILS · p. 13, §A.2 · p. 13

A.1 Single Image Experiment

• The supplement also gives the single-image MLP and random Fourier feature settings used to reproduce Figure 2.The MLP had two 256-unit hidden layers and trained for 1,000 iterations; the encoding used m = 256 and σ = 10.Source: §A.1 · p. 13

A.2 GASP Experiments

• A three-layer, 128-unit function MLP and a 64-dimensional latent were used across image, 3D shape, and climate experiments, with a two-layer hypernetwork.The hypernetwork layers had dimensions 256 and 512; the function and discriminator used LeakyReLU, with tanh on the function output.Source: §A.2 · p. 13

A.3 Things We Tried That Didn’t Work

• The authors found thinner, deeper function networks more expressive per weight, reducing the size burden on the hypernetwork that must output those weights.A two-layer, 256-unit function MLP worked poorly on more complex datasets compared with three 128-unit layers.Source: §A.3 · p. 14
• Batch normalization stabilized training, and deep weight MLPs were needed for expressive PointConv discriminators; learning Fourier frequencies did not boost performance and generally slowed training.The authors hypothesize batch normalization addresses large initial PointConv outputs.Source: §A.3 · p. 14

A.4 ERA5 Climate Data

• Climate data comprised 12,096 monthly-hourly global temperature grids from 1979–2020, subsampled from 721 × 1440 to 46 × 90.The data came from ERA5 surface temperatures at 2 m, with measurements at 0.25-degree spacing before subsampling.Source: §A.4 · p. 15

A.5 Quantitative Experiments

• The convolutional baseline FID scores are not directly comparable because they were computed on CelebA rather than CelebAHQ; the authors note convolutional GANs would also outperform their model on CelebAHQ.FID was computed with pytorch-fid from 30,000 samples at each of CelebAHQ 64 × 64 and 128 × 128.Source: §A.5 · p. 15

A.6 Rendering 3D Shapes

• For 3D visualization, sampled functions were evaluated on a high-resolution grid, thresholded at 0.5, and converted to meshes with marching cubes.The resulting meshes were rendered with PyTorch3D.Source: §A.6 · p. 15

A.7 Baseline Experiments

• The auto-decoder baseline trained for 200 epochs with batch size 64 and learning rate 1e-4 for both latents and generator parameters.Latent initializations were sampled from N(0, 0.012), and the best samples were chosen by visual inspection.Source: §A.7 · p. 15

B.1 Auto-decoders

• Because auto-decoder latents are optimized samples rather than parameters of the prior, maximizing the prior term encourages small norms without ensuring the latents follow that prior.Consequently, sampling from the prior after training will not work, according to the authors.Source: §B.1 · p. 16
• Auto-decoders produced plausible MNIST samples but not good results on CelebAHQ, while their learned latent vectors were unlikely to follow the prior used for generation.The low-variance prior trick sufficed for MNIST but not the more complex, high-frequency CelebAHQ dataset; Figure 13 shows samples from both datasets.Source: §B.1 · p. 16, Figure 13 · p. 16

A.7 Baseline Experiments

• The authors caution that auto-decoders may be unsuitable for generative modeling, while noting their use in 3D shape embedding and inverse rendering.The baseline setup used CelebAHQ 32 × 32 and the same generator as the 64 × 64 experiments.Source: §A.7 · p. 15, §B.1 · p. 16

B.2 Set Discriminators

• Set discriminators generated plausible MNIST samples but failed on CelebAHQ, including after extensive architecture and encoding variations; Figure 14 shows samples from both datasets.The tested set architecture was permutation invariant and based on PointNet/DeepSet, with Fourier feature encodings for coordinates and features.Source: §B.2 · pp. 16–17, Figure 14 · p. 17
• Set Transformers, which model set-element interactions with self-attention, did not improve performance in the authors’ experiments.The authors conjecture that explicitly accounting for the coordinate-space metric, as PointConv does, is crucial for learning complex neural distributions.Source: §B.2 · p. 17

B.3 The Lipschitz Constant of Set Discriminators

• Set discriminators and random Fourier feature encodings can have very large Lipschitz constants, which the authors present as theoretical evidence that set discriminators may be ill-suited to function-distribution generation.The paper motivates limiting discriminator Lipschitz constants as important for stable GAN training.Source: §B.2 · p. 17, §B.3 · p. 17

B.3.1 Architecture

• With fixed RFF encodings, gradient penalties can reduce the Lipschitz constants of the MLPs but not those of the coordinate and feature encodings.If the encoding is learned, its Lipschitz constant could also be penalized.Source: §B.3.1 · p. 17

B.3.2 Lipschitz Constant of Random Fourier Features

• High-frequency detail requires a large Lipschitz constant, whereas stable GAN training requires a low one; the tested frequency matrices had spectral norms in the hundreds.The authors report that these norms were too large for stable GAN training in their settings.Source: §B.3.2 · p. 18

C.1 Prerequisites

• The proof preliminaries define spectral and Lipschitz norms and establish inequalities for matrix concatenation, sums of identical functions, and concatenated functions.These lemmas supply intermediate bounds used in the later discriminator analysis.Source: §C.1 · p. 18, §C.1.1 · p. 18, §C.1.3 · p. 19

C.2 Lipschitz Constant of Fourier Feature Encoding

• The Fourier feature encoding’s derivative is expressed through the frequency matrix, yielding a Lipschitz constant proportional to its spectral norm.The proof derives this by differentiating the cosine and sine components and bounding their diagonal factors.Source: §C.2 · pp. 20–21

C.3 Lipschitz Constant of Set Discriminator

• The set-discriminator proof treats each coordinate-feature pair as an encoded set element and bounds the discriminator’s Lipschitz constant using the encoding and aggregation components.The proof identifies ρ as mapping the encoding to the probability that a sample is real.Source: §C.3 · pp. 21–22

D FAILURE EXAMPLES

• The failure examples include CelebAHQ 64 × 64 samples generated from 50% and 75% pixel coverage.Figure 16 is described as showing unrealistic and incoherent generated samples.Source: Figure 15 · p. 23, Figure 16 · p. 23

E.1 Additional Evaluation on ERA5 Climate Data

• GASP samples are described as largely indistinguishable from real ERA5 samples.Figure 22 places GASP samples beside training data.Source: §E.1 · p. 23, Figure 22 · p. 26
• Latent interpolations between GASP samples illustrate a smooth data manifold, helping check that the model has not memorized training samples.Figure 23 shows the function-space interpolations.Source: §E.1 · p. 23, Figure 23 · p. 26
• GASP's sampled temperature distribution roughly matches the test-set distribution.Figure 24 compares the two temperature distributions.Source: §E.1 · p. 23, Figure 24 · p. 27