Source-linked AI summary
Adversarial Generation of Continuous Images
Ivan Skorokhodov, Savva Ignatyev, Mohamed Elhoseiny
TL;DR
INR-based image generation faces unstable, parameter-intensive hypernetworks and costly dense high-resolution evaluation, while prior approaches did not scale beyond small MNIST-like datasets. The paper introduces factorized multiplicative modulation and multi-scale INRs to build INR-GAN, achieving state-of-the-art continuous image generation and reducing the gap with pixel-based GANs.
Problem
INR decoders are difficult to train, require many parameters, and are costly to evaluate at high resolution; previous INR image-generation attempts did not scale beyond small MNIST-like datasets.
Method
The paper proposes factorized multiplicative modulation for hypernetworks and a multi-scale INR architecture, combining them in the INR-GAN continuous image generator.
Results
INR-GAN outperforms existing continuous image generators by several times on large real-world datasets and greatly reduces the gap with pixel-based analogs.
Takeaways & Limitations
The proposed techniques enable solid state-of-the-art results in continuous image generation using INR representations.
Takeaways & Limitations
INR-based decoders may become sensitive to high-frequency coordinate features, producing random transparent texture artifacts that become more noticeable at higher resolutions.
Abstract
from arXiv · showhide
In most existing learning systems, images are typically viewed as 2D pixel arrays. However, in another paradigm gaining popularity, a 2D image is represented as an implicit neural representation (INR) - an MLP that predicts an RGB pixel value given its (x,y) coordinate. In this paper, we propose two novel architectural techniques for building INR-based image decoders: factorized multiplicative modulation and multi-scale INRs, and use them to build a state-of-the-art continuous image GAN. Previous attempts to adapt INRs for image generation were limited to MNIST-like datasets and do not scale to complex real-world data. Our proposed INR-GAN architecture improves the performance of continuous image generators by several times, greatly reducing the gap between continuous image GANs and pixel-based ones. Apart from that, we explore several exciting properties of the INR-based decoders, like out-of-the-box superresolution, meaningful image-space interpolation, accelerated inference of low-resolution images, an ability to extrapolate outside of image boundaries, and strong geometric prior. The project page is located at https://universome.github.io/inr-gan.
1. Introduction
The paper addresses the instability and computational cost of INR-based image decoders by introducing FMM and multi-scale INR architectures, then uses them to build INR-GAN for continuous image generation on real-world datasets. It also examines decoder properties including superresolution, interpolation, low-resolution inference, extrapolation, and geometric structure.
- Motivation: INRs represent images as continuous coordinate-to-RGB functions rather than cropped, quantized pixel arrays.An INR approximates F(p)=v, mapping a 2D coordinate to an RGB value.
- Challenges: INR-based decoders face unstable hypernetwork training and excessive computation when evaluating dense high-resolution coordinate grids.These difficulties previously limited continuous image generation to small MNIST-like datasets and low resolutions.
- Results: INR-GAN outperforms existing continuous image generators by several times on large real-world datasets, reducing the gap with pixel-based analogs.The introduction reports FID scores of 5.09, 4.96, and 16.32 on LSUN Churches 256^2, LSUN Bedrooms 256^2, and FFHQ 1024^2, respectively.
- Properties: INR-based decoders provide superresolution, meaningful image-space interpolation, faster low-resolution inference, boundary extrapolation, and geometric prior without additional training.The paper describes these as properties explored for INR-based decoders; superresolution and the listed features are presented as naturally arising capabilities.
- Contributions: Factorized multiplicative modulation enables large INR parameter generation while stabilizing hypernetwork training.The technique is proposed specifically as an architectural solution for hypernetworks.
- Contributions: Multi-scale INRs efficiently represent high-resolution images in INR form.The architecture addresses the cost of evaluating INRs over dense high-resolution grids.
2. Related work
Prior work established coordinate-based representations across images and other domains, but generative image applications remained limited in scale. INR-GAN-related work addresses this gap alongside broader hypernetwork and efficient-modeling research.
- Implicit Neural Representations: Coordinate-based neural representations model images and other signals as functions of coordinates rather than fixed pixel arrays.INRs became especially popular for continuous 3D shape representation and were also applied to textures, scenes, audio, and differential equations.
- INR-GAN properties: INR-based decoders are associated with superresolution, extrapolation beyond image boundaries, geometric priors, and meaningful interpolation.The paper presents these properties as part of its broader exploration of continuous image generation.
- Generative models + coordinates: Earlier generative INR experiments were limited to small MNIST-like datasets, while other approaches combined coordinates with conventional generative architectures.The paper distinguishes its hypernetwork-generated INR parameters from methods that feed latent codes directly into coordinate-based decoders.
- Hypernetworks: Hypernetworks generate parameters for other models, offering expressivity and compression but commonly suffering from training instability.Prior work applied hypernetworks to few-shot learning, continual learning, architecture search, generative modeling, and other tasks.
- Computationally efficient models: Efficient INR research uses low-rank factorization and treats INR layers as 1 × 1 convolutions whose weights are produced by a hypernetwork.The paper positions its decoder within work on computationally efficient models and coordinate-based representations.
3. Image Meta Generation
INR-GAN replaces a conventional convolutional generator with a hypernetwork-produced INR while retaining the StyleGAN2 training components. FMM makes parameter generation more practical, and multi-scale INRs reduce the cost of producing high-resolution images.
- 3. Image Meta Generation: INR-GAN takes latent code z and generates parameters θ for an INR, which is evaluated on a coordinate grid to produce an image.Fourier features embed coordinates, and the embedding matrix is also predicted from z.
- 3.2. Factorized Multiplicative Modulation (FMM): FMM generates large INR weight matrices through factorized modulation while preserving full-rank shared weights.The approach addresses the excessive output size of naive hypernetworks and avoids the performance loss observed with directly low-rank weight matrices.
- 3.2. Factorized Multiplicative Modulation (FMM): Sigmoid-bounded modulation stabilizes training by bounding activations and makes the FMM-based INR-GAN architecturally similar to StyleGAN2.The hypernetwork functions like a mapping network, while the INR functions like the synthesis decoder.
- 3.3. Multi-scale INRs: Multi-scale INRs split the decoder into blocks operating at progressively increasing resolutions, with only the final block operating at target resolution.This design avoids evaluating a large hidden network over the entire high-resolution coordinate grid.
- 3.3. Multi-scale INRs: Each multi-scale block contains 2-4 layers and receives Fourier coordinate features at its beginning, with low-resolution features replicated during upsampling.The first block starts at 64^2 resolution and resolution doubles across subsequent blocks until the target size.
4. Experiments
Experiments evaluate INR-GAN across image datasets, architectural ablations, and decoder properties. The proposed components improve scalability and generation quality while retaining continuous-coordinate capabilities.
- Experimental setup: Experiments use LSUN Bedrooms, LSUN Churches, and FFHQ, evaluating FID, parameter count, and multiply-accumulate operations.The datasets include resolutions from 128^2 to 1024^2, and FID statistics use 50k images.
- Experimental setup: The evaluation progressively compares latent-conditioned and hypernetwork-based INR generators before adding Fourier embeddings, FMM, and multi-scale architecture.The baselines differ in whether latent codes or generated parameters condition the INR, and whether factorization is used.
- Standard GAN training: Three times fewer MACs are used for INR-GAN inference than the compared convolutional decoder, although INR-GAN has three times more parameters and remains inferior to StyleGAN2 in performance.Most parameters come from the hypernetwork’s large output projection matrix.
- Ablations: Increasing FMM rank improves convergence, but gains diminish beyond the initial increases; removing its activation worsens FID.The ablations support low-rank factorization and bounded weight magnitudes as important design choices.
- Ablations: StyleGAN2 techniques and bilinear upsampling significantly improve INR-GAN performance, allowing it to surpass StyleGAN2 on LSUN Churches.These experiments are reported on LSUN Churches 256^2 and LSUN Bedrooms 256^2.
- Decoder properties: INR decoders generate meaningful interpolations, extrapolate beyond image boundaries, produce higher-resolution images on denser grids, and evaluate lower resolutions directly.These properties are tested through image-space interpolation, extrapolation, superresolution, and lower-resolution inference experiments.
5. Additional potential of INR-based decoders
INR-based decoders offer additional capabilities arising from coordinate-based, independently evaluated pixels. These include differentiable coordinate control, efficient inference, parallel computation, and reuse across domains.
- Additional potential: Coordinate positions pass through well-differentiable operations, enabling backpropagation through pixel locations.The paper discusses spatial transformer layers in generators and discriminator-produced coordinate grids for zooming into image regions.
- Additional potential: INR inference can be faster because it omits spatial context aggregation, with each layer behaving like a 1 × 1 convolution.Table 1 reports three times fewer MACs for the model’s inference process.
- Additional potential: Independent pixel generation allows INR decoders to compute pixels in parallel, unlike convolutional decoders whose local context limits parallel inference.This property is compared with the parallel generation behavior of non-autoregressive sequence models.
- Additional potential: The same decoder architecture can be applied to images, video, audio, and 3D shapes by changing the input coordinate representation.Examples include (x, y), (x, y, t), and (x, y, z) coordinates.
6. Limitations
INR decoders’ lack of spatially shared context is a core limitation, and multi-scale representations only partially alleviate it. Sensitivity to high-frequency coordinate features can also create texture artifacts at higher resolutions.
- Limitations: Not using spatially common context between neighboring pixels limits INR-based decoders.Multi-scale INRs ground representations in a shared low-resolution representation, but this does not fully resolve the issue.
- Limitations: High-frequency coordinate sensitivity may produce random transparent texture artifacts that become more noticeable at higher resolutions.The paper identifies this as a potential failure mode of INR-based decoders.
7. Conclusion
The paper develops continuous-image GAN generation with factorized multiplicative modulation and multi-scale INRs. It reports strong continuous-generation results while examining the decoders’ properties and limitations.
- Conclusion: Factorized multiplicative modulation and multi-scale INRs enable solid state-of-the-art results in continuous image generation.The paper presents these as its two principal architectural techniques for INR-based GANs.
- Conclusion: The study also explores attractive INR-decoder properties and discusses their future potential and limitations.The conclusion frames these properties and limitations as part of the paper’s broader investigation.
A. Additional implementation details
The experiments retain StyleGAN2’s discriminator, optimization, loss, and training settings while replacing only its generator with the INR-based generator. Configurations vary the number and resolution of multi-scale INR blocks across image resolutions.
- Training setup: Only the StyleGAN2 generator is changed; the discriminator, optimizers, losses, training settings, and other hyperparameters remain unchanged.Training uses non-saturating logistic loss and Adam with β1 = 0.0, β2 = 0.98, and ϵ = 1e−8.
- Training setup: The main experiments omit common ProGAN, StyleGAN, and StyleGAN2 training tricks, including path regularization, progressive growing, and noise injection.Additional ablations reimplemented the decoder with selected StyleGAN2 techniques.
- INR configurations: 256 × 256 experiments use two multi-scale INR blocks at resolutions 128 and 256, with four 512-dimensional layers per block.
- INR configurations: FFHQ 1024 × 1024 uses four blocks at resolutions 128, 256, 512, and 1024, with progressively smaller layer configurations in the later blocks.The first two blocks have three 512-dimensional layers; the 512-resolution block has two layers, and the final block has two 32-dimensional layers.
- Generator and evaluation: The generator has three nonlinear 1024-dimensional layers with residual connections, while the noise vector z has dimensionality 512.FID is computed from 50k images using the TensorFlow script from the BigGAN PyTorch repository.
B. Experiments details
The experiments examine INR-GAN extrapolation, keypoint predictability, and projection into latent space. Keypoint evaluation controls for differences in generated keypoint variability by also reporting KPL (random).
- Extrapolation: Extrapolation examples evaluate the model on coordinate grids [−1.5, 1.5]2 rather than the smaller grid used in the main figure.The larger grid is intended to demonstrate the extent of extrapolation.
- Keypoint prediction: Keypoint prediction fits a linear model from latent codes to keypoints after generating n = 104 latent codes, decoding images, and applying Super-FAN.
- Keypoint prediction: KPL (random) assesses whether keypoint variability differs between models by fitting the predictor on randomly permuted latent codes.The passage reports equal keypoint variability for both models under this metric.
- Keypoint prediction: Real FFHQ images are embedded into each model before testing keypoint prediction, avoiding evaluation based only on synthesized-image variability.
- Extrapolation: The larger extrapolation grid extends beyond the main experiment’s coordinate range to visualize behavior outside the usual image domain.Figure 10 reports rapidly decreasing quality beyond [−0.4, 1.4]2.
- Evaluation procedure: The KPL procedure uses Ntr = 104 training examples and Nts = 256 test examples.
B.3. Additional samples
Additional experiments illustrate projection, positional encoding, superresolution, sampling artifacts, and geometric properties of INR-GAN. The results connect frequency allocation and coordinate-based representations to image structure and detail.
- Keypoint prediction: Keypoints are predicted from latent codes using a linear regression procedure after synthetic images are generated and processed by a keypoint extractor.
- Positional encoding: Fourier positional encoding applies a learned matrix W to coordinates p = (x, y), then concatenates sine and cosine nonlinearities.
- Projection: Projection reconstructs low-frequency details and keypoints well, but INR-GAN reconstruction quality is lower because the vanilla architecture lacks spatial noise injection.
- Positional encoding: Each row of W represents a Fourier-transform parameter, and its norm corresponds to the frequency of the associated wave.
- Geometric prior: Coordinate inputs impose a geometric prior by ordering pixels through a Euclidean coordinate system, allowing geometric shapes to be represented compactly.
- Geometric prior: A hypernetwork can transform coordinates through modulated linear weights, enabling operations such as rotating and zooming the image.
- Sampling results: FFHQ 1024 × 1024 samples with truncation factor 0.9 achieve FID: 16.32, while untruncated samples commonly show stains and patterned textures.
E. FMM as a generalizaton of the common weight modulation schemes
Factorized Matrix Multiplication (FMM) provides a general weight-modulation framework spanning squeeze-and-excitation, AdaIN, and vanilla hypernetworks. Its rank controls the relationship between efficient modulation and expressive parameter generation.
- FMM framework: FMM represents weight modulation through factorized matrices whose effective rank determines the modulation scheme’s expressivity.
- Squeeze-and-excitation: At effective rank 1, FMM corresponds approximately to squeeze-and-excitation, which modulates neuron outputs through coefficients equivalent to scaling weight-matrix columns.
- AdaIN: AdaIN follows the same correspondence with an additional normalization layer and without sigmoid nonlinearity for the style vector.
- Comparisons: Squeeze-and-excitation is described as the least powerful but cheap and simple weight-modulation scheme, whereas vanilla hypernetworks are expensive but flexible.
- Vanilla hypernetworks: Any hypernetwork weight dynamic can be represented by FMM with sufficiently high rank by choosing the rank equal to nin and setting A to the identity.
- Design caveat: A sigmoid applied to predicted modulation values imposes positivity and range restrictions, but the experiments report no resulting harm.
- Comparison: FMM lies between common modulation extremes while retaining benefits from both approaches.
F. Performance on multi-class datasets
Experiments evaluate the proposed architectural design on two diverse multi-class image datasets. Results are reported using FID and IS at 300k iterations.
- The evaluation uses LSUN-10 256^2 and MiniImageNet-100 128^2 as two diverse multi-class datasets.LSUN-10 contains 10 LSUN scenes, while MiniImageNet-100 contains 100 ImageNet classes.
- LSUN-10 contains 1M images across 10 scenes, with 100k images sampled from each scene.
- MiniImageNet-100 contains 100k images across 100 ImageNet classes, with 1k images per class.
- Table 4 reports FID and IS for different models after 300k iterations on the multi-class datasets.