Source-linked AI summary
Image Generators with Conditionally-Independent Pixel Synthesis
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, Denis Korzhenkov
TL;DR
Existing image generators depend heavily on spatial convolutions, motivating a convolution-free alternative for unconditional image synthesis. CIPS computes pixels independently from coordinates and latent information, and achieves quality on par with StyleGANv2 while supporting flexible synthesis settings. The paper also identifies recurring artifacts that limit the architecture.
Problem
Prior image generators rely heavily on spatial convolutions or attention, while convolution-free perceptron methods had not established comparable unconditional image-generation quality.
Method
CIPS uses coordinate encodings and latent-vector conditioning in a fully connected generator that computes each pixel independently, without spatial convolutions or attention.
Results
CIPS achieves image-generation quality on par with StyleGANv2 and supports cylindrical panoramas, foveated rendering, and generative super-resolution.
Takeaways & Limitations
Independent pixel synthesis provides flexibility for nontrivial coordinate topologies, sequential inference, and foveated image synthesis.
Takeaways & Limitations
CIPS can produce wavy textures and locally inconsistent image regions, which the authors associate with periodic activations and the absence of pixel coordination.
Abstract
from arXiv · showhide
Existing image generator networks rely heavily on spatial convolutions and, optionally, self-attention blocks in order to gradually synthesize images in a coarse-to-fine manner. Here, we present a new architecture for image generators, where the color value at each pixel is computed independently given the value of a random latent vector and the coordinate of that pixel. No spatial convolutions or similar operations that propagate information across pixels are involved during the synthesis. We analyze the modeling capabilities of such generators when trained in an adversarial fashion, and observe the new generators to achieve similar generation quality to state-of-the-art convolutional generators. We also investigate several interesting properties unique to the new architecture.
1. Introduction
CIPS investigates whether unconditional image generators can achieve state-of-the-art quality without spatial convolutions or attention. It uses coordinate-based, conditionally independent pixel synthesis and reports comparable quality at 256 × 256.
- Existing state-of-the-art image generators rely mainly on spatial convolutions and sometimes spatial self-attention.
- Prior convolution-free deep perceptron architectures reproduced images well but were restricted to individual scenes.
- CIPS computes each pixel’s color from its coordinate and latent vector without spatial convolutions or spatial attention.
- At 256 × 256 resolution, CIPS achieves generation quality similar to StyleGANv2 and surpasses it on some datasets.
- Independent pixel processing supports cylindrical panoramas, sequential synthesis, improved photo quality, and foveated synthesis.
2. Related Work
Related work established coordinate-based perceptrons for individual-scene image representation, but unconditional synthesis of diverse, complex images remained largely convolutional or unscaled.
- CoordConv and COCO-GAN used pixel coordinates, but retained standard convolutions as their main synthesis operation.
- Earlier coordinate-regression approaches created compositional patterns but did not produce photorealistic high-resolution outputs.
- Perceptron GAN experiments succeeded on MNIST but were not scaled to more complex image data.
- SIREN- and Fourier-feature-based perceptrons encoded individual high-resolution scenes, without addressing diverse unconditional image synthesis.
- GRAF embedded NeRF generators for 3D-aware synthesis, whereas this work studies high-quality 2D synthesis with perceptron-based architectures.
3. Method
CIPS evaluates a shared latent vector and pixel coordinates with a modulated fully connected network, using positional encodings to predict each RGB value independently. Fourier features are combined with learned coordinate embeddings because Fourier encoding alone produced wave-like artifacts.
- Generator formulation: Each pixel is generated by evaluating G(x, y; z) at its coordinate while keeping the shared random vector z fixed.
- Style modulation: A mapping network converts z into style vector w, which supplies the generator’s stochasticity through weight modulation.
- Style modulation: A ModFC layer applies a style-modulated weight matrix and bias to an input feature vector, followed by LeakyReLU.
- Generator formulation: Skip connections sum RGB contributions from intermediate layers for the same pixel without introducing cross-pixel interactions.
- Generator formulation: Independent pixel generation enables parallel inference and permits variants using different noise vectors per pixel, provided z varies gradually for consistency.
- Positional encoding: The architecture uses coordinate encodings and modulated fully connected layers instead of spatial operations to predict pixel colors.
- Positional encoding: Fourier positional encoding alone produced wave-like artifacts, so CIPS concatenates Fourier features with learned coordinate embeddings.
4. Experiments
Experiments show that CIPS achieves image-generation quality comparable to StyleGANv2 while enabling independently sampled pixels, flexible coordinate grids, and sequential or dense synthesis. Ablations and spectral analyses identify coordinate embeddings and architectural connections as important quality factors, while recurring artifacts remain a limitation.
- Evaluation: CIPS-skips achieves image quality comparable to StyleGANv2 across multiple 256 × 256 datasets and performs better on Churches.The comparison uses FID, with the FFHQ CIPS value differing between tables because of longer training and a larger batch size.
- Evaluation: Precision is better but recall is worse on FFHQ, indicating more plausible images alongside more dropped modes.The reported Precision and Recall measurements are evaluated at 256 × 256 resolution.
- Ablations: Coordinate embeddings, residual blocks, and cumulative RGB projections significantly improve CIPS quality, with removing coordinate embeddings causing the largest FID degradation.The ablation compares Fourier features, coordinate embeddings, activation functions, residual connections, and output projections.
- Influence of positional encodings: Coordinate embeddings provide higher-frequency information and encode facial-hair and forelock details that Fourier features do not provide alone.Spectrum and PCA analyses attribute this difference to independently trained per-pixel embeddings versus coordinate-dependent Fourier features.
- Spectral analysis: CIPS spectra avoid some high-frequency artifacts seen in StyleGANv2, but coordinate embeddings and skip connections can reduce similarity to natural-image spectra despite improving FID.CIPS-NE has azimuthal-integration statistics close to real images, whereas adding coordinate embeddings improves FID but worsens spectral realism.
- Applications and artifacts: Independent pixel synthesis supports foveated rendering, denser-than-training-resolution synthesis, and cylindrical panoramas, but recurring wavy textures and spatial inconsistencies remain.Foveated synthesis samples an irregular coordinate grid, while denser grids produce more plausible details than Lanczos upsampling; artifacts are attributed to periodic activations and independent pixel processing.
5. Conclusion
CIPS computes pixel colors from random noise and coordinates without spatial convolutions, attention, or upsampling, yet reaches state-of-the-art-comparable quality and supports non-Cartesian image structures.
- CIPS computes each pixel’s color using only random noise and coordinate position.
- The architecture competes with convolutional generators on FID and precision & recall without spatial convolutions, attention, or upsampling.
- CIPS outputs have spectra that are harder to distinguish from real images, while the CIPS-NE variant has a more realistic spectrum.
- Replacing the coordinate system enables synthesis on complex structures such as cylindrical panoramas.
- CIPS matches StyleGANv2 quality and has been applied to generative foveated rendering and super-resolution.
A. Architecture details
The default CIPS generator uses a fully connected backbone whose repeated modulated layers transform coordinate encodings and Fourier features into RGB outputs.
- Table 4 compares parameter counts across CIPS variants with the StyleGANv2 generator.
- The default CIPS generator is illustrated as a 15-layer fully connected backbone.
- Its first layer projects concatenated coordinate embeddings and Fourier features into a 512-dimensional joint space.
- Seven repetitions of two modulated fully connected layers are followed by a projection to RGB color space.
B. Coordinate embeddings
Coordinate embeddings encode substantial spatial structure in CIPS, with dataset alignment affecting the structure learned in those embeddings.
- PCA visualizations show less recognizable structure in coordinate embeddings for Landscapes and LSUN-Churches than for the more aligned face dataset.
- The LSUN-Churches embeddings contain bottom horizontal structure likely associated with a frequent dataset watermark.
- Patch-based CIPS results are evaluated with FID across 64 × 64 and 128 × 128 patches at fixed 256 × 256 resolution.
C. Patch-based generation
Conditional pixel independence permits patch-based memory-constrained training, but reducing the discriminator’s observed patch size degrades sample quality.
- Patch-based training generates only a K × K patch for discriminator input because pixels are conditionally independent.
- 64 × 64 and 128 × 128 patch-trained models produce meaningful samples, but their quality and diversity are worse than standard training.
- Patch sampling can be applied to real images before discrimination as a differentiable augmentation.
- Smaller patches degrade the overall quality of resulting samples compared with full-image training.
D. Additional samples
The paper provides additional CIPS samples across datasets and shows that the model supports layer-wise latent-variable combination. These examples suggest that different network depths control different image aspects, despite no mixing regularization during training.
- Additional CIPS samples are shown for models trained on different datasets, including further cylindrical panoramas.
- Despite lacking mixing regularization during training, CIPS supports layer-wise combination of latent variables at different depths.The examples suggest that, similarly to StyleGAN, different layers control different image aspects.
E. Nearest neighbors
The paper examines CIPS samples across datasets, cylindrical panorama synthesis, nearest-neighbor relationships for generated faces, and layer-wise style mixing. These visualizations address sample plausibility, generalization beyond memorization, and depth-specific control of image attributes.
- CIPS samples are presented across datasets, with real samples in the top row and generated samples in the remaining rows.The samples are described as plausible and diverse.
- CIPS generates cylindrical panoramas from standard landscape photographs without panoramas in the training data.
- Nearest-neighbor comparisons use FaceNet descriptors to assess whether generated faces extend beyond memorization of the training dataset.Each generated face is compared with the five closest training faces.
- Layer-wise style mixing suggests that early layers control pose and head shape, middle layers finer facial geometry, and final layers skin color and texture.