PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to See by Looking at Noise

Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, Antonio Torralba

arXiv:2106.05963v3cs.CVcs.AI

TL;DR

The paper asks whether visual representations can be learned without massive real-image datasets, which carry costs such as curation, bias, and privacy concerns. It trains a representation learner on images generated by simple noise processes and finds that naturalism and diversity matter, even though the best data need not be the most realistic.

  • Problem

    The paper asks whether vision representations can be learned without massive real-image training sets, which may be costly, biased, private, or difficult to collect.

    Source: §1 · pp. 1–2

  • Method

    The authors use statistical image models and generative models with different random initializations to produce training images for a visual representation learner.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Naturalism and diversity are key properties of useful synthetic data, and naturalistic structural properties can matter even when the data are not real.

    Source: §1 · p. 1, §6 · p. 10

  • Takeaways & Limitations

    Simple noise models can capture some structural properties of real data and successfully train visual representations.

    Source: §1 · p. 1, §6 · p. 10

    Synthetic datasets may still exhibit bias that affects subsets of test data, so bias should be characterized for each application.

    Source: §6 · p. 10

Abstract

from arXiv · show

Current vision systems are trained on huge datasets, and these datasets come with costs: curation is expensive, they inherit human biases, and there are concerns over privacy and usage rights. To counter these costs, interest has surged in learning from cheaper data sources, such as unlabeled images. In this paper we go a step further and ask if we can do away with real image datasets entirely, instead learning from noise processes. We investigate a suite of image generation models that produce images from simple random processes. These are then used as training data for a visual representation learner with a contrastive loss. We study two types of noise processes, statistical image models and deep generative models under different random initializations. Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic. We also find that diversity is a key property to learn good representations. Datasets, models, and code are available at https://mbaradad.github.io/learning_with_noise.

1 Introduction

The introduction questions whether vision systems need massive real-image training sets and explores training representations from procedural noise instead. It identifies naturalism and diversity as key properties of useful synthetic data.

1 Introduction

  • Procedural noise models generate images from simple random processes that are used to train a visual representation learner.The motivation is to question the necessity of massive training sets of real images.Source: §1 · p. 1
  • Naturalism and diversity are key properties of useful synthetic data, and the most naturalistic data is not always best because naturalism can come at diversity's expense.The authors argue that noise need not be real, but should capture structural properties of real data.Source: §1 · p. 1
  • Training without datasets could avoid costs because datasets may be expensive, biased, private, or intractable to collect.The authors do not argue for eliminating datasets entirely, since real data might still be needed for evaluation.Source: §1 · pp. 1–2

2 Related work

Related work traces image generation from models of natural-image statistics to increasingly complex generative systems, alongside growing reliance on large training sets. This paper instead studies simple noise processes as synthetic training data.

2.1 A short history of image generative models

  • Natural-image models progressed from power-law Fourier spectra and dead-leaves or fractal models to wavelet, color, and texture statistics.These lines of work sought to model statistical properties of natural images and produce more realistic-looking textures.Source: §2.1 · p. 2

2.2 Training without real data and training with synthetic data

  • Vision systems and generative models have grown more data-intensive, while prior work also found useful image priors in single-image or untrained architectures.BiT, CLIP, and SEER are reported as trained on 300 million, 400 million, and 1 billion images, respectively.Source: §2.2 · p. 2
  • For synthetic low-level vision tasks, procedural scenes can work well because diversity may matter more than realism.Prior work used rendered data for depth, stereo, and optical-flow estimation.Source: §2.2 · p. 2
  • Earlier deep generative models for synthetic training used real-image-fitted parameters; this paper tests noise processes and untrained StyleGAN priors instead.The authors broaden earlier fractal pretraining work with classic models from image coding and texture synthesis, and describe their approach as extreme domain randomization without a simulation engine.Source: §2.2 · p. 3

3 A progression of image generative models

The paper evaluates image generators as hierarchical stochastic models, sampling parameters and latent noise to produce images. Its suite spans procedural, dead-leaves, statistical, untrained GAN, and feature-visualization sources.

3 A progression of image generative models

  • The generator suite treats image creation hierarchically: sample model parameters and stochastic latent variables, then generate an image.The model parameters may be sampled from simple priors rather than fitted to real data.Source: §3 · p. 3

3.1 Procedural image models

  • Procedural image models include fractal renderings of human-designed shapes and CG environments generated with real-world properties such as illumination, 3D, and semantics.The CG examples include CLEVR, DMLab, and MineRL.Source: §3.1 · p. 3

3.2 Dead leaves image models

  • Dead-leaves images place randomly sized shapes uniformly on a canvas, with variants using squares, rotated squares, varied shapes, or textured leaves.The paper notes that this simple model produces some statistics similar to natural images, including a power-law spectrum.Source: §3.2 · pp. 3–4

3.3 Statistical image models

  • Statistical models combine natural-image-inspired spectrum, wavelet, and color distributions, but the resulting samples do not capture natural images' full richness.The spectrum model constrains Fourier magnitude; the wavelet-marginal model samples wavelet-coefficient histograms, and the properties can be combined.Source: §3.3 · pp. 4–5

3.4 Generative adversarial networks

  • The untrained StyleGANv2 variants alter initialization to produce different image characteristics, including high-frequency content, sparsity, and oriented structures.The default initialization lacks high-frequency content; wavelet filters and tied wavelets are used in other variants.Source: §3.4 · p. 5

3.5 Feature visualizations

  • Feature visualizations turn CNN unit optimization from noise-initialized inputs into a generation process, using units from a network's penultimate linear layer.The paper creates datasets from a randomly initialized ResNet50 and from a ResNet50 trained on diverse dead-leaves images with MoCo v2.Source: §3.5 · p. 5

4 Experiments

The experiments train contrastive visual representations on images generated from procedural noise and evaluate them through linear transfer. Noise-generated data performs differently across task types, while large-scale synthetic-only training reaches reported ImageNet accuracy with ResNet50.

4 Experiments

  • Evaluation uses linear transfer on ImageNet-100 and VTAB’s 19 classification tasks across Natural, Specialized, and Structured categories.The study trains an AlexNet-based encoder with an Alignment and Uniformity contrastive loss, using 105k generated samples; real-dataset and simple-feature baselines are also included.Source: §4 · p. 6

4.2 Large-scale experiments

  • 40.06% top-1 accuracy comes from mixing four synthetic datasets, versus 38.12% for the best-performing single model, without real training data.Each dataset in the four-way mixture contains 1.3M images.Source: Table 1 · p. 7, §4.2 · p. 7

4.1 Image model comparison

  • Natural data improves performance on ImageNet-100 and VTAB Natural tasks, while Structured and Specialized tasks benefit less and show similar performance across tested models.The reported task differences match the qualitative complexity and diversity of the generated samples.Source: §4.1 · p. 7

4.2 Large-scale experiments

  • 38.94% top-1 accuracy is achieved by continuously sampling StyleGAN-Oriented images, improving on training with a fixed 1.3M-image set.The authors note that the image models can generate arbitrary amounts of samples on the fly.Source: §4.2 · p. 7

5 What properties make for good generated data?

Representations learned from noise-like images can be powerful, prompting an analysis of which image-level and dataset-level properties are associated with performance. The findings point to useful structure, dataset resemblance, and diversity, with trade-offs among these properties.

5 What properties make for good generated data?

  • Feature visualizations show that noise-trained datasets can yield varied, complex activations, although Dead leaves - Squares mainly responds to axis-aligned images.The visualizations cover the third and final convolutional layers.Source: §5 · p. 7, Figure 5 · p. 8

5.1 Properties of individual images

  • r = −0.57, greater color distance from natural images is associated with lower performance.Figure 6(a) presents the color-distance comparison.Source: §5.1 · p. 8, Figure 6 · p. 8
  • Image-spectrum statistics within α ∈[0.5, 2.0] are weakly related to performance, though α ∈[1.4, 1.7] seems beneficial.Most datasets fall within the natural-image range except Stylegan-default.Source: §5.1 · p. 8, Figure 6 · p. 8
  • Perceptual variation has a sweet spot around 0.37; beyond it, accuracy decreases as views of the same image become too dissimilar.For datasets below the sweet spot, variation is strongly correlated with accuracy (r = 0.75).Source: Figure 6 · p. 8, §5.1 · p. 9

5.2 Properties of the datasets

  • r = −.85, Fréchet Inception Distance is strongly negatively correlated with accuracy, making training–test distribution resemblance the strongest reported dataset-level predictor.The analysis compares FID between training and test distributions with accuracy.Source: §5.2 · p. 9, Figure 7 · p. 9
  • r = −0.84, datasets with higher precision tend to have lower feature-space volume, and performance benefits as volume increases.The passage reports that precision decreases as volume increases.Source: §5.2 · p. 9
  • Higher recall is positively correlated with both log-volume and accuracy, while precision and recall are negatively correlated, indicating a trade-off in these datasets.When controlling for recall, precision is again positively correlated with accuracy, but more weakly (r = .22).Source: §5.2 · pp. 9–10

6 Conclusion

The paper concludes that structured-noise datasets can train visual representations and raises the possibility of pretraining without a standard real-image dataset. It cautions that such datasets may still produce application-specific biases.

6 Conclusion

  • Structured-noise datasets designed using prior research on natural-image statistics can successfully train visual representations.The authors hope this motivates new generative models of structured noise for stronger performance across visual tasks.Source: §6 · p. 10
  • The authors raise the possibility that task-specific pretraining without a large dataset may favor alternatives to standard real datasets such as ImageNet.The conclusion frames this as a possibility, not a demonstrated result.Source: §6 · p. 10
  • Synthetic datasets may still have bias affecting test-data subsets, so the authors say bias should be characterized for each application.Source: §6 · p. 10

A Experimental Details

The appendix details two contrastive-training setups and lists the linear-evaluation task groups used to report performance. The AlexNet and ResNet-50 experiments use different encoders and training procedures.

A.1 AlexNet-based encoder with Alignment and Uniformity loss

  • Section 4.1 experiments use [58]’s procedure: α=2, t=2, loss weights 1, batch size 768, SGD momentum 0.9, 200 epochs, and learning rate 0.36 decayed by 0.1 at epochs 155, 170, and 185.Detailed linear-evaluation performance is reported on ImageNet-100 and VTAB tasks using the previous setup and proposed datasets.Source: §A.1 · p. 14

A.2 MoCo v2

  • The MoCo v2 experiments use a ResNet-50 encoder, batch size 256, and the ImageNet-1k procedure and augmentations described in.Training runs for 200 epochs on 1.3M samples with a cosine learning-rate schedule.Source: §A.2 · p. 14

A.1 AlexNet-based encoder with Alignment and Uniformity loss

  • Tables 2–4 report linear evaluation after contrastive training across Natural, Specialized, and Structured tasks; bold marks the best synthetic dataset, underlined when it also outperforms real-image training.Source: Table 2 · p. 15, Table 3 · p. 16, Table 4 · p. 17

A.2 MoCo v2

  • Tables 2–4 also identify the Natural, Specialized, and Structured task groups for the MoCo v2 linear-evaluation results.Their captions mark the best synthetic dataset and underline cases that outperform real-image training.Source: Table 2 · p. 15, Table 3 · p. 16, Table 4 · p. 17

B Analysis of datasets

This appendix section supplies additional comparisons of synthetic and real image datasets on selected image statistics.

B Analysis of datasets

  • Additional data provide full distributions across datasets for color, sparsity or spectral magnitude, and self-similarity comparisons.Source: B Analysis of datasets · p. 17

B.1 Color histograms

Synthetic datasets’ color distributions remain different from ImageNet, and augmentation does not eliminate the remaining color gaps. Spectral magnitude patterns vary across datasets, without a strong relationship to accuracy.

B.1 Color histograms

  • Better-performing synthetic datasets have color distributions closer to ImageNet, though all remain fairly different from it.The datasets lack very dark and very bright areas and are oversaturated; augmentation spreads colors and increases saturation but does not remove those gaps.Source: §B.1 · pp. 17–18, Figure 8 · p. 18, Figure 9 · p. 19
  • After MoCo-v2 augmentation, colors spread out and saturation increases, yet synthetic datasets still lack dark colors and remain oversaturated.The augmented distributions also contain many dark but completely desaturated pixels.Source: §B.1 · p. 18, Figure 9 · p. 19

B.2 Sparsity

  • The spectral exponent α varies across datasets, but its distribution does not seem to have a large impact on performance.Although α ≈ 1.35 appears to be a sweet spot, the study found no particularly strong correlation between α and accuracy.Source: §B.2 · p. 19

B.3 Self-similarity

Accuracy is strongly correlated with crop self-similarity up to an optimum, after which the correlation becomes negative. The best-performing datasets also have self-similarity distributions that appear most like ImageNet’s.

B.3 Self-similarity

  • Self-similarity and accuracy are strongly correlated up to an optimal value, after which the correlation becomes negative.Self-similarity is measured as the average perceptual distance between two crops of the same image.Source: §B.3 · p. 19
  • The study treats within-image self-similarity as a core requirement for contrastive learning and measures it across pairs of crops.The figure presents the full distributions of self-similarity scores for the datasets.Source: §B.3 · p. 19, Figure 10 · p. 20, Figure 11 · p. 21
  • StyleGAN-Oriented, StyleGAN-High-freq, StyleGAN-Sparse, and Minecraft are both among the best-performing datasets and appear most similar to ImageNet in their self-similarity distributions.The distributions are shown for different datasets, with color indicating dataset class.Source: §B.3 · p. 19, Figure 11 · p. 21

C Dataset samples

The appendix provides 96 random samples for each of 18 datasets, spanning graphics-based, statistical, StyleGAN-initialization, and feature-visualization generators.

C.1 FractalDB; C.2 CLeVR; C.3 DMLab

C.4 Minecraft; C.5 Dead leaves - Squares; C.6 Dead leaves - Oriented

C.7 Dead leaves - Shapes; C.8 Dead leaves - Textures; C.9 Spectrum

C.10 WMM; C.11 Spectrum + Color; C.12 Spectrum + Color + WMM

C.13 StyleGAN - Random; C.14 StyleGAN - High freq.; C.15 StyleGAN - Sparse

C.16 StyleGAN - Oriented; C.17 Feature vis. - Random; C.18 Feature vis. - Dead leaves

Loading 2106.05963v3…