PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-training without Natural Images

Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, Yutaka Satoh

arXiv:2101.08515v1cs.CVcs.LG

TL;DR

The paper asks whether CNNs can assist natural-image understanding when pre-trained without natural images. It proposes Formula-driven Supervised Learning, which generates fractal images and labels for FractalDB pre-training. FractalDB sometimes surpasses ImageNet- or Places-pre-trained models, although not across all settings, and produces a distinct image representation.

  • Problem

    The paper investigates whether CNNs can be pre-trained without natural images while addressing the privacy, ethics, copyright, collection, and labeling burdens of natural-image datasets.

    Source: §1 · pp. 1–2

  • Method

    The paper generates labeled fractal images from iterated function systems and uses FractalDB to pre-train CNNs before transfer learning.

    Source: §3.1 · p. 3, §3.2 · p. 4, §5 · p. 10

  • Results

    FractalDB models outperform some ImageNet- and Places-pre-trained models on selected transfer tasks, while showing a distinct image representation and faster convergence than training from scratch.

    Source: §1 · p. 2, §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    Formula-driven image generation can provide a natural-image-free source of labeled pre-training data and a different representation for image recognition.

    Source: §1 · p. 2, §5 · p. 10

    FractalDB does not outperform human-annotated pre-training models at all settings, and its configuration may need adjustment for broader tasks such as object detection and semantic segmentation.

    Source: §4.2 · p. 7, §5 · p. 10

Abstract

from arXiv · show

Is it possible to use convolutional neural networks pre-trained without any natural images to assist natural image understanding? The paper proposes a novel concept, Formula-driven Supervised Learning. We automatically generate image patterns and their category labels by assigning fractals, which are based on a natural law existing in the background knowledge of the real world. Theoretically, the use of automatically generated images instead of natural images in the pre-training phase allows us to generate an infinite scale dataset of labeled images. Although the models pre-trained with the proposed Fractal DataBase (FractalDB), a database without natural images, does not necessarily outperform models pre-trained with human annotated datasets at all settings, we are able to partially surpass the accuracy of ImageNet/Places pre-trained models. The image representation with the proposed FractalDB captures a unique feature in the visualization of convolutional layers and attentions.

1 Introduction

The paper asks whether CNNs can be pre-trained without natural images and proposes Formula-driven Supervised Learning using automatically generated fractal images and labels. FractalDB enables large-scale dataset construction without human annotation or image downloading, and sometimes matches or surpasses natural-image pre-training.

  • Pre-training without natural images could address privacy, copyright, ethics, and the cost of collecting and labeling natural-image datasets.Source: §1 · p. 2
  • Formula-driven Supervised Learning generates image patterns and labels from mathematical formulas rather than natural images or closely resembling synthetic images.Source: §1 · p. 2
  • FractalDB-1k and FractalDB-10k contain 1,000 and 10,000 automatically generated fractal categories for pre-training.Source: §1 · p. 2
  • FractalDB pre-training outperformed some human-annotated pre-training models and accelerated convergence similarly to ImageNet pre-training.Source: §1 · p. 2

2 Related work

Prior work established large-scale natural-image pre-training and alternative learning frameworks, but the paper targets dataset generation without natural images. Its framework complements these approaches by generating both new image patterns and labels from mathematical formulas.

Pre-training on Large-scale Datasets

  • ImageNet and Places established natural-image pre-training as a strong approach for learning transferable image representations.Source: §2 · pp. 2–3
  • Large-scale natural-image datasets support image classification, object detection, and video classification, while newer datasets seek further pre-training gains.Source: §2 · pp. 2–3

Learning Frameworks

  • Self-supervised and related frameworks reduce reliance on human labels but still operate on existing image datasets.Source: §2 · p. 3
  • The proposed framework complements these methods by generating new image patterns and training labels from mathematical formulas.Source: §2 · p. 3

Mathematical formula for image projection

  • Fractals, Bezier curves, and Perlin noise are mathematical rendering approaches considered for producing image patterns, with fractals motivated by naturally occurring phenomena.Source: §2 · p. 3

3 Automatically generated large-scale dataset

FractalDB uses iterated function systems to generate fractal images, categories, and intra-category variants without downloading or manually labeling images. The resulting database supports CNN pre-training through standard optimization and transfer learning.

Fractal image generation

  • FractalDB consists of image-category pairs generated with iterated function systems, without human labeling or image downloading.Source: §3 · p. 3, Figure 2 · p. 4
  • A FractalDB pre-trained CNN is trained with labeled fractal mini-batches, then fine-tuned or transferred to other datasets.Source: §3 · p. 3, Figure 2 · p. 4
  • An iterated function system contains transformation functions and probabilities, and repeatedly applies sampled transformations to generate fractal points.Source: §3.1 · p. 3
  • The generated fractal points are rendered as images by drawing dots on a black background.Source: §3.1 · p. 4

Fractal categories

  • Each fractal category is defined by a fixed set of transformation parameters and probabilities, producing FractalDB-1k and FractalDB-10k with 1,000 or 10,000 categories.Source: §3.2 · p. 4

Adaptable parameters for FractalDB

  • Intra-category variation is generated by changing affine-transformation parameters, while rendering experiments vary category count, instances, patches, filling rate, dots, and image size.Source: §3.3 · pp. 4–5

4 Experiments

Experiments show that FractalDB benefits from larger category and instance counts, patch rendering, and selected rendering parameters, while enabling transfer from CNNs pretrained without natural images. FractalDB often beats scratch, self-supervised, and alternative formula-driven pretraining, and partially surpasses ImageNet/Places baselines while producing distinct representations.

Exploration study

  • Larger #category and #instance values generally improve fine-tuning accuracy, with #category producing larger gains than #instance on C10/100.Increasing #category from 16 to 1,000 improves C10/C100 by +7.9/+16.0, while #instance improves them by +5.2/+8.9.Source: §4.1 · p. 5
  • 3 × 3 patch rendering outperforms 1 × 1 point rendering by +4.7 on C10 and +5.9 on C100.Patch rendering scores 92.1 versus 87.4 on C10 and 72.0 versus 66.1 on C100.Source: §4.1 · p. 5

Comparison to other pre-trained datasets

  • FractalDB-1k/10k outperforms scratch and DeepCluster-10k across reported comparisons, while partially surpassing ImageNet/Places pretraining rather than winning at all settings.Examples include 94.1 versus 89.9 on C10 against DeepCluster, and 50.8 versus 50.3 on Places365 against ImageNet-1k.Source: §4.2 · p. 7

Comparison and transfer

  • FractalDB-1k/10k pretraining accelerates convergence similarly to ImageNet pretraining and supports accurate transfer when lower convolutional layers are frozen.Freezing conv1 decreases C10/C100 performance by only -1.4/-2.8 relative to full fine-tuning.Source: §4.3 · p. 8

Additional experiments

  • FractalDB outperforms Perlin-noise and Bezier-curve datasets, better recognizes some recursive-pattern categories, and learns representations distinct from natural-image pretraining.Reported advantages include +3.0 on C10, +4.6 on C100, +3.0 on IN100, and +1.7 on P30 over Perlin-1296.Source: §4.3 · pp. 8–9

5 Discussion and Conclusion

FractalDB enables pre-training without natural images and can partially outperform ImageNet/Places pre-training, while producing distinct visual representations. Its effectiveness depends on the fine-tuning task and remains limited across some settings.

Discussion and Conclusion

  • FractalDB pre-training achieved higher performance than scratch training and some supervised and self-supervised methods across several representative datasets.The evaluations included CIFAR-10/100, ImageNet, Places, and Pascal VOC.Source: §5 · p. 10
  • FractalDB-10k partially outperformed ImageNet-1k and Places-365 pre-trained models on selected transfer tasks.Examples include 77.3 vs. 76.9 on CIFAR-100 against Places-365 and 50.8 vs. 50.3 on Places365 against ImageNet-1k.Source: §5 · p. 10
  • FractalDB outperformed formula-driven datasets based on Bezier curves and Perlin noise, and its category assignment was better than DeepCluster-10k in most cases.The paper also reports that training accuracy decreased with label noise.Source: §5 · p. 10
  • FractalDB pre-training acquired visual patterns distinct from those learned by ImageNet-1k.The difference appeared in visualizations of convolutional layers and attentions.Source: §5 · p. 10
  • The pre-training dataset configuration should be adjusted for the fine-tuning task, and broader applicability to object detection and semantic segmentation remains a future goal.The authors hope to develop a flexibly generated pre-training dataset for a wider range of tasks.Source: §5 · p. 10

Related papers

Loading 2101.08515v1…