PaperlayerView exact revision on arXiv

Source-linked AI summary

A Close Look at Deep Learning with Small Data

L. Brigato, L. Iocchi

arXiv:2003.12843v3cs.LGstat.ML

TL;DR

Training deep models with few samples per class is difficult, so the paper empirically studies architectures and regularization in the small-data regime across three image benchmarks. It finds that low-complexity CNNs can rival or exceed higher-complexity models, while augmentation and dropout often improve performance.

  • Problem

    The paper addresses image classification when only a few labeled samples per class are available, a setting in which deep models can overfit and generalization is difficult.

  • Method

    The authors benchmark CNNs of different complexities and ResNet-20 while varying training-set size and evaluating augmentation and dropout on subsampled CIFAR-10, Fashion-MNIST, and SVHN.

  • Results

    Low-complexity CNNs are comparable to or better than high-complexity models in some small-data settings, while data augmentation and dropout generally improve or preserve recognition performance.

  • Takeaways & Limitations

    Model complexity, data augmentation, and dropout should all be considered when developing deep-learning systems for limited-data image classification.

Abstract

from arXiv · show

In this work, we perform a wide variety of experiments with different deep learning architectures on datasets of limited size. According to our study, we show that model complexity is a critical factor when only a few samples per class are available. Differently from the literature, we show that in some configurations, the state of the art can be improved using low complexity models. For instance, in problems with scarce training samples and without data augmentation, low-complexity convolutional neural networks perform comparably well or better than state-of-the-art architectures. Moreover, we show that even standard data augmentation can boost recognition performance by large margins. This result suggests the development of more complex data generation/augmentation pipelines for cases when data is limited. Finally, we show that dropout, a widely used regularization technique, maintains its role as a good regularizer even when data is scarce. Our findings are empirically validated on the sub-sampled versions of popular CIFAR-10, Fashion-MNIST and, SVHN benchmarks.

I. INTRODUCTION

Deep learning succeeds in vision but typically depends on deep architectures and large annotated datasets, which are often costly or unavailable. This work studies deep models in the small-data regime and reports that low-complexity CNNs, augmentation, and dropout can remain effective.

  • Large annotated datasets support strong deep-learning results but can be expensive or impossible to obtain, especially in medicine.Radiology annotations are cited as a costly example.
  • Systems that recognize objects from only a few exemplars remain desirable because collecting many images can still be difficult without human annotations.
  • The study benchmarks deep models while varying training-set size across three popular computer-vision benchmarks in the small-data regime.
  • Low-complexity CNNs can match or outperform high-complexity models depending on training-set size and whether data augmentation is used.
  • Standard data augmentation consistently improves testing accuracy, while dropout remains a useful regularizer when few samples are available.

II. RELATED WORK

Limited-size datasets make learning and generalization difficult, motivating methods that reduce overfitting or improve performance with scarce samples. Prior work includes ensembles, NTK-based models, cosine loss, CNTK networks, and analyses of network size.

  • Learning from limited-size datasets is challenging and remains largely unsolved because generalizing to novel instances is difficult.
  • Random forests and decomposed low-bias neural sub-networks achieved competitive results on vector-data benchmarks through ensemble-like regularization.
  • Neural tangent kernel architectures produced significant improvements over earlier classifiers in low-data tasks, while CNTK networks outperformed ResNet-34 on small CIFAR-10.
  • Cosine loss was proposed as a hyper-parameter-free regularizer for scarce-sample settings and performed best on fine-grained datasets with 20–80 samples per class.
  • Prior analysis found that larger networks can match or exceed smaller networks when properly optimized and calibrated.

III. SMALL-DATA CLASSIFICATION PROBLEM

The paper formulates supervised image classification with K classes and N training images per class, deliberately keeping N low. It evaluates models across progressively larger subsampled training sets, reflecting data collection over time.

  • The task learns a classifier fθ mapping images x to labels y across K classes, using N training images per class.
  • Experiments keep N low relative to standard computer-vision datasets and vary it using two subsampling protocols.
  • One protocol doubles N from Nmin = 1 through Nmax = 128 to compare against prior networks.
  • A second protocol uses N in {10, 20, 40, 80, 160, 320, 640, 1280} to study configurations with more data.
  • The protocol models applications that collect data over time and need model-selection guidance before large quantities are available.

B. Models

The study compares three standard CNNs of increasing complexity with ResNet-20, using convolutional feature extractors, residual connections, and complexity measured by parameters and FLOPs.

  • Models: Three CNNs—CNN-lc, CNN-mc, and CNN-hc—are defined with increasing complexity and compared with the more complex ResNet-20.
  • Models: Model complexity is reported using trainable parameters and floating-point operations, alongside convolutional filter widths.
  • Models: Standard CNNs use convolutional and max-pooling layers as feature extractors and optimize the standard classification loss.
  • Models: ResNet-20 uses stacked residual blocks with shortcut connections that help gradients flow backward and ease overall training.
  • Models: The CNNs process images through four convolutional layers with 3 × 3 kernels, stride 1, and max-pooling layers with pool-size and stride 2.

C. Datasets

The study evaluates sub-sampled CIFAR-10, Fashion-MNIST, and SVHN datasets with ten classes and fixed original testing sets. Each dataset is down-sampled by samples per class, and every configuration is evaluated over 10 runs.

  • Experiments use CIFAR-10, Fashion-MNIST, and SVHN, each containing ten classes and relatively small images.
  • The authors create sub-sampled datasets by limiting the number of training samples per class while retaining the original testing-set sizes.A dataset name such as sCIFAR-10-20 denotes CIFAR-10 with 20 samples per class.
  • Each sub-sampled dataset configuration is evaluated through 10 runs to improve result consistency.

D. Regularization techniques

The study examines dropout and data augmentation as regularization techniques for recognition with few training samples. Dropout rates are varied across absent, medium, and high settings, while augmentation artificially increases image data.

  • Dropout is applied to the last layer of CNN-lc, CNN-mc, and CNN-hc with rates of 0.0, 0.4, and 0.7.These correspond to absent, medium, and high dropout.
  • Data augmentation artificially increases the images available in a dataset by applying augmentation techniques.

A. Influence of models complexity on performance

The paper studies how model complexity affects classification performance when training data is limited. It compares three CNNs of increasing complexity with the higher-complexity ResNet-20 without data augmentation.

  • Without data augmentation, standard CNNs consistently outperform ResNet-20 on sCIFAR-10 when N is below 320 samples per class.
  • ResNet-20 achieves the best accuracy among the four models on sCIFAR-10 only at N = 1280.
  • The gap between ResNet-20 and CNN-hc is roughly 10% on sCIFAR-10 with 10 through 320 samples per class.
  • CNN-lc maintains a performance gap through N = 320 despite using roughly two orders of magnitude fewer FLOPs than ResNet-20.
  • Similar behavior appears on sFMNIST and sSVHN, although the most complex model improves more rapidly.

B. Influence of regularization techniques on performance

The experiments assess dropout and data augmentation across sub-sampled benchmarks using accuracy averaged over repeated runs. Both techniques can improve recognition with limited data, but their effects depend on dataset, architecture, and sample count.

  • Dropout: High dropout improves generalization on sCIFAR-10, with CNN-lc gaining around 5% over no dropout.CNN-hc also outperforms its absent- and medium-dropout variants.
  • Dropout: CNN-mc with high dropout outperforms its no-dropout version by ∼10% on sSVHN-40.Improvements decrease as the training set grows but remain noticeable.
  • Dropout: Dropout has little effect on final testing accuracy for sFMNIST, except that CNN-hc benefits more with 10 or 20 samples per class.
  • Data augmentation: On sCIFAR-10, augmentation improves CNN-hc by ∼9% at N = 640 and ResNet-20 by ∼20% at N = 320.ResNet-20 matches CNN-hc accuracy at N = 160 with augmentation, compared with at least 640 samples without it.
  • Data augmentation: On sSVHN, augmentation produces maximum gains of ∼14% for CNN-hc at N = 20 and ∼20% for ResNet-20 at N = 40.The largest improvements occur with smaller training sets.
  • Data augmentation: On sFMNIST, augmentation gives CNN-hc gains of around 1.5% and boosts ResNet-20 by ∼8% with 10 samples per class.ResNet-20 matches CNN-hc performance at N = 160 with augmentation.

C. Comparison with the state of the art

Under an extremely limited-sample CIFAR-10 protocol without data augmentation, cosine loss offers no clear advantage, while the computationally simple CNN-hc outperforms CNTK across all subsampled versions.

  • Evaluation setup: The comparison evaluates CNN-hc and ResNet-20 against state-of-the-art techniques using accuracy averaged over 10 runs without data augmentation.CIFAR-10 follows the subsampling protocol with Nmin = 1 and Nmax = 128.
  • State-of-the-art comparison: Cosine loss does not provide a clear advantage over cross-entropy for CNN-hc and ResNet-20 with extremely limited samples.Cross-entropy is more performant in many cases, and cosine loss does not outperform it in the reported setup.
  • Regularization analysis: Dropout experiments compare testing accuracy across model complexities and absent, medium, and high dropout levels without data augmentation.Results are average accuracy over 10 runs.
  • State-of-the-art comparison: CNN-hc outperforms CNTK in all sub-sampled versions of CIFAR-10 under the hard training protocol.The protocol uses extremely limited samples and no data augmentation.

V. CONCLUSIONS

The conclusions emphasize that relatively simple networks can generalize better on small datasets, whereas data augmentation allows deeper networks to gain higher performance. Dropout generally improves results in small-data settings.

  • Model complexity: Relatively simple networks can be less prone to overfitting and generalize better when datasets are small.The authors recommend comparing future architectures with simpler networks, not only state-of-the-art models.
  • Data augmentation: When data augmentation is used, deeper networks rapidly gain higher performance even with basic augmentation policies.The authors attribute this to the artificial addition of training images.
  • Data augmentation: Data augmentation can induce large recognition improvements depending on dataset type and size.The paper proposes studying more sophisticated augmentation techniques in future work.
  • Regularization: Dropout generally improves results and should be used in small-data settings.The paper analyzes dropout alongside data augmentation as a regularization technique.
Loading 2003.12843v3…