Source-linked AI summary
A Data-Centric Approach for Training Deep Neural Networks with Less Data
Mohammad Motamedi, Nikolay Sakharnykh, Tim Kaldewey
TL;DR
Small datasets can contain a higher proportion of invalid samples, making data quality important when training deep neural networks with limited data. The paper combines a systematic data-optimization pipeline with GAN-based sample synthesis for Roman-numeral classification. The resulting dataset improves accuracy while being smaller than the baseline, and GAN-generated samples further improve accuracy.
Problem
The paper asks how to alter training and validation data under a size constraint so a fixed network outperforms a baseline on an undisclosed test set.
Source: §2 · p. 2Method
Results
5% accuracy improvement was achieved by the optimized Roman-MNIST dataset while it was 1.54x smaller than the baseline; GAN-generated samples further improved accuracy by one percent.
Source: §6 · p. 4Takeaways & Limitations
On Roman-MNIST, dataset optimization improved accuracy with a smaller dataset, and adding GAN-generated samples further improved accuracy by one percent.
Source: §6 · p. 4The stated setting constrains the combined size of the training and validation sets to less than Nmax.
Source: §2 · p. 2
Abstract
from arXiv · showhide
While the availability of large datasets is perceived to be a key requirement for training deep neural networks, it is possible to train such models with relatively little data. However, compensating for the absence of large datasets demands a series of actions to enhance the quality of the existing samples and to generate new ones. This paper summarizes our winning submission to the "Data-Centric AI" competition. We discuss some of the challenges that arise while training with a small dataset, offer a principled approach for systematic data quality enhancement, and propose a GAN-based solution for synthesizing new data points. Our evaluations indicate that the dataset generated by the proposed pipeline offers 5% accuracy improvement while being significantly smaller than the baseline.
1 Introduction
Data-centric AI seeks to help deep neural networks train effectively on smaller datasets, where invalid samples can have a greater impact. The paper proposes systematic dataset-quality enhancement and GAN-based sample generation for handwritten Roman numeral classification.
1 Introduction
- Data-centric AI aims to develop dataset-optimization techniques that support effective deep-network training with smaller datasets.The paper notes potential relevance to applications with constrained data-collection budgets and domains where training samples are inherently scarce, such as manufacturing flaw detection.Source: §1 · p. 1
- Invalid samples have less impact when they are a small share of a dataset, but their increasing proportion in small datasets makes quality improvements more consequential for accuracy.The paper attributes the limited impact in larger datasets to mini-batch gradient averaging and the learning rate.Source: §1 · p. 1
- The paper combines systematic identification of invalid data with GAN-based sample synthesis to optimize existing data and increase training-sample diversity.Evaluation uses handwritten Roman numeral classification with Roman-MNIST as the baseline.Source: §1 · p. 1
2 Problem statement
The problem is to alter the training and validation sets under a size constraint so a fixed network outperforms a baseline on an undisclosed test set. The paper frames this question around a Roman-numeral classification competition.
2 Problem statement
- The training and validation sets together must remain below a specified maximum size, Nmax.The paper uses scarce-sample flaw detection in steel industries as an example of this constraint.Source: §2 · p. 2
- The central question is how to alter the training set R and validation set V so a fixed architecture ψ performs better than the baseline on undisclosed test set T.Source: §2 · p. 2
- For the competition, the task is handwritten Roman-numeral classification using a cutoff ResNet50, with Nmax = 10,000, |T| = 2420, |R| = 2067, |V| = 813, and accuracy as the metric.The test set was undisclosed during the competition.Source: §2 · p. 2
3 Dataset optimization pipeline
The pipeline iteratively checks dataset quality, addresses class imbalance, and prepares cross-validation splits. Human review supports sample certification and correction throughout the investigation process.
3 Dataset optimization pipeline
- The pipeline uses multi-stage hashing to find identical samples, eliminating redundancy and preventing overlap between training and validation sets.Source: §3 · p. 2
- A human-certified sample subset trains an auxiliary classifier, with heavy augmentation used to support training on limited data.For Roman numerals, 180 samples train the classifier and 20 validate it; these auxiliary models are no longer needed after dataset investigation.Source: §3 · p. 2
- The auxiliary model ranks remaining samples by loss; humans confirm low-loss samples and review high-loss or potentially mislabeled cases, with the process iterating until all competition samples are certified.Figure 1 includes examples of augmented images, false positives, and high-loss samples flagged for review.Source: §3 · p. 2, Figure 1 · p. 3
- Additional examples from each class are moved to a surplus dataset to equalize class counts, then prioritized for return to training as data are augmented.Source: §3 · p. 2
- After random test-set selection, the remaining data are divided into N folds; Roman-numeral experiments use N = 8.Source: §3 · pp. 2–3
4 GAN-based sample synthesis
The paper proposes a conditional GAN to generate additional class-conditioned samples for handwritten Roman numerals. Its generator is trained to produce samples judged real by a discriminator but misclassified by a classifier.
4 GAN-based sample synthesis
- The conditional GAN combines a generator, discriminator, and truncated pre-trained ResNet50 classifier to synthesize additional samples.The generator has eight convolutional layers; the discriminator has five, and the classifier is truncated at “conv2_block3_out.”Source: §4 · p. 3
- The generator uses discriminator and classifier losses so generated samples appear real to the discriminator while being misclassified by the classifier.The paper describes these outputs as expected edge cases that the classifier may benefit from seeing more often.Source: §4 · p. 3
- The generator takes a 64-element standard-normal noise vector and class labels, and produces grayscale samples.The loss weights δ and γ govern the relative impact of the two loss components.Source: §4 · p. 3
5 Experimental results
The experiments use the competition’s truncated ResNet50 setup to assess auxiliary models and dataset variants. The optimized dataset improved accuracy while using fewer samples.
5 Experimental results
- 5% accuracy improvement was reported for the optimized dataset, which was 1.54x smaller than the competition’s initial dataset.Table 2 also includes variants with additional handwritten or machine-font samples and a final row using 1,226 GAN-generated samples.Source: Table 2 · p. 4, §5 · p. 4
- The final classifier used the competition’s truncated ResNet50, trained at 32x32 with batch size 8 for 100 epochs and early stopping.Reported times were measured on an NVIDIA TITAN V, while NVIDIA A100 GPUs accelerated GAN training.Source: §5 · pp. 3–4
- 92% of first-round validity predictions were confirmed by a human supervisor after an auxiliary model was trained on 180 samples.Auxiliary models used the competition’s reference architecture, trained for 200 epochs with augmentation, and later rounds reused selected samples.Source: §5 · p. 4, Table 1 · p. 4
6 Conclusion
The paper presents a data-quality enhancement approach and a GAN that synthesizes hard-to-classify samples. On Roman-MNIST, the optimized dataset improved accuracy while being smaller than the baseline.
6 Conclusion
- The paper proposes a principled approach for enhancing dataset quality.Source: §6 · p. 4
- 5% accuracy improvement was reported for the pipeline-generated dataset, which was 1.54x smaller than the Roman-MNIST baseline.Source: §6 · p. 4
- The GAN uses classifier and discriminator feedback to synthesize hard-to-classify samples, and those samples further improved accuracy by one percent.Source: §6 · p. 4