PaperlayerView exact revision on arXiv

Source-linked AI summary

TAPAS: Train-less Accuracy Predictor for Architecture Search

R. Istrate, F. Scheidegger, G. Mariani, D. Nikolopoulos, C. Bekas, A. C. I. Malossi

arXiv:1806.00250v1cs.LGstat.ML

TL;DR

Architecture-search methods commonly require costly training, and existing predictors do not reliably handle unseen datasets without new experiments. TAPAS combines dataset-difficulty characterization with accumulated cross-dataset experiments to predict peak accuracy without training candidate networks. It enables large-scale search within minutes on a single GPU and identifies accurate CNNs using far less time and computation than training-based approaches.

  • Problem

    Existing architecture-search and accuracy-prediction methods require substantial training and do not provide reliable out-of-the-box prediction for unseen datasets.

  • Method

    TAPAS adapts a train-less accuracy predictor to dataset difficulty using a DCN, prior experiments, and encoded network architecture information.

  • Results

    TAPAS enables large-scale architecture searches in a few minutes on a single GPU without new training experiments.

  • Takeaways & Limitations

    TAPAS identifies accurate CNN architectures without executing training jobs during the search, using substantially less time and computational resources than training-based approaches.

  • Takeaways & Limitations

    The DCN is only a rough estimate of dataset difficulty and is tolerant to approximations.

Abstract

from arXiv · show

In recent years an increasing number of researchers and practitioners have been suggesting algorithms for large-scale neural network architecture search: genetic algorithms, reinforcement learning, learning curve extrapolation, and accuracy predictors. None of them, however, demonstrated high-performance without training new experiments in the presence of unseen datasets. We propose a new deep neural network accuracy predictor, that estimates in fractions of a second classification performance for unseen input datasets, without training. In contrast to previously proposed approaches, our prediction is not only calibrated on the topological network information, but also on the characterization of the dataset-difficulty which allows us to re-tune the prediction without any training. Our predictor achieves a performance which exceeds 100 networks per second on a single GPU, thus creating the opportunity to perform large-scale architecture search within a few minutes. We present results of two searches performed in 400 seconds on a single GPU. Our best discovered networks reach 93.67% accuracy for CIFAR-10 and 81.01% for CIFAR-100, verified by training. These networks are performance competitive with other automatically discovered state-of-the-art networks however we only needed a small fraction of the time to solution and computational resources.

1 Introduction

CNN architecture search methods often require many costly training experiments, while existing predictors remain dataset-specific or require partial training. TAPAS addresses this gap with dataset-adaptive, train-less prediction and supports rapid search on one GPU.

  • Large-scale CNN architecture search methods based on evolution or reinforcement learning require many training experiments, creating substantial time and resource costs.
  • Existing accuracy predictors either require partial training for each network or hundreds of trained networks on the target dataset, preventing near-real-time search.
  • TAPAS predicts peak accuracy on unseen datasets by adapting to dataset difficulty and reusing experiments accumulated across datasets.
  • TAPAS is designed to be dataset-independent, improve through accumulated experiments, and enable large-scale search on a single GPU within minutes.
  • The paper reviews prior approaches, presents TAPAS's three-component design, compares experimental results with the state of the art, and concludes with a summary.

2 Related work

Prior architecture-search methods rely on costly training, while accuracy predictors reduce cost only within narrower dataset or network settings. TAPAS is positioned to transfer knowledge across datasets by incorporating dataset difficulty.

  • Peephole [4] predicts accuracy from network structure but requires hundreds of networks trained on a new dataset and does not transfer knowledge to unseen datasets.
  • Learning-curve extrapolation, BNN, and ν-SVR [2] use partial learning curves but remain dataset- and network-specific and require extensive training.
  • Evolutionary and reinforcement-learning search methods train many networks; some use hundreds of GPUs for more than ten days, while ENAS [15] finds a CIFAR-10 network in 10 hours on one GPU.

3 Methodology

TAPAS combines dataset characterization, a lifelong experiment database, and a train-less predictor. The workflow filters prior experiments by dataset difficulty before predicting peak accuracy for many candidate architectures.

  • TAPAS estimates peak accuracy at low cost by combining compact dataset characterization with a dynamically growing database of trained networks and performance.
  • Dataset Characterization computes a Dataset Characterization Number (DCN) for an unseen dataset, which ranks datasets by difficulty.
  • The Lifelong Database of Experiments stores neural-network training experiments conducted across varied image-classification datasets.
  • The Train-less Accuracy Predictor uses an architecture and its dataset DCN to predict potentially reachable peak accuracy without training that network.
  • The workflow computes the new dataset's DCN, selects experiments from similarly difficult datasets, trains TAP for up to a few minutes, and then predicts across many architectures in seconds.

3.1 Dataset characterization (DC)

TAPAS characterizes dataset difficulty with a probe network and uses the resulting DCN to filter prior experiments and condition prediction. The DCN is intentionally a rough, low-cost estimate.

  • The DCN captures dataset difficulty because identical CNNs can achieve different accuracies on easy and challenging datasets despite shared basic properties.
  • The DCN is computed as the ten-epoch peak accuracy of a modest Deep normalized ProbeNet, producing a value in [0, 1].
  • DCN computation is inexpensive because characterization occurs once, needs only a few epochs, uses a modest network, and can subsample large datasets in images and pixels.
  • The DCN is a rough dataset-difficulty estimate and is therefore tolerant to approximations.

3.2 Lifelong database of experiments (LDE)

The Lifelong Database of Experiments (LDE) continuously accumulates architecture, training, dataset, and accuracy records, while initializing its contents with sampled networks across available datasets. For a new dataset, it selects experiments from datasets with similar characterization scores.

  • 3.2 Lifelong database of experiments (LDE): The LDE continuously ingests CNN architecture, training hyper-parameters, dataset characterization, and achieved accuracy from every experiment.
  • 3.2 Lifelong database of experiments (LDE): The initially empty LDE is populated by sampling 800 networks for each available dataset from a varied architecture search space.The search space includes convolutional settings, residual blocks, and skip connections.
  • 3.2 Lifelong database of experiments (LDE): Incremental layer-wise training obtains accuracies for all intermediary sub-networks at the cost of training the complete network.All networks use identical optimizer, learning rate, batch size, and weight initializer settings, which may limit peak accuracy by a few percent.
  • 3.2 Lifelong database of experiments (LDE): For an unseen dataset, the LDE returns experiments from datasets whose characterization scores satisfy a predefined similarity threshold.The threshold τ is set to 0.05 in the experiments.

3.3 Train-less accuracy predictor (TAP)

TAP predicts CNN accuracy iteratively from layer and sub-network encodings while conditioning predictions on dataset difficulty. Its encoding captures detailed architecture, computational, memory, and intermediate-accuracy information for progressively larger sub-networks.

  • 3.3 Train-less accuracy predictor (TAP): TAP uses dataset-difficulty knowledge from similar LDE experiments and iteratively predicts each larger sub-network from the preceding sub-network's accuracy.This combines DCN-based experience reuse with layer-by-layer prediction rather than one prediction from the entire network structure.
  • 3.3.1 Neural network architecture encoding: The architecture encoding combines current-layer information with information about the input-to-layer sub-network.Layer types include convolution, pooling, batch normalization, dropout, residual blocks, skip connections, and fully connected layers.
  • 3.3.1 Neural network architecture encoding: The sub-network encoding records layer count, inference FLOPs, inference memory, and accuracy initialized or updated from prior predictions.The first-layer accuracy is 1/Nc, prediction inputs use zero where appropriate, and prior TAP outputs supply subsequent intermediate accuracies.
  • 3.3.2 TAP architecture: TAP consists of two stacked LSTMs with 50 and 100 hidden units followed by a sigmoid fully connected output, and it receives DCN as a second input.The first input concatenates encoding vectors for two consecutive layers.
  • 3.3.4 TAP prediction: TAP training takes a few minutes on one GPU, and a trained predictor can be reused on datasets with similar DCN values without retraining.
  • 3.3.4 TAP prediction: The iterative mechanism feeds the predicted accuracy of sub-network l1:i into the next evaluation to predict l1:i+1, ending with the complete-network accuracy.

4 Experiments

TAPAS is evaluated against established predictors across single-dataset, multi-dataset, unseen-dataset, and simulated architecture-search scenarios. It consistently outperforms comparison methods while reducing search cost and identifying strong CIFAR networks without training during evolution.

  • Experimental setup: The experiments used nineteen datasets, including public datasets and ImageNet subsamples, with 800 networks generated per dataset under shared training settings.Images were resized to 32 × 32 pixels, and the datasets were ranked by difficulty for LDE construction.
  • Scenario A: TAP outperformed Peephole, LCE, BNN, and ν-SVR on all considered metrics in the single-dataset scenario.The comparison used MSE, Kendall’s Tau, and R2; BNN reported MSE 0.007 and ν-SVR reported R2 0.9 with 20% of the initial learning curve.
  • Scenario A: Peephole’s lower benchmark results were attributed to its encoding being insufficient for complicated structures such as ResNets.Its encoding tuple contains layer type, kernel height, kernel width, and channel ratio.
  • Scenario B: TAP also outperformed all comparison methods on every metric when trained on experiments from all datasets, without filtering by dataset.The comparison with TAP without DCN was considered fair because neither method used dataset-difficulty information.
  • Simulated large-scale evolution: 400 seconds on one GPU replaced a 256-hour search using 250 parallel workers, with no network trained during the simulated evolution.The simulation used the same mutations as reference work [16], excluding mutations unsuitable for prediction-only evaluation.

5 Conclusion

TAPAS predicts CNN peak validation accuracy for arbitrary input datasets by combining architecture information with dataset-difficulty characterization and accumulated experiment knowledge. The framework outperforms prior methods and enables accurate architecture searches without new training jobs.

  • Conclusion: TAPAS forecasts CNN performance at convergence for any given input dataset using a lifelong database of experiments and dataset-difficulty characterization.The conclusion identifies dataset-difficulty characterization as the key differentiation from prior methods.
  • Conclusion: TAPAS outperformed preexisting methods both when tuned for a specific dataset and when applied across datasets without dataset-specific bias.The conclusion states that this advantage also holds for completely new datasets without additional training experiments.
  • Conclusion: TAPAS identified highly accurate CNN architectures in a few minutes using one GPU and no training jobs during the search.The paper describes this as several orders of magnitude faster than training-based approaches.
Loading 1806.00250v1…