Source-linked AI summary
Peephole: Predicting Network Performance Before Training
Boyang Deng, Junjie Yan, Dahua Lin
TL;DR
Large architecture search and costly training make network design difficult to evaluate. Peephole predicts performance before training by encoding layers and integrating their representations with an LSTM. Experiments on more than a thousand networks report reliable, highly correlated predictions and strong performance across three metrics.
Problem
The large network-design space and costly training make it difficult to evaluate architectures and obtain performance feedback efficiently.
Source: §1 · p. 1Method
Results
Studies on over a thousand networks trained on CIFAR-10 and MNIST report reliable predictions highly correlated with actual performance, significantly outperforming previous methods on three metrics.
Source: §6 · p. 8Takeaways & Limitations
The authors conclude that Peephole’s predictions can support searching for better network designs while reducing the cost of evaluating architectures.
Source: §1 · p. 2The predictor is task-specific, and this work fixes other training choices such as initialization, learning-rate adjustment, and optimizer settings.
Source: §4.3 · p. 5
Abstract
from arXiv · showhide
The quest for performant networks has been a significant force that drives the advancements of deep learning in recent years. While rewarding, improving network design has never been an easy journey. The large design space combined with the tremendous cost required for network training poses a major obstacle to this endeavor. In this work, we propose a new approach to this problem, namely, predicting the performance of a network before training, based on its architecture. Specifically, we develop a unified way to encode individual layers into vectors and bring them together to form an integrated description via LSTM. Taking advantage of the recurrent network's strong expressive power, this method can reliably predict the performances of various network architectures. Our empirical studies showed that it not only achieved accurate predictions but also produced consistent rankings across datasets -- a key desideratum in performance prediction.
1. Introduction
Network design offers substantial performance gains, but its large search space and costly training make improvement difficult. Peephole predicts architecture performance before training, using layer representations and an LSTM to support faster design exploration.
1. Introduction
- The large network-design space and hours- or days-long training make exhaustive evaluation difficult and leave intuition-driven design increasingly challenging.Automatic search can spend substantial resources evaluating unpromising options without effective prior guidance.Source: §1 · p. 1
- Peephole predicts an architecture’s final performance before training, aiming to reduce the cost of finding good network designs.The paper contrasts post-training loops that take hours with predictor feedback available within 1 second per architecture.Source: §1 · pp. 1–2
- Unified Layer Code represents individual layers as fixed-dimensional vectors, and an LSTM integrates them across architectures of different depths and topologies.The predictor treats architecture as input and produces a score estimating performance, such as validation accuracy.Source: §1 · p. 2
- Block-based sampling generates architectures by combining blocks sampled from a Markov process, enabling broad design-space exploration on a limited budget while keeping samples reasonable.This addresses the cost of obtaining predictor training examples, each of which requires training a network.Source: §1 · p. 2
- Studies over more than a thousand networks report reliable predictions across a wide range of architectures and consistent rankings across datasets.The authors state that these predictions can provide an effective way to search for better network designs.Source: §1 · p. 2
2. Related Work
Prior network design relied heavily on manual intuition, while automatic search methods still require costly training feedback. Earlier performance predictors either target hyperparameters or use partially observed learning curves; this work instead seeks architecture-only feedback.
2. Related Work
- Manual CNN design based on intuition and experience has become less rewarding as the design space and training cost make improved designs harder to obtain.Representative architectures include AlexNet, VGGNet [21], GoogLeNet [23], ResNet [7], DenseNet [9], and DPN [3].Source: §2 · p. 2
- Automatic network-design methods use reinforcement learning but rely on actual training for feedback, consuming time and computational resources.Peephole differs by focusing on quickly obtaining performance feedback rather than developing an automatic design technique.Source: §2 · p. 2
- Earlier performance-prediction work includes hyperparameter surrogates and learning-curve extrapolation, with some methods adding handcrafted architecture features.These learning-curve methods use partially observed training, whereas Peephole predicts performance purely from architecture.Source: §2 · p. 2
3. Network Performance Prediction
Peephole predicts accuracy at a specified training epoch from network architecture alone, rather than extrapolating a curve after observing part of training. It encodes layers uniformly and uses an LSTM to build a network representation for prediction.
3. Network Performance Prediction
- Peephole predicts accuracy at a specified epoch from architecture alone, unlike earlier methods that observe an initial training segment and extrapolate the rest.The prediction function takes the architecture and epoch index as inputs; the method aims to predict the entire curve.Source: §3 · p. 3
3.1. Unified Layer Code
- Unified Layer Code represents each operation as (TY, KW, KH, CH), encoding type, kernel dimensions, and quantized output-to-input channel ratio.Layer Embedding maps the integer codes through lookup tables and concatenates the resulting vectors into a real-valued layer representation.Source: §3.1 · p. 4
3.2. Integrated Prediction
- For sequential networks, an LSTM processes layer representations in order; its final hidden state represents network structure, then an MLP combines that feature with the epoch index to predict accuracy.The stated scope focuses on sequential network structures.Source: §3.2 · p. 4
4. Training Peephole
This section explains how Peephole obtains training samples from constrained network architectures and learns to predict final-epoch performance. It also limits the predictor’s scope to a specific dataset and performance metric, with other training choices held fixed.
4. Training Peephole
- Preparing Peephole’s training data is difficult because architecture space grows exponentially with depth, training each sample is expensive, and many layer sequences are impractical.Random layer-sequence sampling could therefore spend resources on networks that are not reasonable options.Source: §4 · pp. 4–5
4.1. Block-based Generation
- Block-based Generation first designs individual blocks, then stacks them into a network following a skeleton.The approach draws on practice of constructing networks from structurally alike blocks [7] [24].Source: §4.1 · p. 5
- Each generated block has at most 10 layers and begins with a convolution; later layer types are sampled by a Markov chain using transition probabilities estimated from practical networks.The transition probabilities favor combinations such as convolution followed by batch normalization and an activation.Source: §4.1 · p. 5
- The skeleton uses three resolution stages of blocks, with max pooling between stages and average pooling followed by a linear classifier at the end.Inserted 1 × 1 convolutions adapt dimensions when adjacent layers do not match.Source: §4.1 · p. 5, Figure 4 · p. 5
- Block-based Generation constrains the sample space so generated networks are mostly reasonable and a training set can be prepared affordably..Source: §4.1 · p. 5
4.2. Learning Objective
- The learning objective uses smooth L1 loss to match the predictor’s output at epoch T to the network’s final-epoch performance.Although performance curves are available, the authors report that using only final-epoch results already yields reasonably good results.Source: §4.2 · p. 5
4.3. Discussions
- Peephole is task-specific, and the study holds other factors affecting performance, such as initialization and optimizer settings, fixed.The authors describe this setting as restrictive but aligned with typical architecture ablation studies, and plan to incorporate additional factors.Source: §4.3 · pp. 5–6
5. Experiments
The experiments evaluate Peephole on CIFAR-10 and MNIST, test transfer to ImageNet, and examine the learned representations. Across the two datasets, Peephole outperforms the compared predictors on all three metrics; further studies provide preliminary transfer results and qualitative evidence about its representations.
5.1. Experiment Configurations
- The experiments use CIFAR-10 and MNIST to study prediction across diverse architectures, training over a thousand networks because these moderate-scale datasets make that affordable.The study explores prediction rather than pursuing state-of-the-art performance on large-scale vision benchmarks.Source: §5.1 · p. 6, §5 · p. 6
5.2. Comparison of Prediction Accuracies
- Peephole consistently outperforms BNN and ν-SVR across all three metrics on CIFAR-10.The metrics are MSE, Tau, and R2; smaller MSE indicates more accurate predictions, while higher Tau and R2 indicate more consistent rankings and closer predictions.Source: §5.2 · p. 7, Table 3 · p. 7
- Figure 5 shows higher correlation between Peephole's predicted and actual accuracies than other methods, especially among high-accuracy networks.The scatter plots place actual CIFAR-10 validation accuracy on the x-axis and predicted accuracy on the y-axis.Source: §5.2 · p. 7, Figure 5 · p. 7
- On MNIST, Peephole significantly outperforms the other two methods across all metrics, despite easier accuracy prediction and harder ranking.Most networks achieve high MNIST accuracy, making precise accuracy predictions easier but consistent rankings harder.Source: §5.2 · p. 7, Table 4 · p. 7
5.3. Transfer to ImageNet
- On ImageNet, the CIFAR-10-selected network achieves moderately better accuracy than VGG-13 with substantially fewer parameters.This is a preliminary transfer study using a network selected by Peephole-predicted CIFAR-10 accuracy and then scaled up for ImageNet.Source: §5.3 · p. 7, Table 5 · p. 7
5.4. Studies on the Representations
- A learned LSTM cell's response rises at convolution layers across different blocks, suggesting it detects those layers without explicit direction.Figure 6 visualizes the cell response, with darker colors representing higher responses.Source: §5.4 · p. 8, Figure 6 · p. 8
- The t-SNE visualization shows a gradual transition from low- to high-performance networks, indicating that learned structural features contain performance-related information.Points represent validation-set networks and are colored by accuracy.Source: §5.4 · p. 8
6. Conclusion
Peephole combines a unified layer representation, an LSTM, and block-based architecture generation to predict network performance before training. Studies on more than a thousand CIFAR-10 and MNIST networks report reliable predictions and better performance than prior methods on three metrics.
6. Conclusion
- Peephole encodes individual layers with Unified Layer Code, integrates their information with an LSTM, and uses Block-based Generation to explore diverse reasonable designs within a constrained search space.Source: §6 · p. 8
- Across more than a thousand CIFAR-10 and MNIST networks, Peephole produces reliable predictions highly correlated with actual performance and significantly outperforms prior methods on three metrics.Source: §6 · p. 8
- The authors describe this work as a first step toward fast network-design search and plan to incorporate additional design factors and explore more effective optimization.Source: §6 · p. 8
A. Appendix
The appendix describes the ImageNet block selected using Peephole predictions and gives implementation details of Block-based Generation.
A. Appendix
- The ImageNet block was selected from randomly sampled validation architectures using Peephole, rather than generated by the Block-based Generation algorithm.Source: A. Appendix · p. 10
- Block-based Generation begins with a convolution layer, then chooses each next layer type using a Markov chain conditioned on the current layer.Source: A. Appendix · p. 10, Table 1 · p. 10
- The sampling scheme inserts batch normalization after convolution with probability 0.6 and limits blocks to fewer than 11 layers and fewer than 4 convolution layers.Source: A. Appendix · p. 10