Source-linked AI summary
Auto-Keras: An Efficient Neural Architecture Search System
Haifeng Jin, Qingquan Song, Xia Hu
TL;DR
Existing NAS methods are computationally expensive because they evaluate and train many architectures from scratch. The paper uses Bayesian optimization to guide network morphism with an edit-distance kernel and tree-structured acquisition optimization, achieving strong benchmark performance and packaging the method as Auto-Keras.
Problem
Existing neural architecture search methods are computationally expensive because they evaluate many architectures and train each network from scratch.
Method
The method combines Bayesian optimization with network morphism, using an edit-distance neural network kernel and a tree-structured acquisition-function optimizer.
Results
The proposed method achieved the lowest error rates on MNIST, CIFAR10, and FASHION-MNIST within a limited search time.
Takeaways & Limitations
Auto-Keras makes the method locally usable through a concise interface, parallel CPU/GPU execution, and adaptation to different GPU-memory limits.
Takeaways & Limitations
Network morphism tends to grow architectures, so the search tree must also expand inner nodes to revisit smaller architectures.
Abstract
from arXiv · showhide
Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpful for NAS by enabling more efficient training during the search. In this paper, we propose a novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search. The framework develops a neural network kernel and a tree-structured acquisition function optimization algorithm to efficiently explores the search space. Intensive experiments on real-world benchmark datasets have been done to demonstrate the superior performance of the developed framework over the state-of-the-art methods. Moreover, we build an open-source AutoML system based on our method, namely Auto-Keras. The system runs in parallel on CPU and GPU, with an adaptive search strategy for different GPU memory limits.
1 INTRODUCTION
NAS automates neural architecture selection but is computationally expensive because many networks are evaluated and trained from scratch. The paper combines network morphism with Bayesian optimization and delivers an open-source AutoML system.
- NAS searches for neural architectures suited to a task and dataset, but existing methods often evaluate many networks and train each from scratch.Its time complexity is O(n¯t), where n is the number of evaluated architectures and ¯t is their average evaluation time.
- Network morphism changes a trained network's architecture while preserving its functionality, requiring only a few additional epochs for further training.Operations include inserting layers or adding skip-connections, reducing average training time.
- Bayesian optimization guides network morphism by selecting promising operations while reducing the number of trained neural networks.The approach targets expensive black-box evaluations analogous to hyperparameter tuning.
- The method uses an edit-distance neural network kernel and a tree-structured acquisition-function optimizer to handle architecture similarity and operation selection.The optimizer balances exploration and exploitation in the tree-structured search space.
- On MNIST, CIFAR10, and FASHION-MNIST, the proposed method achieved the lowest error rates within a limited search time.It was compared with state-of-the-art NAS methods.
- Auto-Keras provides a concise open-source interface, parallel CPU/GPU execution, and memory adaptation for different GPU limits.These features support local use by people without specialized programming and data-science expertise.
2 PROBLEM STATEMENT
The paper formulates NAS as finding an architecture with the lowest evaluation cost over a morphable architecture space. The space can vary convolutional filter counts and is broader than fixed-width alternatives.
- NAS seeks an architecture f* in search space F that minimizes a cost function on training and validation data.The cost may be an evaluation metric such as accuracy or mean squared error.
- The search space contains architectures reachable by morphing initial architectures, including changes to convolutional-layer filter counts.This makes the space larger than methods with fixed layer width.
3 NETWORK MORPHISM GUIDED BY BAYESIAN OPTIMIZATION
The method uses Bayesian optimization to guide network morphism through a tree-structured NAS space. It combines an edit-distance neural-network kernel, approximate distance computation, acquisition-function search, and graph-level operations that preserve tensor-shape consistency.
- Bayesian optimization: Bayesian optimization iteratively updates a Gaussian process, generates an architecture by optimizing an acquisition function, and observes its trained performance.The search alternates among update, generation, and observation steps.
- Neural network kernel: The edit-distance neural network kernel measures architectural similarity by estimating the operations needed to morph one network into another.More required edits correspond to greater distance and lower similarity.
- Neural network kernel: Network distance combines layer-edit distance and skip-connection edit distance, with λ balancing their contributions.Layer distance uses matched layers, while skip-connection distance accounts for matched and non-matched connections.
- Neural network kernel: Layer matching is optimized by dynamic programming, while skip-connection matching is formulated as a weighted bipartite graph problem solved with the Hungarian algorithm.The layer matching preserves topological order and minimizes the layer edit distance.
- Tree-structured search: The acquisition function is optimized over a tree in which architectures are nodes and morphism operations generate children.The search expands architectures according to acquisition or cost values, addressing the non-Euclidean, tree-structured NAS space.
- Graph-level network morphism: Graph-level morphism propagates layer-level changes through the computational graph to maintain intermediate tensor-shape consistency.The supported operations are inserting layers, widening nodes, and adding additive or concatenative connections.
4 AUTO-KERAS
Auto-Keras is an open-source deep-learning AutoML system designed for accessible local deployment. Its architecture coordinates CPU search, GPU training, storage, parallel execution, and GPU-memory adaptation.
- Auto-Keras targets domain experts who are not familiar with machine learning technologies and focuses on deep-learning tasks.
- The system is designed for concise, configurable use while accommodating limited local computation resources and varying GPU memory sizes.
- The workflow sends user requests through the API and Searcher, builds architectures in RAM, copies networks to GPU for training, and stores trained models.
- The system uses RAM for currently useful information and storage devices for the remainder to use computational resources and memory efficiently.
- Auto-Keras provides recovery from interrupted searches, addressing the longer running time caused by training multiple deep neural networks.
- CPU generation runs in parallel with GPU training through separate processes and a training queue, reducing idle time for both resources.
5 EXPERIMENTS
Experiments evaluate search effectiveness, efficiency, parameter sensitivity, and kernel quality across benchmark datasets. The proposed AK method achieves the lowest error rates, while Bayesian optimization and network morphism provide complementary efficiency benefits.
- Experimental setup: Experiments assess search effectiveness, efficiency gains from Bayesian optimization and network morphism, parameter sensitivity, and kernel-performance similarity.The evaluation uses MNIST, CIFAR10, and FASHION, with methods run for 12 hours on a single GPU.
- Effectiveness: AK achieves the lowest error rate on MNIST, CIFAR10, and FASHION, finding simple architectures on small datasets and more complicated structures on CIFAR10.The deployed AK-DP system achieves state-of-the-art performance on all three datasets.
- Effectiveness: SEAS trails AK because its hill-climbing search takes one morphing step at a time, whereas NASBOT’s training inefficiency limits its short-budget performance.The authors attribute AK’s desirable performance under limited resources and time to network morphism combined with its search strategy.
- Efficiency: Bayesian optimization finds lower-error architectures than BFS with similar numbers of searched networks, while its overhead is slightly higher.AK searches slightly fewer architectures than BFS within twelve hours because Bayesian optimization has higher time complexity.
- Efficiency: Network morphism preserves comparable search performance to BO while allowing AK to search many more architectures and achieve lower error within the same time.When BO and AK search similar numbers of architectures, their lowest error rates are similar; AK’s advantage comes from greater training efficiency.
- Parameter sensitivity: Error rate first decreases and then increases as β or λ grows, indicating that exploration–exploitation and layer–skip-connection distances require balance.Small β limits exploration, large β overexplores distant points, and extreme λ values ignore either skip-connections or layer differences.
- Kernel quality: Kernel-similar architectures tend to have similar performance, as shown by corresponding patterns in the proposed kernel and performance matrices.The matrices compare edit-distance kernel similarity with actual performance similarity on CIFAR10 using error rate as the cost metric.
6 CONCLUSION AND FUTURE WORK
The paper proposes efficient neural architecture search with network morphism guided by Bayesian optimization, implemented in the open-source Auto-Keras system. Experiments show good performance, including results that outperform traditional hyperparameter-tuning and state-of-the-art NAS methods.
- Bayesian optimization guides network-morphism search through a neural network kernel and a tree-structured acquisition-function optimization algorithm.
- The method is packaged as Auto-Keras, an open-source AutoML system with a simple interface.
- Experiments report good performance and outperformance of several traditional hyperparameter-tuning and state-of-the-art neural architecture search methods.
- Future work includes expanding the search space to recurrent neural networks, jointly tuning architectures and training hyperparameters, and designing task-oriented NAS.
APPENDIX: REPRODUCIBILITY
The reproducibility appendix documents implementation details and provides the code and documentation needed to use the method.
- The appendix covers default architectures, network-morphism operations, dataset preprocessing, and the training process.
- It also provides a proof of kernel validity and describes using ρ(·) to distort approximated architecture edit distances d(·, ·).
- Code and detailed documentation are available through the official Auto-Keras website.
A DEFAULT ARCHITECTURES
The default architectures use specified convolutional backbones and a shared fully connected design. Most methods use a three-layer CNN, while AK-DP additionally starts from ResNet18 and DenseNet121.
- Most methods use the same three-layer convolutional neural network, whereas AK-DP is initialized with ResNet18, DenseNet121, and the three-layer CNN.
- Each convolutional block contains ReLU, batch normalization, convolution, and pooling, with 3×3 kernels, stride 1, and 64 filters.
- All default architectures share global average pooling, dropout, two fully connected layers, ReLU, and softmax after the convolutional layers.
B NETWORK MORPHISM IMPLEMENTATION
The implementation initializes newly added weights for network morphism and lets Bayesian optimization select from a sampled pool of operations. Tree-structured search limits exhaustive enumeration while supporting varied insertion choices.
- The implementation initializes new weights and provides an operation pool, including selectable start and end points for skip connections.
- New weights start at zero, then receive the largest output-preserving noise that breaks symmetry and enables different values during backpropagation.
- Because the operation space is large, tree search samples possible operations until eight children are obtained rather than exhaustively enumerating them.
- Deep operations sample insertion locations and can add convolutional, activation, batch-normalization, dropout, or fully connected layers with equal probability.
C PREPROCESSING THE DATASETS
The benchmark datasets are normalized and augmented before neural architecture search, using channel-specific statistics to account for differing distributions.
- Benchmark datasets such as MNIST, CIFAR10, and FASHION are preprocessed before neural architecture search.Preprocessing includes normalization and data augmentation.
- The data are normalized to the standard normal distribution using a mean and standard deviation calculated separately for each channel.Channel-specific statistics account for different channel distributions.
- The mean and standard deviation are calculated from the training and validation sets together, and the testing set uses the same normalization.
D PERFORMANCE ESTIMATION
Performance estimation trains each candidate architecture while adapting training to architecture size and reducing noise in validation metrics. Early stopping and averaging recent metrics provide the estimation strategy.
- Candidate neural architectures are trained directly to estimate performance for updating the Gaussian process in Bayesian optimization.The paper emphasizes observation quality as essential to the neural architecture search algorithm.
- Training must adapt to different architectures because networks require different numbers of epochs to converge.The performance estimate should also avoid noise in the validation performance curve.
- Training stops when validation loss fails to decrease for τ epochs, making the procedure more adaptive than fixed-epoch training.This is implemented as a maximum-threshold early-stopping criterion.
- The estimated performance is the mean metric value over the last τ validation epochs rather than the final validation metric.The paper states that this is more accurate because it reduces the effect of performance noise.
E VALIDITY OF THE KERNEL
The paper establishes that the neural-network distance d is a metric and that the associated kernel is valid. It then uses Bourgain embedding to convert architecture edit distances into Euclidean distances during Bayesian optimization.
- Metric validity: Theorem 1 establishes that d(fa, fb) is a metric-space distance.The proof addresses non-negativity, definiteness, symmetry, and the triangle inequality.
- Metric validity: The proof derives non-negativity, definiteness, and symmetry from the layer and skip-connection distance definitions and their matching restrictions.Zero distance implies equal matched layers and skip connections, while the matching and component distances are symmetric.
- Metric validity: The distance satisfies the triangle inequality by constructing a direct matching between architectures through an intermediate architecture.The construction decomposes layer sets and bounds the resulting distances before extending the argument to skip connections and complete networks.
- Kernel validity: Theorem 2 states that κ(fa, fb) is a valid kernel, using the positive definiteness of a generalized RBF kernel based on squared distances.
- Euclidean embedding: Bourgain embedding takes the neural-architecture edit-distance matrix as input and outputs Euclidean vectors whose distances approximate the original metric.The embedding is applied during Bayesian optimization updates, expanding the stored distance matrix when new architectures are added.