Source-linked AI summary
Designing Neural Network Architectures using Reinforcement Learning
Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar
TL;DR
CNN architecture design is a large, labor-intensive search problem because designers must choose layer structures and hyperparameters. MetaQNN uses reinforcement learning to sequentially build architectures and improve its choices from validation rewards. On image-classification benchmarks, it outperforms comparable handcrafted and prior automated designs while remaining competitive with methods using more complex layers.
Problem
CNN architecture design involves numerous layer and hyperparameter choices, making the architecture space extremely large and manual exhaustive search infeasible.
Method
MetaQNN trains a Q-learning agent to sequentially select layers in a finite architecture space using ε-greedy exploration, validation accuracy rewards, and experience replay.
Results
MetaQNN networks beat existing designs using the same layer types, are competitive with more complex networks, and outperform previous automated network-design methods across image-classification tasks.
Takeaways & Limitations
Reinforcement-learning-based meta-modeling can generate tailored CNN designs and multiple varied high-performing architectures for image-classification tasks.
Takeaways & Limitations
The method restricts the state-action space with coarse discrete bins and currently uses shared Q-learning-phase hyperparameters across network topologies.
Abstract
from arXiv · showhide
At present, designing convolutional neural network (CNN) architectures requires both human expertise and labor. New architectures are handcrafted by careful experimentation or modified from a handful of existing networks. We introduce MetaQNN, a meta-modeling algorithm based on reinforcement learning to automatically generate high-performing CNN architectures for a given learning task. The learning agent is trained to sequentially choose CNN layers using $Q$-learning with an $ε$-greedy exploration strategy and experience replay. The agent explores a large but finite space of possible architectures and iteratively discovers designs with improved performance on the learning task. On image classification benchmarks, the agent-designed networks (consisting of only standard convolution, pooling, and fully-connected layers) beat existing networks designed with the same layer types and are competitive against the state-of-the-art methods that use more complex layer types. We also outperform existing meta-modeling approaches for network design on image classification tasks.
1 INTRODUCTION
MetaQNN addresses the large, labor-intensive CNN architecture design space with reinforcement-learning-based automated search. It discovers strong task-specific architectures using standard layer types and outperforms comparable handcrafted and automated designs.
- Motivation: CNN design requires choices about layer counts, ordering, and hyperparameters, creating a space too large for exhaustive manual search.These choices include convolution receptive-field size, stride, and number of receptive fields.
- Method: MetaQNN uses a Q-learning agent to sequentially select CNN layers in a finite architecture space without human intervention.The space is made finite by discretizing and limiting layer parameters.
- Method: The agent uses validation accuracy as reward, ε-greedy exploration, and experience replay to improve architecture selection.Experience replay accelerates learning by repeatedly sampling stored memories.
- Results: MetaQNN discovers architectures that beat existing networks using the same layer types, remain competitive with more complex designs, and outperform previous automated methods.The experiments use standard convolution, pooling, and fully connected layers on CIFAR-10, SVHN, and MNIST.
- Results: Top architectures from one dataset are competitive on other datasets, and multiple varied high-performing designs can be ensembled.This supports transfer-learning evaluation and ensemble construction within the reported experiments.
2 RELATED WORK
Prior work automated neural-network design using evolutionary, biologically inspired, and Bayesian-optimization methods, while reinforcement learning supplied exploration and experience-replay techniques used by MetaQNN.
- Automated design: Genetic algorithms were proposed to search neural-network architectures and weights, but NEAT-generated networks had not matched handcrafted networks on standard benchmarks.The cited limitation concerns performance relative to hand-designed networks.
- Automated design: Bayesian optimization was applied to network architectures and hyperparameters, including TPE-based selection of layer types and feed-forward-network hyperparameters.The passage reports that this approach failed to match handcrafted-network performance.
- Reinforcement learning: Reinforcement-learning research distinguishes exploration through environmental sampling from exploitation of learned information to find better paths.The cited applications include game-playing agents and robotic control.
- Reinforcement learning: MetaQNN incorporates Q-learning, ε-greedy exploration, and experience replay into its network-design algorithm.Experience replay had previously proved useful in experimental reinforcement-learning settings.
3 BACKGROUND
The paper formulates architecture search as a finite-horizon, finite-state, finite-action Markov decision process. Q-learning estimates action values through recursive reward maximization, while ε-greedy exploration and replay balance discovery with exploitation.
- MDP formulation: The architecture-search problem is modeled as a finite-horizon MDP with discrete finite state and action spaces and stochastic transitions.Finite horizon guarantees termination after finitely many time steps.
- MDP formulation: The agent receives transition-dependent rewards and seeks to maximize expected total reward across possible trajectories.The reward may be stochastic according to a transition-conditioned distribution.
- Q-learning: Even finite state and action spaces yield combinatorially many trajectories, motivating reinforcement learning and the action-value function Q∗.Q∗(s_i,u) represents the maximum expected total reward for an action from a state.
- Q-learning: Q-learning iteratively updates action values toward immediate reward plus discounted future value, without explicitly modeling environmental dynamics.The learning rate α weights new versus old information, while γ weights short-term versus future rewards.
- Exploration and replay: ε-greedy behavior takes random actions with probability ε and greedy actions with probability 1−ε, while annealing ε shifts the agent from exploration toward exploitation.Experience replay is beneficial when exploration costs are large.
4 DESIGNING NEURAL NETWORK ARCHITECTURES WITH Q-LEARNING
The method models CNN layer selection as a finite-horizon Markov Decision Process and uses Q-learning to search a constrained architecture space. The agent explores architectures with ε-greedy sampling, validation-accuracy rewards, and replay-based Q-value updates.
- MDP formulation: The agent sequentially selects CNN layers along feasible state-action paths until reaching a termination state, defining the resulting network architecture.Figure 2 represents convolutional and pooling layers by parameterized states and uses G for Softmax or Global Average Pooling termination.
- Q-learning training procedure: The agent selects layers with an ε-greedy strategy, trains each sampled topology, and receives validation accuracy as its reward.The exploration schedule decreases ε from 1.0 to 0.1, retaining a stochastic final policy.
- State and action space: Each state is a tuple of layer parameters, and layer depth constrains transitions into a directed acyclic graph with a maximum architecture depth.The state space includes convolution, pooling, fully connected, global average pooling, and softmax layer types.
- State and action space: The search space prevents invalid or impractical architectures by restricting transitions, limiting fully connected layers to two, and controlling representation size before fully connected layers.Consecutive pooling states are disallowed, and fully connected widths cannot increase across consecutive fully connected layers.
- Q-learning training procedure: Experience replay stores sampled topologies and validation performance, then applies Q-value updates to randomly selected sequences in reverse temporal order.Previously trained architectures are not retrained when resampled.
5 EXPERIMENT DETAILS
The experiments evaluate architecture search using rapid training during exploration and longer fine-tuning of the best discovered models. Experiments cover SVHN, CIFAR-10, and MNIST with dataset-specific validation and training procedures.
- Exploration training: During exploration, each topology is trained for 20 epochs with a quick training scheme and a 5,000-sample validation set.Dropout is inserted after every two layers, with increasing dropout probabilities across dropout layers.
- Fine-tuning and ensembling: After exploration, the ten best models are fine-tuned with a longer schedule, and the top five are used for ensembling.The selected models are ranked from those found throughout the exploration process.
- Datasets: For SVHN, exploration uses the original training set with 5,000 validation samples, while fine-tuning uses the original plus extended training data.The dataset contains 73,257 original training samples, 26,032 test samples, and 531,131 additional training samples.
- Datasets: For CIFAR-10, exploration uses 5,000 validation samples, the maximum layer depth is increased to 18, and final training runs on the entire training set.Hyperparameters are tuned using the same validation set before the final training scheme.
- Datasets: For MNIST, images undergo global mean subtraction, and final training runs for 40 epochs with the learning rate reduced every five epochs.The learning rate is decreased by a factor of 0.2 during final training.
6 RESULTS
MetaQNN’s Q-learning agent improves its ability to select CNN architectures as exploration decreases, and the discovered models outperform comparable manually designed and automated alternatives. The method also shows transfer-learning potential, while CIFAR-10 results reveal sensitivity to exploration time and coarse training.
- Model Selection Analysis: As ε decreases from 1 to 0.1, the agent consistently selects better-performing models; on SVHN, mean accuracy rises from 52.25% to 88.02%.Figure 3 reports rolling accuracy over 100 models and mean accuracy at different ε values for CIFAR-10 and SVHN.
- Model Selection Analysis: Top CIFAR-10 models range from 11.26 million to 1.10 million parameters with only a 2.32% decrease in test error, and often begin with a C(N, 1, 1) layer.The C(N, 1, 1) layer provides N learnable linear transformations of the input, analogous in spirit to color-space preprocessing.
- Prediction Performance: MetaQNN’s top model and five-model ensemble outperform CNNs using only standard convolution, pooling, and fully connected layers, while remaining competitive with complex-layer methods.The comparisons cover CIFAR-10, CIFAR-100, MNIST, and SVHN, with dataset-specific augmentation settings.
- Prediction Performance: MetaQNN outperforms prior automated network-design methods, achieving 6.92% error versus 21.2% on CIFAR-10 and 0.32% versus 7.9% on MNIST.These comparisons are reported against Bergstra et al. and Verbancsics & Harguess, respectively.
- Prediction Performance: An ensemble of the ten MNIST models achieves 0.28% test error, beating the reported state of the art without data augmentation.The ensemble was formed because validation errors among the top ten models were very similar.
- Limitations: On CIFAR-10, limited exploration and coarse training may have prevented discovery of more similarly strong models and caused some models to underperform.The best model performs 1–2% better than the next four, making the ensemble less accurate than the best individual model.
- Transfer Learning Ability: The best CIFAR-10 MetaQNN model performs well on other computer-vision tasks both from random initialization and through finetuning from existing weights.This evaluates transfer-learning ability using the CIFAR-10 model.
7 CONCLUDING REMARKS
MetaQNN uses reinforcement learning to generate tailored CNN architectures and can incorporate constraints such as model size or inference speed. Its current design limits the state-action space with coarse discrete bins, leaving larger-space methods and hyperparameter optimization as future directions.
- MetaQNN uses reinforcement learning to generate problem-specific CNN designs and outperforms previous meta-modeling methods and hand-crafted networks using the same layer types.
- The method can target supervised or unsupervised settings and optimize constraints such as model size, speed, and accuracy through state-action thresholds or reward penalties.
- The current implementation uses shared topology-training hyperparameters and coarse discrete state-action bins, motivating hyperparameter optimization and Q-function approximation for larger spaces.
- Q-learning trains by sampling CNN models, recording validation accuracy, storing experiences, and repeatedly updating Q-values from replayed trajectories.
- The agent samples architectures by selecting actions until termination, using either the highest-valued action or a uniformly random action under ε-greedy exploration.
- Representation-size bins restrict actions that would make intermediate signal representations too small, but this restriction introduces uncertainty in state transitions.
C MNIST EXPERIMENT
The MNIST experiment addressed overfitting in final models with increased dropout, a small regularization search, and weight warm-starting during tuning and final training.
- MNIST models were prone to overfitting, so the authors increased dropout and performed a small grid search over weight regularization.
- Tuning and final training were warm-started with weights learned after the first epoch of initial training.
D FURTHER ANALYSIS OF Q-LEARNING
Additional experiments indicate that Q-learning improves architecture selection as exploitation increases and can consistently find high-performing models across randomized runs.
- As the agent shifts toward exploitation, its architecture selection improves, with performance distributions varying across ε values.
- Across 10 independent runs on 10% of SVHN, randomized exploration produced variance but every run improved architecture performance.
- 88.25% mean accuracy with 0.58% standard deviation characterized the best model found in each run.
D.2 Q-VALUE ANALYSIS
The learned Q-values provide design insights: deeper networks assign higher value to larger convolutional receptive fields, suggesting a possible preference for larger filters at greater depth.
- The agent updates Q-values for layer-selection paths during ε-greedy exploration and analyzes final values by layer type and depth.
- At greater depths, convolutional layers with receptive field size 5 have higher Q-values than those with sizes 1 and 3.
- These Q-value patterns may inform new CNN designs by revealing relative benefits of state-space design parameters.
E TOP TOPOLOGIES SELECTED BY ALGORITHM
The appendix reports the highest-ranked architectures selected by Q-learning across CIFAR-10, SVHN, and MNIST, including test error and parameter counts. Additional figures show how accuracy distributions and average Q-values vary with exploration, layer depth, layer type, and convolutional receptive-field size.
- Selected architectures: Tables A1–A3 list the top five Q-learning-selected architectures for CIFAR-10 and SVHN and the top ten for MNIST.MNIST includes ten models because all ten were used in the final ensemble.
- Selected architectures: Each listed architecture is accompanied by its test-set prediction error and total number of parameters.
- Reported model selection: For SVHN and MNIST, the best test-set accuracy is not reported in the main-text tables because validation-selected models achieved 2.28% and 0.44% test error, respectively.
- Search behavior: At smaller ϵ values, accuracy distributions become more concentrated in high-accuracy ranges across SVHN, CIFAR-10, and MNIST.
- Search behavior: Average Q-values are plotted against layer depth for different layer types and against convolutional receptive-field sizes.