Hyperparameter and Neural Architecture Search via Evolutionary Optimization for Reinforcement Learning: HPNAS-RL

1 Introduction

• HPNAS jointly evolves discrete policy architectures and continuous RL hyperparameters, addressing EMNAS’s architecture-only search.NAS targets convolutional layers for visual feature extraction; HPO tunes parameters such as learning rate, batch size, discount factor, and time horizon.Source: §1 · p. 2

1.1 Autonomous Driving Task:

• The Unity3D task trains PPO agents on a three-lane oval track to drive smoothly while avoiding collisions and abrupt maneuvers.Agents receive semantic-segmentation images from a forward-facing camera and predict trajectory points executed by an external vehicle controller.Source: §1.1 · pp. 2–3
• The reward promotes accurate trajectory prediction, penalizes collisions and lane deviations, and encourages stable motion with minimal jerk.The study focuses on the convolutional policy architecture and PPO hyperparameters in this RL driving task.Source: §1.1 · p. 3

2 Related Work

• NAS approaches include RL-based search [6], differentiable methods such as DARTS, and evolutionary methods such as AmoebaNet.The cited discussion describes gradient-based search as efficient but often locally optimal and limited to differentiable objectives, while RL-based search requires extensive sequential evaluations.Source: §2 · pp. 3–4
• Most evolutionary and multi-objective NAS frameworks remain focused on supervised learning, limiting their fit to RL’s sample-efficiency and stability demands.RL also involves stochastic rewards, non-stationary environments, and scalable exploration across parallel agents.Source: §2 · p. 4
• Prior joint NAS–HPO approaches include Bayesian optimization, weight sharing, and evolutionary methods, primarily for static supervised tasks.The paper introduces a dual-branch framework for jointly optimizing architectures and RL-specific hyperparameters, using DAG-based representation, parallel population training, and transfer learning.Source: §2 · p. 4

3 HPNAS Methodology

• The architecture branch evolves convolutional network structures, while the hyperparameter branch tunes PPO settings such as learning rate, batch size, and discount factor.The branches extend the earlier EMNAS system.Source: §3 · p. 4
• The framework combines discrete architecture evolution with optimization of continuous RL hyperparameters.Source: §3 · pp. 4–5
• Both branches operate in a unified multi-objective evolutionary process and share fitness evaluations to jointly improve policy performance and model efficiency.Source: §3 · pp. 4–5

3.1 EMNAS Framework

• EMNAS evolves candidate architectures to maximize PPO cumulative reward while minimizing computational cost and model complexity.Its objectives are negative reward, FLOPS, and parameter count, all minimized to find a balance between performance and efficiency.Source: §3.1 · p. 5

3.1.1 Genotype Encoding Scheme

• Each normal or reduction cell is encoded as genes specifying convolutional operations and their feature-map connections.Normal cells preserve spatial resolution, while reduction cells halve it; the search space includes separable, dilated, and inverted convolutions, pooling, and skip connections.Source: §3.1.1 · pp. 5–6, Table 1 · p. 6
• Recursive connection indexing links operations to earlier nodes, maintaining an acyclic feature-flow structure and enabling flexible search over CNN cell configurations.The encoding jointly captures architecture and interlayer dependencies.Source: §3.1.1 · p. 6

3.1.2 Multi-objective Evolutionary Algorithm

• EMNAS ranks candidate architectures on three objectives and uses NSGA-II non-dominated sorting and crowding distance to preserve diverse Pareto-optimal solutions.The objectives balance PPO reward, computational cost, and model complexity, with equal weighting across the trade-off surface.Source: §3.1.2 · pp. 6–7

3.1.3 Genetic Operations

• Architectures are encoded as DAGs whose nodes are convolutional operators and whose directed edges specify input connections.Crossover, mutation, and survival operations generate offspring while preserving structural validity.Source: §3.1.3 · p. 7

Crossover:

• Uniform crossover forms a child by inheriting each gene from either parent with probability 0.5, mixing convolutional operations and input connections.This can produce novel layer combinations while retaining structural traits from both parents.Source: Crossover · p. 7

Mutation:

• Mutation replaces a randomly selected gene with probability pm, while inversion reverses a selected subgraph and maintains the DAG structure.Typical values are pm = 0.1 and pinv = 0.1; mutation is described as encouraging exploration and helping prevent premature convergence.Source: Mutation · p. 7

A survival

• Survival retains the best-performing architectures in the next generation, preserving strong solutions while offspring maintain population diversity.The passage describes this balance of exploitation and exploration as supporting convergence toward the Pareto-optimal frontier.Source: A survival · p. 8

Early Exit Parameter Initialization (EEPI):

• EEPI discards and reinitializes architectures exceeding the maximum parameter constraint, limiting early search to feasible model sizes.The passage says this reduces computational overhead during early search.Source: Early Exit Parameter Initialization… · p. 8

3.1.4 Optimized Transfer Learning (OTL)

• OTL transfers weights and policies from the previous generation’s best-performing model to student architectures through behavior cloning.This gives new populations an initialized policy rather than requiring training from scratch.Source: §3.1.4 · p. 8
• At each generation’s end, expert trajectories from the top policy provide state-action examples for supervised imitation by the next generation.The cited loss trains the student policy to maximize the log probability of expert actions.Source: §3.1.4 · p. 8
• OTL retains optimizer state, including the final learning rate and PPO clipping value, and reuses expert data and policy initialization to reduce PPO training time while maintaining stability and fairness.EMNAS combines OTL with EEPI and multi-objective optimization to search CNN architectures for RL performance and computational cost.Source: §3.1.4 · p. 8

3.2 Dual-Branch Extension: Joint NAS and HPO

• The dual-branch framework jointly evolves architectures and RL hyperparameters because both architectural and hyperparameter choices influence RL performance.NAS targets convolutional layers, while HPO tunes training hyperparameters.Source: §3.2 · p. 8

3.2.1 Hyperparameter Search Space

• The HPO search varies batch size, discount factor, learning rate, and time horizon; PPO epochs and entropy coefficient are fixed at 3 and 0.01.Both branches use a shared fitness balancing reward, FLOPs, and parameter count; hyperparameters are initially sampled within predefined ranges summarized in Table 2.Source: §3.2.1 · p. 9, Table 2 · p. 9

Polynomial mutation:

• Polynomial mutation perturbs continuous hyperparameters, with larger ηm producing smaller changes around the parent and smaller ηm allowing larger deviations.Mutated values are clipped to their allowable interval to maintain feasibility.Source: Polynomial mutation · pp. 9–10

Simulated Binary Crossover (SBX):

• SBX generates two offspring hyperparameter values from two parents, with larger ηc keeping offspring closer to parents and smaller ηc allowing wider spread.The framework uses SBX and polynomial mutation for continuous variables, while retaining EMNAS’s discrete crossover and mutation for architectures.Source: §3.2.2 · p. 9, Simulated Binary Crossover (SBX) · p. 10

3.2.2 Crossover and Mutation Mechanisms

• The continuous search uses SBX and polynomial mutation, while the architecture branch retains EMNAS’s discrete mutation and crossover.These operators support search across both architecture and training-dynamics spaces.Source: §3.2.2 · p. 9, Simulated Binary Crossover (SBX) · p. 10

4 Experimental Setup

• Candidates are evaluated with a multi-objective fitness balancing policy reward, FLOPs, and parameter count in Unity3D simulation using PPO.The setting is described as resource constrained.Source: §4 · p. 11
• The search uses 20% survival, 10% uniformly random offspring, and scheduled operator probabilities and distribution indices to shift from exploration toward exploitation.Low-fidelity candidate training uses 40 PPO epochs, 84×84×3 inputs, and eight stacked cells; architecture size is bounded at 5 million parameters.Source: §4 · p. 11
• Six generation-population settings are tested, evaluations run in parallel on NVIDIA Tesla H100 GPUs, and the final winning architecture is fully trained for comparison with NAS-only and HPO-only baselines.The study benchmarks against original EMNAS but omits the prior standalone HPO baseline because its full-epoch training is incompatible with the low-fidelity setting.Source: §4 · p. 11

• Table 3 compares EMNAS and HPNAS across six experiments using final PPO cumulative reward, parameter count, and FLOPs.Source: §5 · p. 11, Table 3 · p. 12
• In Experiment 4, NAS achieved reward 753 with 1.45M parameters and 1.44 GFLOPs, while HPNAS achieved reward 745 with 0.77M parameters and 0.76 GFLOPs.The authors describe this as a minimal reward trade-off for a more compact model.Source: §5 · p. 12
• With larger search budgets, HPNAS consistently outperformed NAS in reward, while model sizes remained comparable rather than always smaller.Source: §5 · p. 12
• In the two larger searches, HPNAS increasingly occupied the higher-reward, smaller-model region; Experiment 6 hyperparameter samples concentrated in high-reward regions over generations.The reported optimal settings were batch size 216, time horizon 88, learning rate 2.8×10−3, and discount factor 0.976.Source: §5 · pp. 12–13, Figure 3 · p. 13, Figure 4 · p. 13
• The fully trained HPNAS policy reached a peak cumulative reward of 1205, versus 1190 for NAS-only, 1190 for HPO-only, and 1140 for the manually tuned baseline.Source: §5 · p. 13

6 Conclusion

• Across multiple search budgets, HPNAS outperformed standalone NAS in cumulative reward and model efficiency, especially in larger-scale experiments.Source: §6 · p. 14

Outlook

• The current implementation only gathers training data for a Bayesian Neural Network surrogate; future versions plan to use it in the final 20% of search once predictive accuracy is sufficient.Source: Outlook · p. 14

• The publication states that its results, opinions, and conclusions are not necessarily those of Volkswagen Aktiengesellschaft.Source: Disclosure of Interests · p. 14