PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach

Romain Claret, Michael O'Neill, Paul Cotofrei, Kilian Stoffel

arXiv:2609.00449v1cs.NEcs.AI

TL;DR

The paper addresses limited evidence on optimizing ES-HyperNEAT hyperparameters and transferring optimized configurations across tasks. It applies TPE to MNIST and evaluates transfer to logic operations and Fashion-MNIST, finding stronger transfer to the more complex Fashion-MNIST task than to simpler logic operations.

  • Problem

    Hyperparameter choices strongly affect ES-HyperNEAT performance, but evidence on TPE optimization and cross-task transferability remains limited.

  • Method

    The study uses TPE to optimize ES-HyperNEAT on MNIST and tests the best configuration on logic operations and Fashion-MNIST.

  • Results

    Transfer was effective for Fashion-MNIST but less conclusive for simpler logic operations, while TPE found a 29.00% MNIST configuration.

  • Takeaways & Limitations

    Optimized hyperparameters may generalize across tasks when source and target problems have suitable complexity and similarity.

  • Takeaways & Limitations

    Future work should explore alternative optimization algorithms, expand the search space, and study transferability across broader task domains.

Abstract

from arXiv · show

Neuroevolution of Augmenting Topologies (NEAT) and its advanced version, Evolvable-Substrate HyperNEAT (ES-HyperNEAT), have shown great potential in developing neural networks. However, their effectiveness heavily depends on the selection of hyperparameters. This study investigates the optimization of ES-HyperNEAT hyperparameters using the Tree-structured Parzen Estimator (TPE) on the MNIST classification task, exploring a search space of over 3 billion potential combinations. TPE effectively navigates this vast space, significantly outperforming random search in terms of mean, median, and best accuracy. During the validation process, the best hyperparameter configuration found by TPE achieves an accuracy of 29.00\% on MNIST, surpassing previous studies while using a smaller population size and fewer generations. The transferability of the optimized hyperparameters is explored in logic operations and Fashion-MNIST tasks, revealing successful transfer to the more complex Fashion-MNIST problem but limited to simpler logic operations. This study emphasizes a method to unlock the full potential of neuroevolutionary algorithms and provides insights into the hyperparameters' transferability across tasks of varying complexity.

1 INTRODUCTION

This study investigates TPE-based hyperparameter optimization for ES-HyperNEAT on MNIST and examines whether optimized configurations transfer across tasks of varying complexity. The motivation is that hyperparameter choices strongly affect neuroevolutionary search, while MNIST remains challenging for pure HyperNEAT.

  • 1 INTRODUCTION: TPE is used to optimize ES-HyperNEAT hyperparameters on the MNIST classification task.ES-HyperNEAT is presented as an indirect-encoding extension of NEAT.
  • 1 INTRODUCTION: Neuroevolutionary performance depends heavily on hyperparameter selection because hyperparameters affect the search process and resulting network architectures.The paper positions hyperparameter optimization as central to improving ES-HyperNEAT performance.
  • 1 INTRODUCTION: TPE models well-performing hyperparameter distributions to explore configurations and evaluates transferability to logic operations and Fashion-MNIST.The transfer study targets tasks with differing complexity.

2 BACKGROUND

The background introduces neuroevolution, the NEAT family, Bayesian optimization, and TPE's density-based expected-improvement search. ES-HyperNEAT extends HyperNEAT by evolving substrate topology alongside CPPN-generated connectivity patterns and weights.

  • 2 BACKGROUND: NEAT evolves neural-network weights and structure from a minimal network through mutation and crossover.Its topology grows by adding nodes and connections.
  • 2 BACKGROUND: ES-HyperNEAT extends HyperNEAT by evolving the substrate topology together with the CPPN.HyperNEAT uses CPPNs to generate connectivity patterns and weights for a substrate network.
  • 2 BACKGROUND: Bayesian optimization builds a probabilistic objective model to guide evaluations toward promising regions of expensive black-box search spaces.This makes it suitable for hyperparameter optimization.
  • 2 BACKGROUND: TPE models separate hyperparameter densities for low- and high-objective outcomes.The estimators are denoted l(x) and g(x), respectively.
  • 2 BACKGROUND: At each iteration, TPE selects hyperparameters by maximizing expected improvement and updating its density estimators.The acquisition function uses the current best objective value and the probability of observing an objective value given x.

3 RELATED WORK

Prior work established NEAT-family capabilities and the importance of hyperparameters, but largely relied on hand-tuned settings. This study addresses the limited evidence on applying TPE specifically to ES-HyperNEAT and transferring optimized configurations across tasks.

  • 3 RELATED WORK: NEAT research has expanded through indirect encodings, modular and hierarchical structures, and applications in deep learning, reinforcement learning, and robotics.A systematic review categorizes these developments by contributions and application areas.
  • 3 RELATED WORK: Prior studies established that hyperparameters affect NEAT and HyperNEAT performance, while ES-HyperNEAT demonstrations commonly used hand-tuned settings.Hand-tuned hyperparameters produced strong results across several tasks.
  • 3 RELATED WORK: Bayesian optimization and TPE have shown promise for efficiently exploring high-dimensional machine-learning hyperparameter spaces.This supports considering TPE for ES-HyperNEAT optimization.
  • 3 RELATED WORK: More research is needed on applying TPE to ES-HyperNEAT and transferring optimized hyperparameters from a source task to targets with lower or similar complexity.The paper frames these topics as its research gap.

4 EXPERIMENTAL SETUP

The experiments compare TPE and random search for ES-HyperNEAT on MNIST, then test the best MNIST configuration on logic operations and Fashion-MNIST. Evaluation uses broad performance metrics, statistical tests, and controlled task-specific protocols.

  • 4 EXPERIMENTAL SETUP: The study conducts hyperparameter investigation on MNIST and transferability experiments on logic operations and Fashion-MNIST.The MNIST experiment compares TPE with random search, while transfer tests the best MNIST configuration.
  • 4 EXPERIMENTAL SETUP: The experiments run in Python with Pureples, NEAT-Python, Optuna, and CPU-only infrastructure across 13 machines.The machines vary in CPU performance and RAM size.
  • 4 EXPERIMENTAL SETUP: Sixteen ES-HyperNEAT hyperparameters define a search space exceeding 3 billion potential configurations.Parameters outside the search space use the XOR configuration or library defaults.
  • 4 EXPERIMENTAL SETUP: MNIST fitness uses batches of 200 class-balanced random images per generation and compares predicted classes with ground-truth labels.The phenotype output index with the maximum value determines the predicted class.
  • 4 EXPERIMENTAL SETUP: The transfer study evaluates the best MNIST configuration on XOR, OR, AND, NOR, XNOR, NAND, and Fashion-MNIST tasks.Logic tasks use all possible input combinations, whereas Fashion-MNIST follows the MNIST evaluation approach with 20 generations per trial.
  • 4 EXPERIMENTAL SETUP: Performance is assessed with mean, median, best, worst, and standard-deviation accuracy, alongside unequal-variance two-sample t-tests.The tests compare TPE with random search and transferred configurations with random search.

5 EXPERIMENTAL RESULTS

The experiments evaluate TPE against random search for ES-HyperNEAT on MNIST and test whether optimized configurations transfer to logic operations and Fashion-MNIST. TPE outperforms random search on MNIST, while transfer performance varies across target tasks.

  • MNIST classification: TPE-Search consistently outperforms RandomSearch-292 on MNIST mean, median, and best accuracy across 30 independent validation runs.The comparison used TPE-Search with 2013 trials and RandomSearch-292 with 292 trials.
  • MNIST classification: 28.0% accuracy was achieved by TPE-Best during MNIST validation, compared with 27.5% for TPE-Search and 23.90% in a previous HyperNEAT study.TPE-Best used a population size of 256 and 2500 generations, smaller than the cited previous study’s configuration.
  • MNIST classification: Highly significant differences with large effect sizes separate TPE-Search, TPE-Best, and RandomSearch-292 on MNIST.The tests used two-sample t-tests assuming unequal variances.
  • MNIST classification: The TPE-discovered optimal configurations varied in CAP, NDP, IC, CF, AD, MD, VT, MW, and Act, while several other ES-HyperNEAT parameters remained constant.The varying parameters include both NEAT and substrate parameters.
  • Logic operations: MNIST-Config outperforms RandomSearch on every logic operation, with the largest improvements reported for OR, NAND, and XOR.MNIST-Config reaches 100% accuracy for OR, 87.68% mean accuracy for NAND, and 83.33% mean accuracy for XOR.
  • Fashion-MNIST classification: Transfer to Fashion-MNIST improves mean accuracy but does not produce the highest best accuracy, indicating task-dependent transfer performance.MNIST-Config reaches 20.00% mean accuracy versus 11.93% and 11.60% for the two random-search baselines, but its best accuracy is 22.50%.

6 DISCUSSION

The study finds that optimized ES-HyperNEAT hyperparameters transfer unevenly across tasks: performance benefits are strongest for Fashion-MNIST, while several logic-operation results are less conclusive. Validation supports TPE’s robustness, but corrected baselines and task complexity constrain interpretation.

  • Transferability: Transfer to XOR, OR, and NAND was promising, whereas AND, NOR, and XNOR showed limited or statistically nonsignificant benefits.The contrast may reflect differences in task simplicity and the specific target-task characteristics.
  • Transferability: The best MNIST hyperparameter configurations achieved 20.77% mean accuracy on Fashion-MNIST, exceeding RandomSearch-30’s 11.93%.This supports transfer to a related, more complex task and may reduce the need for extensive target-task search.
  • Robustness: The parallel-execution bug overestimated random-search baselines for AND, XOR, NOR, and XNOR, with p ≤0.001.It did not significantly affect MNIST, OR, NAND, or Fashion-MNIST baselines; Fashion-MNIST search occurred after correction.
  • Robustness: Validation assessed 885 TPE configurations, and 72 surpassed the corrected random-search best of 22.00%.Statistical comparisons also found highly significant differences and large effect sizes between TPE-Val-Best and corrected random-search results.
  • Limitations and Future Work: Future work should compare alternative optimizers, expand the hyperparameter search space, and study transferability across broader task domains.Proposed extensions include CMA-ES, finer-grained parameters, altered substrate initialization, additional hidden layers, and larger populations.

7 CONCLUSION

The study demonstrates TPE-based optimization of ES-HyperNEAT and reports stronger performance than random search, including a 29.00% MNIST accuracy with fewer resources than previous studies. Transferability is most effective for the related, more complex Fashion-MNIST task, while simpler logic tasks yield less conclusive results.

  • Optimization Results: 29.00% accuracy was achieved on MNIST by the best TPE-discovered configuration, using a smaller population and fewer generations than previous studies.The result is presented as evidence that TPE can identify performant ES-HyperNEAT hyperparameters.
  • Optimization Results: TPE outperformed random search in identifying performant ES-HyperNEAT hyperparameter configurations on MNIST.The study also emphasizes rigorous validation and consistent comparisons across experiments.
  • Transferability: The best MNIST configuration transferred effectively to Fashion-MNIST but less conclusively to certain simpler logic operations.The findings suggest transfer effectiveness depends on task complexity and similarity.
  • Implications: The study highlights rigorous validation as important for assessing hyperparameter optimization and transferability results.Its implications include groundwork for more efficient and adaptable neuroevolutionary systems and further transferability research.
Loading 2609.00449v1…