PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement learning to choose optimizers

Martin van der Schelling, Deepesh Toshniwal, Miguel A. Bessa

arXiv:2609.01811v1cs.NEcs.LGmath.OC

TL;DR

Existing optimizer-switching methods fix parts of the schedule, limiting online adaptation across optimizer families and decision intervals. RL2CO learns optimizer identity and duration sequentially with context conditioning and a decoupled actor-critic. It is evaluated on held-out and shifted problems after jointly designing the training tasks and portfolio to avoid optimizer dominance.

  • Problem

    Existing switching approaches restrict optimizer families, fix switch timing, or treat decision frequency as a hyperparameter rather than learning the complete schedule.

  • Method

    RL2CO uses a contextual sequential decision process whose recurrent policy selects both the next optimizer and its duration from a mixed gradient-based and derivative-free portfolio.

  • Results

    On 100 held-out problems and unseen problem families, RL2CO is evaluated against every portfolio optimizer and a random schedule, with the reported outcome showing robustness under distribution shift.

  • Takeaways & Limitations

    Learning the schedule online allows optimizer-family and duration choices to adapt across problems and stages of an optimization run.

  • Takeaways & Limitations

    The study focuses on differentiable design problems, so its scope is narrower than full analytical transparency.

Abstract

from arXiv · show

No single optimization method is uniformly best for all problems, and the most suitable optimizer choice can change during a run. Existing approaches that change optimizer during execution typically predetermine part of the strategy: the portfolio is restricted to one algorithm class, the switch occurs once at a fixed time, or the frequency of decisions is treated as a hyperparameter rather than a learned one. We introduce "Reinforcement Learning to Choose Optimizers", which formulates the optimization algorithm choice as a sequential decision-making problem. At each decision, a recurrent policy reads the current run state and decides both which optimizer should be used next and for how long. The portfolio includes both gradient-based and derivative-free optimizers, and each switch passes on the current best solution and a representative step size. A context proxy conditions a gating network over expert heads, and training employs a decoupled actor-critic whose return is expressed in the same empirical runtime distribution metric used at evaluation. Training tasks and portfolio are designed jointly so that no optimizer dominates. On unseen problems, the learned policy outperforms every portfolio optimizer at all but the smallest budgets, and it remains robust under distribution shift.

1 Introduction

RL2CO treats optimizer scheduling as a sequential decision problem that learns both which optimizer to use and how long to run it. It combines diverse optimizers, context conditioning, and joint task–portfolio design, then evaluates generalization on held-out and shifted problems.

  • Existing switching methods predetermine part of the strategy through restricted portfolios, single fixed switches, or fixed decision intervals.
  • RL2CO formulates an optimization run’s schedule as sequential control, selecting an optimizer and its duration at each decision point.The portfolio spans gradient-based and derivative-free methods, and switches transfer the best solution and a representative step size.
  • The policy conditions its decisions on a proxy for problem context and uses a gating network over expert heads.
  • Training tasks and the optimizer portfolio are designed jointly so that no portfolio member dominates across the training problems.
  • The policy is trained on 100 tasks and evaluated on 100 held-out problems against every portfolio optimizer and a random schedule, including tests on unseen problem families.

2 Related Work

Prior work addresses optimizer choice, switching, duration, or state transfer separately, but typically fixes the remaining decisions. RL2CO targets a unified schedule that adapts online across optimizer families and problem distributions.

  • A schedule represents update rules applied for specified iteration counts before switching, making schedule design the meta-algorithmic task.
  • Differentiable design problems are the paper’s scope because first-order optimizers require gradients, although the method’s requirement is narrower than full analytical transparency.
  • Automatic algorithm selection often commits to one optimizer, despite evidence that the best local-search operator depends on both the instance and search time.
  • Prior portfolios commonly restrict runtime selection to similar algorithms or one evolutionary family, while cross-family composition is often fixed in advance.
  • Existing approaches separately address which optimizer to use, when to switch, and what state to transfer, while usually fixing duration or reducing it to one iteration.The decision interval is therefore a framework hyperparameter, creating a trade-off between optimizer integrity and scheduling flexibility.
  • RL2CO learns how long each optimizer runs before revisiting the choice, enabling budget trade-offs between optimizer families with different iteration costs.

3 Methods

RL2CO models optimizer scheduling as a contextual Markov decision process in which actions choose an optimizer and duration. A recurrent, context-conditioned policy is trained with a decoupled actor-critic using convergence-based rewards and low deployment overhead.

  • 3.1 Optimizer Selection as a Contextual Markov Decision Process: Each task defines a context-dependent decision process, and the policy observes a projected run state before selecting actions and receiving rewards.
  • 3.1 Optimizer Selection as a Contextual Markov Decision Process: An action selects the next optimizer and a duration from a fixed set, while the optimization run itself realizes the context-dependent state transition.
  • 3.1 Optimizer Selection as a Contextual Markov Decision Process: The state combines each optimizer’s runtime state with global progress, and rewards measure potential reduction based on convergence area over the evaluation budget.
  • 3.2 Context Conditioning: A context proxy conditions the policy because identical trajectories can require different actions on different objective landscapes.A gating network combines recurrent expert heads specialized by context region.
  • 3.3 Training With a Decoupled Actor-Critic: Decoupled advantage actor-critic uses separate policy and value networks plus an auxiliary advantage head to reduce interference and support task generalization.
  • 3.4 Computational Overhead: After offline training, deployment requires one millisecond-scale policy forward pass per switching decision, leaving optimizer updates and objective evaluations as the dominant costs.

4 Results

The study jointly constructs diverse training tasks and a complementary optimizer portfolio, then evaluates performance using empirical runtime distributions. The resulting design avoids optimizer dominance and supports comparisons with virtual-best and static baselines.

  • Task-set design: Training uses BBOB and noisy BBOB suites to vary conditioning, modality, separability, and stochastic effects.
  • Task-set design: Over-parameterized neural-network loss landscapes are modeled as high-dimensional but effectively low-rank through explicit structural constraints.
  • Evaluation: Performance is measured by function evaluations needed to reach target objectives, aggregated across runs and targets into an empirical runtime distribution.The aggregation includes successful and unsuccessful trials, with unsuccessful trials represented by simulated restarts.
  • Portfolio design: The virtual best solver–single best solver gap quantifies algorithm-selection potential and serves as the joint design criterion for tasks and portfolio.
  • Portfolio design: The selected portfolio comprises four optimizers, including L-BFGS, Rprop, CR-FM-NES, and MR15-GA.The listed methods span quasi-Newton, first-order, natural-evolution-strategy, and genetic-algorithm approaches.

Appendix C.3 details the resulting selection and search procedure.

The learned procedure selects both an optimizer and its run duration, transferring the incumbent solution and search scale between heterogeneous methods. Across benchmark suites and unseen task families, it exploits exploitable selection structure and retains strong baseline behavior when that signal is weak.

  • The training portfolio separates across the budget range, with no optimizer leading throughout, supporting learned selection rather than a dominant default.
  • The learned policy reaches a larger fraction of targets than every portfolio optimizer at all but the smallest budgets, while informed decisions explain its margin over random scheduling.Only below roughly one evaluation per dimension does L-BFGS lead.
  • On CEC 2005, the policy finishes ahead of every portfolio constituent, with separation becoming clear only late along the budget axis.
  • On CEC 2017 and CEC 2013, the policy matches the random schedule and remains much better than random search and close to the single best solver.
  • On unseen problem families, the policy acts as a static selector for convex quadratics, adapts between optimizers for Gaussian classification, and outpaces individual methods on meta-optimization tasks.For meta-optimization, CR-FM-NES is about an order of magnitude faster than gradient-based methods, while all optimizers eventually hit the targets.
  • The policy chooses an optimizer and commitment duration from run-state observations, transfers the incumbent and search scale at switches, and uses a potential-based reward over evaluation progress.Episodes terminate at budget exhaustion, target attainment, or the decision horizon during training.

A.1 The Handshake

The handshake transfers enough search information for a new optimizer to continue from the incumbent solution’s current scale, while handling optimizer-specific internal state separately.

  • Switching optimizers initiates a handshake that converts outgoing trajectory data into the incoming optimizer’s internal-state parameters.
  • Transferring σ prevents a new optimizer from resetting to an unnecessarily global search scale after a switch.
  • The transfer preserves the incumbent solution, its objective value, a scalar search scale σ, and problem dimensionality d.
  • Incoming optimizer state is reset or resumed before the transferred quantities are used; L-BFGS is reset because its curvature pairs are optimizer-specific.

Appendix B Training Details

Training uses decoupled actor-critic updates over recurrent-policy rollouts, with decisions consuming variable numbers of optimizer iterations and policy objectives combining PPO, entropy, and auxiliary terms.

  • Training procedure: DAAC training samples task subsets, runs policy rollouts, updates the policy with minibatches, and updates the value network every N_V rounds.
  • Training procedure: Each rollout records observations, actions, rewards, termination flags, value predictions, and the optimizer-iteration duration Δ_t at every decision.
  • Advantage estimation: The backward pass computes temporal-difference residuals, generalized advantage estimates, and return targets, then normalizes advantages across the pooled batch.
  • Policy objective: The policy objective combines the clipped PPO surrogate, entropy over optimizer and duration heads, and an auxiliary advantage-prediction loss.
  • Critic and mixture updates: The mixture arm adds load balancing across experts, while the critic separately fits return targets by squared error every N_V rounds.
  • Policy state: A recurrent hidden state summarizes observation history because the optimization process is partially observable, while multi-iteration commitments shorten episodes.

Appendix C Benchmark Construction

This appendix explains how the profiling pool was produced for constructing the paper’s two task sets.

  • The appendix describes the procedure used to produce the profiling pool.
  • The profiling pool supplies the source material for the two task sets in Section 4.1.
  • Its stated purpose is task-set construction rather than evaluation of the learned policy.

C.1 Benchmark Functions

The benchmark construction implements noiseless and noisy BBOB functions, including high-dimensional embeddings, bounded evaluation protocols, and a broad optimizer candidate set.

  • Benchmark functions: The benchmark pool includes noiseless and noisy BBOB suites implemented according to their original specifications, with seeded instance transformations.
  • Training diagnostics: Figures 12 and 13 report mean cumulative return and critic loss against policy updates, respectively.
  • High-dimensional construction: High-dimensional functions embed a d_int-dimensional BBOB function in d dimensions using an orthonormal-row projection, shifted and scaled within the unit box.
  • Evaluation protocol: Each instance is optimized for up to 10^4 × d iterations, capped at 25,000, over 25 independent runs with Gaussian initial points.
  • Optimizer portfolio: The candidate set contains 52 optimizers plus random search: 28 gradient-based methods and 24 derivative-free methods.

C.3 Joint Problem and Portfolio Selection

The paper jointly selects benchmark problems and a complementary optimizer portfolio so that algorithm selection remains meaningful and no single optimizer dominates. A constrained binary program identifies four optimizers from 52 candidates across 100 instances, using exact portfolio enumeration.

  • Design criteria: Joint selection requires complementary optimizers, problem-dependent best choices, and no optimizer strong enough to serve as a default.These properties depend jointly on the selected problems and optimizers.
  • Optimization objective: The objective sums the per-problem gap between the portfolio’s virtual best solver and its average performance, measuring the benefit of informed selection.The virtual best is the strongest portfolio optimizer on each selected problem, while the average summarizes portfolio performance.
  • Constraints: A mean-performance constraint limits the best optimizer’s advantage over the portfolio average to ε, preventing a single optimizer from dominating.This permits the portfolio average to stand in for a single best solver whose identity depends on the selected problems.
  • Constraints: The virtual best over the selected portfolio must outperform every candidate optimizer by at least µ on the selected problems.This ties the chosen portfolio to the full candidate set rather than only comparing its members internally.
  • Implementation: The program uses 100 instances and selects K = 4 optimizers from 52 candidates, with ε = µ = 0.02.The portfolio search enumerates candidate portfolios, ranks them by linear-relaxation upper bounds, and solves promising cases exactly.

C.4 ERTD Breakdown

The ERTD breakdown compares training performance with test performance across dimensionality and objective stochasticity. Separate test distributions expose how the comparison varies across these problem groupings.

  • Training set: Figure 14a reports the empirical runtime distribution for the training set as a whole.It provides a reference distribution for the subsequent test-set comparisons.
  • Problem dimensionality: Figures 14b–14j show one ERTD for each test-set problem-dimensionality group.These panels resolve the comparison by dimensionality d.
  • Objective stochasticity: Figures 14k and 14l split the test set into deterministic and stochastic objective tasks.The split provides separate ERTDs for the two objective types.

Appendix D Additional Evaluation Families

The appendix evaluates additional optimization families, including random quadratic problems and synthetic neural-network classification tasks. These families vary conditioning, architecture, data generation, and outer hyperparameter optimization structure.

  • Random quadratics: Because Gaussian W is nonsingular with probability one, each quadratic has a unique solution and global minimum zero.The unique solution follows from W x = y for the randomly drawn square matrix W.
  • Synthetic classification: Synthetic classification instances train MLPs on 100 points from four Gaussian components using full-batch cross-entropy with L2 regularization.The MLPs use two inputs, ReLU hidden units, and a two-way softmax output, with regularization 5 × 10−4 on every weight.
  • Synthetic classification: The classification family varies 11 random seeds, three depths, and three widths, producing problems with 18–158 trainable parameters.Global minima are approximated through three optimization restarts combining cosine-annealed Adam and L-BFGS runs.
  • Nested optimization: A nested classification family tunes three Adam hyperparameters through an outer optimization over [0, 1]3.Each outer evaluation trains a small MLP for 20 Adam steps, with gradients obtained by backpropagating through the unrolled training steps.
Loading 2609.01811v1…