PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Orchestrate Evolutionary Search: Progression-Aware Deep Reinforcement Learning for Dynamic DE-CMA-ES Coordination in Optimization and Structural Model Updating

Lechen Li, Rongye Shi, Wanhuan Zhou

arXiv:2610.11546v1cs.NEcs.AI

TL;DR

High-dimensional structural model updating involves coupled parameters and nonconvex search landscapes, while evolutionary search must balance exploration and exploitation. DRL-DCO uses a learned actor–critic policy to continuously coordinate DE and CMA-ES and adjust search resources. The authors report improved efficiency and stability on high-dimensional benchmark functions and a practical FEMU task.

  • Problem

    High-dimensional FEMU has strongly coupled parameters and a nonconvex, multimodal objective, while evolutionary search also faces costly evaluations and difficult exploration–exploitation calibration.

    Source: §1.1 · p. 2, §2.1 · p. 4

  • Method

    DRL-DCO uses a DDPG-based actor–critic agent to continuously coordinate DE and CMA-ES using progression-aware state information and adaptive control.

    Source: §4 · p. 24

  • Results

    DRL-DCO improves efficiency and stability on high-dimensional nonconvex benchmark functions and the practical FEMU task.

    Source: §4 · p. 25

  • Takeaways & Limitations

    After training, the actor generates search controls through forward inference alone, without critic evaluation or further policy updates.

    Source: §4 · p. 25

    The approach inherits stochastic uncertainty from neural-network training, coupling target-problem optimization with stochastic policy training.

    Source: §4 · p. 25

Abstract

from arXiv · show

Solving high-dimensional structural model updating problems requires an algorithm capable of navigating complex, non-convex landscapes with correlated parameters. Existing hybrid evolutionary algorithms typically rely on static architectures or fixed switching rules, resulting in disjointed search phases. To address this, this study proposes a Deep Reinforcement Learning-governed dynamic DE-CMAES Orchestration (DRL-DCO) algorithm, in which a Deep Deterministic Policy Gradient (DDPG)-based actor-critic agent continuously governs the evolutionary process as a single, unified system rather than a mechanical concatenation of algorithms. Guided by a progression-aware state representation and a diversity-informed reward, the agent fluidly reallocates computational resources between the differencevector-based exploration of Differential Evolution (DE) and the covariance-guided exploitation of CMA-ES, while jointly regulating population size, elite preservation, and a restart mechanism to escape local optima. This allows DRL-DCO to autonomously transition between exploration-dominant, exploitation-dominant, and mixed-strategy regimes across generations. Beyond the training phase, the trained actor can operate in a supervision-free inference mode, where the internalized policy autonomously orchestrates DE and CMA-ES control from observed search states through forward inference alone, without critic evaluation or weight updates, enabling faster deployment while retaining full effectiveness. Validated on high-dimensional single-objective optimization benchmarks and the IASC-ASCE structural health monitoring benchmark, DRL-DCO achieves superior convergence accuracy and robustness compared to state-of-the-art adaptive and hybrid evolutionary algorithms, as well as single-operator DRL-governed baselines.

1. Introduction

Structural model updating and continuous-variable optimization pose demanding search problems, while existing approaches leave gaps in adaptive optimization and generalizable RL-guided evolutionary control. DRL-DCO coordinates DE and CMA-ES through learned control, with progression-aware state information and adaptive search resources.

1.1 Background and existing studies

  • Evolutionary model updating faces costly model evaluations and experience-intensive hyperparameter calibration, while balancing exploration and exploitation to avoid local minima remains difficult.Bayesian optimization can accelerate convergence and reduce FEM evaluations, but remains a surrogate-based framework rather than a direct optimization-strategy advance.Source: §1.1 · p. 2
  • Existing RL-assisted optimization remains challenged by problem-specific generalization, limited hyperparameter control, and relatively little work on continuous-variable structural updating.Many ERL methods also rely on supervised-style feedback, large populations, and lengthy training, limiting efficient inference after training.Source: §1.1 · pp. 2–3

1.2 Research focus and key contributions of this study

  • The study evaluates DRL-DCO on high-dimensional single-objective benchmarks and a practical finite element model updating task.The evaluation spans synthetic optimization landscapes and structural engineering generalization.Source: §1.2 · p. 3
  • DRL-DCO continuously allocates search effort between DE and CMA-ES on the non-elite population, rather than hard-switching operators, retaining CMA-ES distribution structure and DE exploration.The authors report that neither DE-only nor CMA-ES-only configurations match the full method across all tested problem classes.Source: §1.2 · p. 3
  • Its landscape-informed state augments candidate solutions with progression descriptors, including population diversity, distributional shape statistics, and convergence-rate indicators.These descriptors give the actor a temporally coherent view of search dynamics and geometry for adjusting operator hyperparameters.Source: §1.2 · p. 3
  • Learnable population scaling and elite preservation adjust population size and inherited elite proportion according to search state.The method uses smaller populations during exploitation-dominant phases and expands population coverage during exploration-demanding phases.Source: §1.2 · p. 3
  • A learned restart probability uses stall-count and diversity statistics to determine when to reinitialize populations, while step-size and covariance-scaling actions regulate CMA-ES search.The latter actions regulate search radius and distribution shape in support of the exploration–exploitation balance.Source: §1.2 · pp. 3–4
  • After critic-supervised training, the trained actor controls search through forward inference alone, without critic evaluation, network updates, or reward feedback.The policy can be reused across optimization runs without online adaptation.Source: §1.2 · p. 4

2. Methodology

The methodology frames evolutionary optimization as a controlled stochastic process governed by a learned policy. The agent observes search statistics and fitness progression to coordinate DE and CMA-ES within one process.

2. Methodology

  • DRL-DCO treats evolutionary optimization as a controlled stochastic process, replacing static architectures or predefined switching rules with learned regulation of DE and CMA-ES.The framework integrates dynamic orchestration, population evolution, and reinforcement-learning-based adaptive search.Source: §2 · p. 4
  • The policy uses population statistics, fitness-progression signals, and convergence indicators to coordinate DE’s difference-vector search with CMA-ES’s covariance-based search.These search mechanisms operate within a single unified process.Source: §2 · p. 4
  • The methodology proceeds from FEMU formulation to the controlled-process view, DRL mechanism, orchestration mechanism, and implementation details.These components are presented in Sections 2.1–2.5, respectively.Source: §2 · p. 4

2.1 Problem formulation of FEMU

The FEMU problem is formulated as identifying uncertain structural parameters by minimizing differences between simulated and measured modal characteristics. Parameter coupling and model uncertainty create a nonconvex, multimodal landscape, and this study addresses modal-property updating rather than preceding system identification.

2.1 Problem formulation of FEMU

  • FEMU identifies uncertain structural parameters by minimizing discrepancies between simulated and measured characteristics, using a composite objective of natural-frequency MSE and MAC-based mode-shape error.The frequency and mode-shape error weights are each set to 0.5.Source: §2.1 · p. 4
  • Strong parameter coupling and modeling uncertainty produce a nonconvex, multimodal objective landscape in which distinct parameter combinations can yield similar modal responses.Gradient-based methods can remain prone to local optima and sensitive to the initial parameter guess.Source: §2.1 · p. 4
  • These landscape properties motivate gradient-free, population-based strategies that explore broadly without relying on gradients or prior parameter information.The goal is to locate the global optimum.Source: §2.1 · pp. 4–5
  • The study focuses on second-stage modal-property optimization, not the preceding system-identification procedure.Modal properties are obtained through eigenvalue analysis, reducing repeated finite-element evaluation burden while retaining dynamic information for parameter identification.Source: §2.1 · p. 5

2.2 Evolutionary Optimization as a Controlled Stochastic Process

The section recasts population evolution as a controlled stochastic system: conventional evolutionary operators and hyperparameters are fixed or heuristically adjusted, whereas DRL-DCO makes operator behavior depend on an agent’s observed state and action.

2.2 Evolutionary Optimization as a Controlled Stochastic Process

  • At generation t, the population comprises Ns candidate solutions, each represented by an Nd-dimensional parameter vector.The population evolves through an evolutionary transformation.Source: §2.2 · p. 5
  • The evolutionary operator covers mutation, recombination, selection, and adaptation, while its hyperparameters govern those operations.Conventional EAs specify these mechanisms before optimization or adjust them with predefined heuristics.Source: §2.2 · p. 5
  • In contrast to fixed or heuristic controls, the policy makes search dynamics responsive to the landscape encountered during optimization.Conventional hyperparameters can leave those dynamics largely decoupled from the actual landscape.Source: §2.2 · p. 5
  • DRL-DCO formulates evolution as a controlled stochastic system in which the agent’s action vector parameterizes the operator and changes its behavior as optimization progresses.The population is the underlying system state, and the action regulates exploration, exploitation, and search adaptation.Source: §2.2 · p. 5

2.3 Markov Decision Process Formulation for the Optimization

The optimization is formulated as a state-dependent control process in which a DDPG agent regulates evolutionary search using population and progress information. Its continuous controls coordinate DE and CMA-ES mechanisms, while reward feedback balances improvement, diversity, and stagnation.

2.3 Markov Decision Process Formulation for the Optimization

  • The optimization is formalized as an MDP in which the agent learns a policy to govern the evolution operator from observed search states and reward feedback.Population fitness supplies the basis for evaluating candidates and constructing the reward at each generation.Source: §2.3 · p. 5

2.3.1 State Space

  • The state combines normalized performance indicators with population geometry, including diversity, progress through generations, recent improvement, and dimension-wise spread.These descriptors let the agent account for convergence status and population shape rather than relying only on objective values.Source: §2.3.1 · p. 6

2.3.2 Action Vector and Control Variables

  • The continuous action vector controls DE mutation and crossover, population scaling, CMA-ES covariance and step-size adaptation, restart probability, elite retention, and offspring allocation.The allocation ratio β(t) governs the relative share of DE- and CMA-ES-generated offspring.Source: §2.3.2 · p. 6
  • Coordinated controls let search shift smoothly among exploration-dominant, exploitation-dominant, and mixed regimes instead of switching discretely between algorithms.The action formulation applies across FEMU and general-optimization landscapes.Source: §2.3.2 · p. 6

2.3.3 Reward Function

  • The reward combines incremental fitness improvement, a diversity incentive, and a stagnation penalty, with a terminal reward when solution accuracy reaches a predefined threshold.A stall counter records consecutive generations without improvement; non-negative coefficients balance the reward terms.Source: §2.3.3 · p. 6

2.3.4 DDPG-based actor–critic policy gradient scheme

  • The DDPG actor–critic agent learns a policy to maximize expected discounted return, with γ balancing immediate and future rewards across an episode of T generations.The actor maps the current state to a continuous action, while the critic estimates its action value.Source: §2.3.4 · p. 7
  • The critic is trained on replay-buffer transitions using temporal-difference error, while the actor follows the deterministic policy gradient to improve its actions.The resulting actor–critic interaction regulates evolutionary search from observed optimization states; reward feedback emphasizes sustained objective reduction while diversity and stall terms preserve exploratory flexibility.Source: §2.3.4 · p. 7

2.4 Dynamic DE–CMA-ES orchestration

DRL-DCO translates learned action variables into coordinated, generation-by-generation regulation of DE and CMA-ES. The mechanism combines DE exploration with covariance-guided sampling, shares directional information across operators, and adaptively controls population composition and search behavior.

2.4 Dynamic DE–CMA-ES orchestration

  • The effective population size is scaled from a base size, elite individuals are retained, and remaining offspring are allocated between DE and CMA-ES by β(t).β(t) continuously sets the relative contribution of DE-based exploration and CMA-ES covariance-guided exploitation.Source: §2.4 · p. 8
  • The DE component generates candidates through mutation and crossover; larger F(t) and CR(t) generally favor broader directional exploration, while smaller values favor localized refinement.Crossover is controlled by CR(t).Source: §2.4 · p. 8
  • CMA-ES samples offspring from an adaptive multivariate Gaussian, with covariance learning shaped by successful search directions and step-size bias adjusting the search radius.Positive Δσ(t) broadens exploration, whereas negative Δσ(t) promotes local refinement.Source: §2.4 · pp. 8–9
  • CMA-ES covariance learning uses top-performing candidates from the mixed offspring population, so DE-discovered directions also inform the Gaussian sampling distribution.This lets covariance adaptation align with promising directions found across both operators.Source: §2.4 · p. 9
  • A stochastic restart probability reinitializes candidate generation occasionally, enabling exploration of previously unvisited regions in highly multimodal landscapes.The restart mechanism is applied to offspring generated by both DE and CMA-ES.Source: §2.4 · p. 9
  • Unlike fixed-rule or staged hybrids, DRL-DCO continuously regulates population size, offspring allocation, and evolutionary update rates to move among exploration-, exploitation-, and mixed regimes.The control policy adapts these decisions to the observed optimization state.Source: §2.4 · p. 9
  • The orchestration preserves DE's global exploration capability while using CMA-ES covariance learning for local refinement, and reduces the need for manual algorithm configuration.The authors state these practical benefits for high-dimensional FEMU and related optimization problems.Source: §2.4 · p. 9

2.5 Implementation details and algorithm workflow

DRL-DCO uses a nested episode-generation workflow for both training and inference, with training adding reward-based actor–critic updates. Its constrained action space coordinates DE, CMA-ES, and population-level controls.

2.5 Implementation details and algorithm workflow

  • Eight actor-generated action variables receive bounded ranges intended to preserve adaptive flexibility while avoiding extreme controls that could destabilize evolution or policy learning.Table 1 lists the prescribed lower and upper bounds.Source: §2.5 · p. 9, Table 1 · p. 11
  • DE mutation scaling F(t) and crossover probability CR(t), plus CMA-ES covariance learning rate LR_cov(t) and step-size factor Δσ(t), are bounded to regulate search behavior.The stated bounds support local refinement and long-range exploration while limiting abrupt distribution distortion, excessive exploration, or premature convergence.Source: §2.5 · pp. 9–10
  • Population scaling λ(t), restart probability p_res(t), and elite-preservation ratio r_eli(t) regulate population size, stochastic reinitialization, and the balance between retaining strong solutions and population renewal.The described aims include limiting computational cost and helping escape local optima without excessive elitism or instability.Source: §2.5 · p. 10
  • The orchestration ratio β(t) generally lies in [0.10, 0.90] to keep both operators active; strongly ill-conditioned, nonconvex problems may use [0.30, 0.70].The narrower interval is described as maintaining a balanced exploration–exploitation tradeoff.Source: §2.5 · pp. 10–11
  • Both phases use an outer episode loop and an inner evolution loop of up to 1000 generations; training additionally calculates rewards, stores replay, and updates the actor–critic.Figure 1 overviews agent–operator interaction, and Table 2 summarizes the procedure.Source: §2.5 · pp. 9–10, Figure 1 · p. 11, Table 2 · p. 11
  • The first five training episodes use independently randomized populations, after which the best initial state seeds inheritance-based policy-guided evolution.This warm-up generates diverse trajectories before population information and the evolving policy cooperate.Source: §2.5 · p. 10

3.1 Testing on benchmark optimization problems

The benchmark evaluation spans five diverse landscapes and compares training-phase DRL-DCO with adaptive, hybrid, and single-operator baselines. DRL-DCO performs competitively across the suite, while its trained inference policy generalizes across benchmark runs without online adaptation.

3.1.1 Benchmark functions and baseline algorithms

  • The suite used Ackley, Griewank, Rastrigin, Rosenbrock, and Schwefel at Nd=20, 30, and 50, with a 40,000-generation budget and convergence defined by ε=1 × 10^-8.Source: §3.1.1 · pp. 12–13

3.1.2 Benchmark analysis of agent-training phase

  • On Rastrigin and Schwefel, DRL-CMA-ES failed to escape suboptimal basins, whereas DE-based methods converged; DRL-DE needed more generations than DRL-DCO on Rastrigin.Source: §3.1.2 · p. 13
  • On Rosenbrock, classical JADE and LSHADE outperformed LSHADE-SPACMA, but DRL-DE stagnated while DRL-CMA-ES reached the optimum slowly.Source: §3.1.2 · p. 14
  • During training, DRL-DCO favored DE while retaining CMA-ES on Rastrigin and Rosenbrock, and both DRL-DCO and DRL-CMA-ES solved Griewank within 1,000 generations.Source: §3.1.2 · p. 14
  • Fixing the orchestration ratio to either operator produced less reliable convergence, with at least one ablation exhausting the budget on Rastrigin, Rosenbrock, or Schwefel.Source: §3.1.2 · p. 14
  • At Nd=50, DRL-DCO's generation cost, function evaluations, and wall-clock time ranked at or above the six-algorithm median across five benchmarks.Source: §3.1.2 · p. 13, Table 4 · p. 16

3.1.3 Benchmark analysis of agent-inference phase

  • Inference required fewer generations than training across all benchmarks and was faster per generation, although its reductions were usually smaller than the 5,000-generation warm-up.Source: §3.1.3 · p. 17
  • Across five functions, DRL-DCO's inference policy generalized across independent runs and was competitive with or superior to the best non-DRL baselines in total generation cost.Source: §3.1.3 · p. 17, Table 5 · p. 17

3.2 Application to FEMU: IASC-ASCE benchmark validation

DRL-DCO is evaluated on the 12-DOF IASC-ASCE benchmark using a high-fidelity reference FEM and a simplified stiffness-identification model. It identifies baseline stiffness accurately and localizes damage across five scenarios, with deviations bounded by the identification model’s simplifications and the difficulty of low-severity damage.

3.2.1 Essential configurations for FEMU

  • The Case 4 benchmark couples lateral and torsional responses across 12 DOFs; identification uses a simplified shear-type FEM with 12 principal stiffness parameters.The high-fidelity analytical FEM supplies reference modal data, while the identification model omits stiffness coupling terms for tractability.Source: §3.2.1 · p. 19, Figure 6 · p. 19

3.2.2 Parameter identification for the baseline (undamaged) condition

  • Baseline training reaches convergence on average in fewer than 300 generations and consistently within the first 1,000-generation warm-up episode.Convergence is sustained across all five warm-up episodes, before meaningful policy training.Source: §3.2.2 · p. 20
  • During inference, the trained actor retains effective search behavior across independently initialized instances without critic evaluation, reward feedback, or policy updates.The effective population size and elite-ratio trajectories are reported as more stable during inference than during training.Source: §3.2.2 · p. 20
  • Mean percentage errors remain below 0.1% across all 12 baseline stiffness parameters despite using only eight available modes.The study reports small standard deviations across independent runs.Source: §3.2.2 · p. 20, Figure 8 · p. 21

3.2.3 Parameter identification for the damaged conditions

  • In Damage Scenarios 1 and 2, the algorithm closely estimates affected stiffness components while unaffected parameters remain consistent with baseline values.These scenarios involve complete brace removal at the first floor and at both the first and third floors.Source: §3.2.3 · p. 22
  • For Damage Scenarios 3 and 4, lateral stiffness estimates are reliable, while torsional components show somewhat larger discrepancies.Scenario 3 reports k(1)_2 as 55.39 MN/m against 55.84 MN/m and k(1)_3 as 202.21 MN/m against 209.12 MN/m.Source: §3.2.3 · p. 22
  • In the low-severity fifth scenario, DRL-DCO identifies affected DOFs and the direction of stiffness reduction, but some numerical deviations remain.The scenario involves a one-third stiffness reduction in a single first-floor brace; the authors attribute deviations to reduced-order identification sensitivity under incomplete modal information.Source: §3.2.3 · pp. 22–23
  • Across all five damage scenarios, DRL-DCO identifies the correct damage locations and estimates stiffness in reasonable-to-close agreement with references, with stable results across runs.The reported discrepancies are attributed to structural simplifications in the identification model rather than the optimizer’s search behavior.Source: §3.2.3 · p. 23, Figure 9 · p. 24

4. Conclusions

The conclusions present DRL-DCO as a continuously learned coordination of DE and CMA-ES, with progression-aware control over search operators and resources. The authors report reusable inference policies and strong performance, while noting that stochastic neural-network training remains a source of uncertainty.

Conclusions

  • Continuous orchestration allocates search effort between DE and CMA-ES on the non-elite population, and ablations find neither single-operator configuration matches the full method across tested problem classes.The authors identify the orchestration ratio as structurally essential.Source: §4 · p. 24
  • The state representation adds six progression-aware descriptors, including fitness-improvement rate, population diversity, and distributional shape statistics.The enriched state is intended to give the actor a temporally coherent view for adjusting DE and CMA-ES hyperparameters.Source: §4 · pp. 24–25
  • Learnable population scaling and elite preservation adjust population size and inherited elite proportion to the observed search state.The method uses smaller populations during exploitation-dominant phases and expands coverage during exploration-demanding phases.Source: §4 · p. 25
  • A learned restart probability uses stall-count and diversity statistics to trigger reinitialization, alongside CMA-ES step-size and covariance-scaling controls.The restart policy is intended to escape deceptive local minima on highly multimodal landscapes.Source: §4 · p. 25
  • After training, the actor generates control actions by forward inference alone; one 20–30-episode training run is reported to generalize across independently initialized instances and tested problem classes.Inference uses no further policy updates or online adaptation.Source: §4 · p. 25
  • The approach inherits stochastic uncertainty from neural-network training, coupling target-problem optimization with policy training.The authors identify physically informed, model-based reinforcement learning as a future direction for reducing training uncertainty and improving interpretability.Source: §4 · p. 25
Loading 2610.11546v1…