PaperlayerView exact revision on arXiv

Source-linked AI summary

Differentiable Evolutionary Reinforcement Learning

Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, Difan Zou

arXiv:2512.13399v2cs.AIcs.CL

TL;DR

Complex reasoning tasks make effective reward design difficult, while automated methods often treat reward functions as black boxes. DERL uses a Meta-Optimizer to compose structured rewards and learns from inner-policy validation performance. The paper reports strong results across reasoning domains, including state-of-the-art performance on agent benchmarks and improved OOD robustness.

  • Problem

    Complex reasoning faces sparse outcome signals and costly annotation for dense reward models, while automated reward search treats reward configurations as black boxes and misses their relationship to policy performance.

    Source: §1 · p. 1

  • Method

    DERL trains an outer Meta-Optimizer with inner-policy validation feedback to compose atomic primitives into structured reward functions for policy learning.

    Source: §1 · p. 2

  • Results

    DERL outperforms human-designed heuristics across robotic, scientific, and mathematical reasoning, with state-of-the-art performance and superior OOD robustness on ALFWorld and ScienceWorld.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported evolution analysis finds that Meta-Rewards encode task logic and converge toward stable reward structures without human intervention.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Crafting effective reward signals remains a central challenge in Reinforcement Learning (RL), especially for complex reasoning tasks. Existing automated reward optimization methods typically rely on derivative-free search heuristics that treat the reward function as a black box, failing to exploit the causal dynamics between reward structure modifications and policy performance. We introduce Differentiable Evolutionary Reinforcement Learning (DERL), a bi-level framework for the autonomous discovery of optimal reward structures. DERL employs a Meta-Optimizer that evolves a reward function through the composition of structured atomic primitives to guide an inner-loop policy. Unlike prior black-box methods, DERL introduces differentiability into the meta-optimization process by updating the Meta-Optimizer using policy gradients derived from inner-loop validation performance. This allows for the progressive learning of a "meta-gradient" for task success, providing the system with dense, actionable feedback. We validate DERL across diverse reasoning domains: embodied agent (ALFWorld), scientific simulation (ScienceWorld), and mathematical reasoning (GSM8K, MATH). Results show that DERL achieves state-of-the-art performance on agent benchmarks, substantially outperforming non-differentiable baselines-especially in out-of-distribution generalization. Trajectory analyses confirm that DERL captures the intrinsic causal structure of tasks, enabling fully autonomous, self-improving agent alignment.

1 Introduction

Sparse outcome rewards and costly annotation complicate reward design for complex reasoning, while automated black-box search misses how reward changes relate to policy performance. DERL uses validation-guided policy gradients to evolve structured rewards, and is reported to perform strongly across reasoning domains, including state-of-the-art results on ALFWorld and ScienceWorld.

1 Introduction

  • Automated reward-evolution methods use genetic mutations or prompted LLMs, but treat configurations as black boxes and do not systematically capture how reward changes relate to policy performance.Sparse outcome rewards and the annotation costs of dense reward models are additional challenges in complex, long-horizon tasks.Source: §1 · p. 1
  • DERL frames reward generation as a higher-level RL problem, using inner-policy validation performance to train a Meta-Optimizer through policy gradients.This provides a learned estimate of the meta-gradient between reward-structure changes and task success.Source: §1 · p. 2
  • The Meta-Optimizer composes modular, executable atomic primitives into reward functions, constraining the search to structured reward compositions rather than arbitrary code.The design pairs this parameterization with inner-loop validation performance as the evolution signal, reducing reliance on manual annotation.Source: §1 · p. 2
  • DERL achieves state-of-the-art performance on ALFWorld and ScienceWorld and shows superior OOD robustness across evaluations spanning robotic, scientific, and mathematical reasoning.The analysis reports that its evolving Meta-Rewards encode task logic without human intervention.Source: §1 · p. 2
  • DERL is evaluated on ALFWorld, ScienceWorld, GSM8K, and MATH, with reported generalization across tasks and stronger OOD robustness on the agent benchmarks.The reported trajectory analysis says the Meta-Optimizer's rewards evolve to encode task logic.Source: §1 · p. 2

2 Differentiable Evolutionary Reinforcement Learning (DERL)

DERL casts reward design as a bi-level optimization problem: an outer Meta-Optimizer generates structured rewards, and inner policies learn under those rewards. Inner-policy validation performance trains the outer optimizer, while the instantiation composes atomic signals and uses GRPO for policy updates.

2.1 Bi-Level Evolutionary Training

  • The outer Meta-Optimizer generates reward configurations from a task instruction, and inner policies optimize expected parameterized reward on the training data.Each trained inner policy is evaluated on validation performance, which serves as the outer-loop feedback signal.Source: §2.1 · p. 3
  • DERL replaces discrete evolutionary reward search with continuous, gradient-guided optimization of a Meta-Optimizer that generates rewards for inner-loop policies.The framework is described as learning a search direction rather than relying on stochastic mutations.Source: §2 · p. 2, §2.1 · p. 3

2.2 Instantiation of DERL

  • The reward parameterization composes atomic signals—such as correctness, formatting, or process heuristics—using operators and weights specified by the configuration.Separating signal definitions from their use lets the Meta-Optimizer search structured compositions rather than arbitrary functions.Source: §2.2 · p. 4
  • GRPO updates the inner policy using group-wise advantages derived from outputs' parameterized rewards.The objective samples groups of outputs for each training question and includes clipping and a KL-divergence penalty.Source: §2.2 · p. 4
  • The population-based variant initializes later inner loops from the best-performing prior checkpoint, while standard DERL resets to the base model each iteration.Invalid configurations receive zero feedback after execution errors, and the Meta-Optimizer is cold-started with supervised fine-tuning and constrained decoding.Source: §2.2 · pp. 4–5
  • For each outer-loop iteration, reward configurations train corresponding policies on an inner-training split, and validation pass@1 scores provide their feedback.The validation scores are used to update the Meta-Optimizer, while final evaluation is conducted on held-out test data.Source: §2.2 · pp. 4–5
  • DERL uses non-differentiable validation signals to form estimated meta-gradients that feed back to the Meta-Optimizer in a closed-loop process.The paper describes this feedback as supporting dense, actionable reward signals from sparse outcomes.Source: §2.2 · p. 5

3 Experiments

DERL is evaluated across robotic, scientific-simulation, and mathematical reasoning tasks against reward baselines, with tests of generalization under distribution shifts. It achieves leading agent-benchmark results and outperforms the compared reward functions on mathematical reasoning.

3 Experiments

  • The study asks whether DERL discovers better rewards than heuristics and generalizes to OOD scenarios; it reports results under the stated benchmark protocols and training settings.Robotic and scientific results include statistics across four random seeds; the outer-loop tuning validation split is excluded from final test evaluation.Source: §3 · p. 5, §3.2 · p. 5

3.1 Experiment Setups

  • The experiments cover ALFWorld household tasks, ScienceWorld elementary-level science tasks, and GSM8K and MATH; ALFWorld and ScienceWorld use L0, L1, and L2 evaluation levels.ALFWorld L2 holds out two task types after training on four; mathematical training uses either MATH alone or combined MATH and GSM8K data.Source: §3.1 · p. 5

3.2 Implementation Details

  • DERL is compared with outcome, averaged-primitive, format-augmented, and finer-grained process-reward baselines, with a shared final-policy training budget for the agent tasks.For agent tasks, the atomic reward signals include binary outcome and average step rewards from the first, middle, and final trajectory thirds; the mathematical primitives include outcome, format, step indicators, and soft outcome.Source: §3.1 · p. 5, §3.2 · p. 6

3.3 Results

  • DERL achieves state-of-the-art success rates across all difficulty levels on ALFWorld and ScienceWorld, outperforming all baselines.Table 1a covers three generalization levels; DERL-pop. reaches 91.8% on ALFWorld (L0) and 98.2% on ScienceWorld (L0).Source: §3.3 · p. 6, Table 1 · p. 7
  • DERL attains 65.0% success on ALFWorld and 30.1% on ScienceWorld in L2 OOD settings, more than doubling the outcome-reward baseline.GRPO + Avg. improves L0 performance by approximately 10% over GRPO + Out. but yields only a 0.8% improvement in OOD settings.Source: §3.3 · p. 7
  • On MATH, DERL reaches 60.2% versus 58.8% for the outcome reward baseline, while naive auxiliary rewards can lower performance from 58.8% to 55.8%.DERL-pop. further improves performance; the passage reports that DERL outperforms the other baseline reward functions on GSM8K and MATH.Source: §3.3 · p. 7

4 Analysis

The analysis tracks Meta-Optimizer performance and reward-structure composition during training. Validation and testing performance rise together, while stable reward structures become more prevalent on ALFWorld.

4 Analysis

  • The analysis investigates whether Meta-Optimizer improvement reflects a progressive strategy rather than random search, and whether gradient feedback yields stable reward compositions.Figure 3 omits ScienceWorld because the Meta-Optimizer converges faster there.Source: §4.1 · p. 7, §4.2 · p. 8

4.1 Optimization Dynamics

  • Validation and testing performance rise monotonically with outer-loop training, supporting convergence without overfitting to outer-loop instances.Mathematical-reasoning trends are more stable, while agent tasks involve more exploration; the analysis uses average inner-policy validation accuracy over Meta-Rewards.Source: §4.1 · pp. 7–8, Figure 3 · p. 8

4.2 Evolution Dynamics of Reward Structures

  • Stable structures use linear combinations or normalization, whereas unbounded products can make one near-zero signal collapse the reward and destabilize gradients.Invalid structures assign negative coefficients to positive signals, thereby penalizing desirable behavior.Source: §4.2 · p. 8
  • On ALFWorld, stable structures increase and become dominant as unstable structures decline during training.Unstable structures appear frequently during early exploration; the authors interpret the later shift as an evolutionary filter for bounded, numerically stable rewards.Source: Figure 4 · p. 8, §4.2 · p. 9

5 Related Work

DERL is situated against automated configuration search, meta-learning, and reward-modeling work. The related-work discussion distinguishes its validation-guided symbolic reward search from black-box configuration evolution and full inner-loop differentiation.

  • Prior agent-configuration evolution uses discrete black-box search or prompted agents, often relying on sparse final fitness scores and ignoring training dynamics.DERL instead uses a parameterized Meta-Optimizer and validation performance as feedback for gradient-guided configuration search.Source: §5 · p. 9
  • Unlike meta-gradient RL, DERL tunes symbolic reward configurations without backpropagating through the full inner-loop optimizer or directly optimizing continuous reward parameters.The inner policies are independently optimized, and the approach does not require preference-labeled reward-model training.Source: §5 · p. 9
  • Reward modeling faces a trade-off between sparse outcome rewards and costly human-annotated dense rewards, while manually coordinated heuristic rewards may degrade performance when combined naively.DERL generates reward functions without external human preference data.Source: §5 · p. 9

6 Conclusions and Limitations

DERL automates reward discovery by composing atomic primitives and training a Meta-Optimizer from inner-loop validation performance. The conclusion reports strong cross-domain results and identifies the predefined primitive set as a limitation, while the appendix describes the reward search space and example signals.

6 Conclusions and Limitations

  • DERL automates reward discovery through atomic-primitive composition and Meta-Optimizer training on inner-loop validation performance.The conclusion characterizes its search as gradient-guided and reports outperformance of standard RL baselines and human-designed heuristics across robotic, scientific, and mathematical reasoning tasks.Source: §6 · p. 9
  • A major limitation is that DERL relies on a predefined set of atomic primitives, although the authors report insensitivity to the primitive vocabulary.They propose adapting the framework to downstream tasks by defining a set of primitives.Source: §6 · p. 9, §A.2 · p. 14

A.1 The merits of our reward parameterization design.

  • The reward parameterization is described as expressive and continuous, with extensibility from separating atomic-signal definitions from their use in reward composition.The authors say the Meta-Optimizer can consider problem aspects structurally and that evolution filters and weights potentially useful or detrimental signals.Source: §A.1 · p. 14

A.2 Concrete Examples

  • For robotic agent tasks, primitives comprise binary task-success outcome and average heuristic step scores from the first, second, and final trajectory thirds, with scores based on textual-action alignment to task meta-tags.Source: §A.2 · p. 14

B Experimental Protocol and Detailed Training Setups

The protocol separates reward tuning from final evaluation, defines task-generalization levels, and specifies training settings across domains. The preliminary experiment tests whether a compact primitive-composition graph can yield useful rewards using fixed trajectories, without the main framework’s closed-loop validation feedback.

B Experimental Protocol and Detailed Training Setups

  • The original training set is split 8:2 into inner-training and validation data; validation guides outer-loop reward tuning, while held-out test data is reserved for final evaluation.After reward selection, the tuning policies are discarded and a fresh policy is trained on the full original training set.Source: B Experimental Protocol and Detailed… · p. 15
  • DERL uses the same original training data and no more final-policy steps than baselines, while its additional outer-loop tuning cost is reported separately.Default settings include a Qwen-2.5-0.5B-Instruct Meta-Optimizer, GRPO, and constrained decoding; inner-loop settings vary by domain.Source: B Experimental Protocol and Detailed… · p. 15
  • ALFWorld generalization levels combine training-distribution coverage with test-instance novelty: L0 is in-distribution/seen, L1 in-distribution/unseen, and L2 out-of-distribution/unseen.Seen tests alter environmental variables for familiar concepts; unseen tests introduce novel object variants and instructions.Source: B Experimental Protocol and Detailed… · pp. 15–16

C Preliminary Experiments – Feasibility and Generality

  • The preliminary study tests a 12-weight computation graph over atomic reward primitives as a proof of concept, not as a substitute for DERL’s full outer-loop.The graph reward is fitted offline on fixed Qwen2.5-3B-Instruct trajectories, then frozen for downstream policy training.Source: §C.1 · pp. 16–17, C Preliminary Experiments –… · p. 16

C.1 Experimental Setup

  • The offline graph weights are optimized either by supervised regression toward binary outcome rewards or by an RL-style objective on the same fixed trajectories.The RL-style objective assigns 1.0 when the Meta-Reward matches the outcome reward and 0.0 otherwise; neither protocol uses closed-loop validation feedback.Source: §C.1 · p. 17

C.2 Results and Analysis

The preliminary results support the usefulness of primitive-composed dense rewards, while distinguishing this evidence from the full DERL optimization. The regression Meta-Reward improves on the sparse outcome baseline on MATH, but a fixed supervised target does not replace validation-driven outer-loop tuning.

C.2 Results and Analysis

  • The results support that primitive composition contains useful dense reward signals, but supervised fitting to a predefined binary outcome does not replace DERL’s outer loop.The main framework instead uses validation performance from independently optimized inner policies to adapt reward generation to downstream learning.Source: §C.2 · p. 17
  • 62.9% vs. 58.8%: Meta-Reward (Regression) improves over the Outcome baseline on MATH.Table 2 compares the outcome reward, average reward, and Meta-Reward variants on mathematical reasoning benchmarks.Source: §C.2 · p. 17
  • The fitted implicit dense reward guides the policy more effectively toward correct reasoning paths than the raw outcome signal alone.The authors characterize this as a discovered reward rather than a human-annotated process reward.Source: §C.2 · pp. 17–18

D Training Dynamics of DERL-pop.

DERL-pop’s training trajectory shows an initial exploration phase followed by adaptation and stronger late-stage gains, while outer-loop examples show reward quality improving over iterations. Additional comparisons report that DERL outperforms RLAIF and compute-controlled random search, with progressive optimization not observed in those baselines.

D Training Dynamics of DERL-pop.

  • After performing on par through steps 0–33, DERL-pop pulls ahead during steps 34–66 and accelerates gains beyond 66 steps.Each later inner-loop phase starts from the previous generation’s highest-performing checkpoint; the trajectory is described as bypassing static-reward plateaus.Source: D Training Dynamics of DERL-pop · p. 18

E Examples of Outer-loop Evolution

F Additional Experiments

  • On GSM8K, additional experiments compare DERL with RLAIF and compute-controlled random search, alongside primitive-vocabulary and rollout ablations.The experiments use Qwen2.5-1.5B; the reported comparisons include a GPT-4o-based RLAIF baseline.Source: F Additional Experiments · p. 19

F.1 Comparison with RLAIF Method

  • RLAIF finds rewards better than naive baselines but does not surpass DERL.Table 4 presents the comparison with the RLAIF method.Source: Step DERL RLAIF · p. 20, Table 4

Step DERL RLAIF

  • DERL improves progressively across outer-loop steps, whereas RLAIF does not show progressive optimization across its generated rollouts.The comparison reports average test-set performance across multiple rollouts at each training step.Source: Step DERL RLAIF · p. 20, §F.1 · p. 20

F.2 Comparison with Compute-controlled Random Search Method

  • Compute-controlled random search can find relatively good rewards but cannot surpass DERL or optimize them stepwise.The random-search comparison holds computation, reward space, and rollout count equal to DERL.Source: §F.2 · p. 20

F.3 Ablation and Perturbation on the Primitive Vocabulary

Primitive-vocabulary tests indicate that DERL can learn useful rewards despite primitive removal or a flipped signal. Rollout-count tests report similar performance for 4, 6, and 8 rollouts, with fewer rollouts reducing computational cost.

F.3 Ablation and Perturbation on the Primitive Vocabulary

  • Removing any of the three added primitives does not prevent learning a beneficial reward, and with reversed g2 the model learns a negative coefficient.The authors report that DERL remains effective even with arbitrary primitive removal or a toxic primitive signal.Source: §F.3 · p. 21, Table 6 · p. 21
  • DERL’s reported performance is not sensitive to its human-designed primitive vocabulary, including removal or reversal of a primitive.The tested additions are three primitives beyond vanilla GRPO, with g2 flipped to create a signal where higher reward corresponds to a smaller value.Source: §F.3 · pp. 20–21

F.4 Sensitivity to Outer-loop Rollouts

  • Rollout counts of 4, 6, and 8 yield similar performance, and using fewer rollouts can reduce computational cost.The ALFWorld results report mean ± standard deviation and t-distribution confidence intervals across four random seeds.Source: §F.4 · p. 21, Table 7 · p. 22, Table 8 · p. 22

G Detailed Statistical Results in Table 1a

The reported robotic-agent and scientific-simulation results account for task randomness using four seeds and show DERL and DERL-pop statistically outperforming the baselines.

G Detailed Statistical Results in Table 1a

H Computational Cost Analysis

DERL’s cost analysis identifies inner-loop policy training as its main computational bottleneck and describes parallel execution, resource estimates, and possible efficiency improvements. The framework also propagates validation-performance gradients to its Meta-Optimizer, while the paper notes computational and interpretability considerations.

H Computational Cost Analysis

H.1 Computational Breakdown

  • DERL’s inner-loop policy evolution is the primary computational bottleneck; eight rollouts run in parallel, and iteration time depends on the slowest successful trial plus overhead.Mathematical tasks use an approximate 1.3× standard binary-outcome training FLOP cap; validation is parallelized and the lightweight 0.5B-parameter outer-loop update takes minutes.Source: §H.1 · pp. 22–23

H.2 Resource Estimation

  • DERL’s estimated cost scales with outer-loop iterations, while DERL-pop selects from one population generation and uses substantially less wall-clock time.The authors report using high-memory data-center accelerators for parallel inner-loop training.Source: §H.2 · pp. 23–24

H.3 Efficiency Improvements and Future Work

  • The current implementation uses eight rollouts and full inner-loop training, while preliminary experiments suggest 12-parameter reward parameterizations can be competitive.The authors propose lightweight RL algorithms such as REINFORCE++ to reduce inner-loop computational burden.Source: §H.3 · p. 24

I.1 Gradient Propagation Flow

I Gradient Propagation on DERL

  • The optimizer updates its parameters by maximizing expected future validation performance, with the gradient signal flowing from evaluation back to the optimizer.Source: §I.1 · pp. 24–25, Figure 7 · p. 24

I.2 Meta-Gradient Propagation Compared with Previous Evolution

  • DERL propagates validation-performance gradients to the Meta-Optimizer, enabling it to update its parameters and iteratively improve reward structures.The authors contrast this learnable optimizer with prior approaches that treat the optimizer as fixed, losing the meta-gradient.Source: §I.1 · p. 25, §I.2 · p. 25

J Broader Impacts

  • DERL is presented as reducing reliance on costly human annotation and heuristic reward engineering, but the authors call for interpretability analysis and note higher resource use than single-level RL.They identify unintended behavior in long-horizon tasks and energy consumption as concerns.Source: J Broader Impacts · p. 25

Related papers

Loading 2512.13399v2…