Source-linked AI summary
Interpretable Policies for Reinforcement Learning by Genetic Programming
Daniel Hein, Steffen Udluft, Thomas A. Runkler
TL;DR
Interpretable RL policies are sought for industrial settings where understandable controllers may be more acceptable to domain experts, but data-driven methods have produced relatively few such policies. The paper introduces GPRL, combining model-based batch RL with genetic programming to learn policy equations from existing trajectories. Across mountain car, cart-pole balancing, and industrial benchmarks, GPRL outperforms symbolic regression in producing adequately performing interpretable policies.
Problem
Data-driven RL has produced relatively few human-interpretable policies, despite the importance of understandable controllers for academic and industrial applications.
Method
GPRL trains genetic-programming policies on a world model learned from existing system trajectories, searching compact algebraic or Boolean policy representations.
Results
Across mountain car, cart-pole balancing, and industrial benchmarks, GPRL outperformed the symbolic-regression strategy that imitated a non-interpretable neural-network policy.
Takeaways & Limitations
GPRL can produce well-performing interpretable RL policies from pre-existing default trajectory data, supporting more understandable policy inspection and safety assessment.
Takeaways & Limitations
The related-work approach requires creating all policy candidates initially, causing computational effort to grow combinatorially for more complex control problems.
Abstract
from arXiv · showhide
The search for interpretable reinforcement learning policies is of high academic and industrial interest. Especially for industrial systems, domain experts are more likely to deploy autonomously learned controllers if they are understandable and convenient to evaluate. Basic algebraic equations are supposed to meet these requirements, as long as they are restricted to an adequate complexity. Here we introduce the genetic programming for reinforcement learning (GPRL) approach based on model-based batch reinforcement learning and genetic programming, which autonomously learns policy equations from pre-existing default state-action trajectory samples. GPRL is compared to a straight-forward method which utilizes genetic programming for symbolic regression, yielding policies imitating an existing well-performing, but non-interpretable policy. Experiments on three reinforcement learning benchmarks, i.e., mountain car, cart-pole balancing, and industrial benchmark, demonstrate the superiority of our GPRL approach compared to the symbolic regression method. GPRL is capable of producing well-performing interpretable reinforcement learning policies from pre-existing default trajectory data.
1. Introduction
The paper introduces GPRL, which learns interpretable RL policies from existing trajectory data by combining genetic programming with model-based batch reinforcement learning. Across three benchmarks, GPRL is compared with symbolic regression that imitates a non-interpretable neural-network policy.
- Motivation and contribution: GPRL learns interpretable RL policies from previously recorded state transitions using genetic programming.The policies can be represented as basic algebraic equations or Boolean logic terms.
- Motivation and contribution: Industrial high-level controllers are often based on default strategies and expert knowledge, while first-principles models may be infeasible to construct.The paper motivates learning controllers from available system data instead.
- Motivation and contribution: Batch RL learns policies from existing data without requiring online exploration that could be unsafe for the system.Historic trajectories contain observations, actions, rewards, and next observations.
- GPRL approach: GPRL evaluates algebraic policy populations on a world model using Monte Carlo returns and iteratively maximizes their fitness through genetic programming.The world model is trained from real system data before policy training.
- Evaluation: Experiments on mountain car, cart-pole balancing, and an industrial benchmark compare GPRL with symbolic regression of an existing non-interpretable neural-network policy.The reported results demonstrate that GPRL solves the benchmarks and produces interpretable policies.
2. Related Work
Prior work used genetic programming for controllers, online reinforcement learning, and Monte Carlo policy search, but these approaches have important data or computational constraints. The paper positions GPRL as combining genetic programming with model-based batch RL.
- Genetic programming and reinforcement learning: Genetic programming has been used to create rule-based policies, controllers, game-playing systems, and robotic behaviors.Earlier work includes GP controllers that outperformed conventional PID controllers on an industrially representative plant set.
- Genetic programming and reinforcement learning: Online RL approaches learn through environment interaction, whereas batch RL extracts policies from pre-generated data.The paper notes that online learning may be unsuitable when exploration is prohibited for safety reasons.
- Genetic programming and reinforcement learning: GP and related genetic network programming methods have been combined with online RL for maze-search and robotic adaptation tasks.These studies primarily address online learning settings.
- Policy search: Monte Carlo simulations have been used to evaluate GP policy fitness, but exact simulations are generally unavailable in industrial applications.The paper instead motivates evaluation on an environment approximation.
- Policy search: Creating and evaluating every policy candidate at once causes computational effort to grow combinatorially as solution complexity increases.This is identified as a fundamental drawback of the former approach.
3. Model-based Reinforcement Learning
Model-based RL replaces costly real-system policy evaluation with an approximation of system dynamics learned from prior data. Policy fitness is computed from discounted future rewards and averaged across starting states.
- RL formulation: An RL policy maps observed system states to actions, and the desired policy maximizes expected accumulated rewards.States transition according to system dynamics and produce rewards based on state, action, and next state.
- Policy optimization: GPRL searches over equations built from predefined function blocks subject to a maximum complexity, with each equation producing an action for a state.Policy performance is measured by accumulated future rewards.
- Policy optimization: The finite-horizon return sums future rewards generated by recursively applying the policy and system transition function.The next state satisfies s_t+k+1 = g(s_t+k, a_t+k).
- Policy optimization: The overall policy performance averages starting-state returns using state probabilities and discounts later rewards with γ.The performance function serves as the fitness function for optimization.
- Model-based evaluation: Model-based RL approximates the transition function with a model learned from previously gathered data, producing an approximate fitness function.The study uses neural-network models, while the method can extend to other model types.
4. Genetic Programming
The genetic programming framework represents interpretable policies as algebraic function trees and evolves them through selection, crossover, reproduction, mutation, cancellation, and random generation. Complexity is explicitly measured so policy equations can remain suitable for interpretation.
- Policy representation: GP encodes policies as function trees containing algebraic functions, floating-point constants, and state variables.These trees can be stored efficiently in memory arrays.
- Evolutionary operators: Selection and reproduction evolve populations by crossing compatible function-tree subtrees selected through tournament selection.Two offspring are created by interchanging subtrees beneath compatible cutting points.
- Complexity control: Automatic cancellation replaces reducible subtrees with computed terminals, reducing equation complexity and freeing space for other subtrees.For example, a subtree adding two float terminals can be replaced by their summed terminal.
- Initialization and diversity: Randomized subtree depths and positions promote population diversity and reduce the risk of early convergence to small chromosomes.The grow procedure randomizes tree depth and subtree placement during individual construction.
- Evolutionary cycle: The genetic algorithm repeatedly evaluates fitness, applies evolutionary operators, and returns the best individual found for each complexity level.Float terminals in selected high-performing individuals are also mutated using Gaussian perturbations.
- Complexity evaluation: Complexity is measured by weighted node counting, allowing domain experts to assign different importance to functions, variables, and terminals.The experiments use large populations and empirically selected generation counts and ratios.
5. Genetic Programming Reinforcement Learning
GPRL learns algebraic policies from state-transition data by training world models and evaluating policy trajectories on those models. It is compared with symbolic regression that imitates a neural-network policy, a strategy reported to become unstable on more complex tasks.
- Data and world models: GPRL begins with transition data D containing tuples (s, a, s′, r) collected from real-system dynamics.The data can generally be generated by any sufficiently exploratory policy before training.
- Data and world models: World models predict state transitions from (s, a), with separate models optionally learning state differences and the reward modeled from observed transitions.The resulting state combines the current state with predicted changes.
- Policy evaluation: GPRL evaluates basic algebraic policy equations by generating trajectories from fixed benchmark states on the learned world model.The resulting model-based trajectory performance supplies each individual’s fitness.
- Symbolic-regression comparison: The comparison method uses GP symbolic regression to imitate actions produced by a well-performing neural-network policy on states from its trajectories.The neural-network policy and GPRL are trained with model-based reinforcement learning on the same world models.
- Comparison outcome: Symbolic-regression imitation is successful mainly on small, simple problems and produces unstable, unsatisfactory results on more complex tasks.This finding motivates evaluating GPRL directly through model-based policy fitness rather than only matching an existing policy’s actions.
- Model and policy roles: The neural-network policy generally has higher training fitness because it has substantially more degrees of freedom than GPRL’s basic algebraic equations.The experiments use neural networks as world models, distinct from the neural-network policy.
6. Experiments
The experiments evaluate GPRL and related policies on mountain car, cart-pole balancing, and an industrial benchmark using learned world models and real dynamics. The benchmarks span continuous control tasks and an industrial setting with partial observability, stochastic and delayed effects, and multi-criteria rewards.
- Mountain Car: Mountain car requires driving an underpowered car uphill by first building momentum in the opposite direction, with success at position ρ ≥0.6.The policy outputs actions in [−1, 1].
- Cart-pole Balancing: Cart-pole balancing applies force to keep the pole upright while controlling cart position, using pole angle, angular velocity, cart position, and cart velocity as state variables.The task prevents the pole from falling over while moving the cart on a one-dimensional track.
- Cart-pole Balancing: Cart-pole episodes fail when pole angle or cart position leaves restricted intervals, while actions range from −10 N to +10 N at 0.025-second intervals.The restricted intervals are [−0.7, 0.7] for angle and [−2.4, 2.4] for cart position.
- Industrial Benchmark: The industrial benchmark represents challenging industrial conditions with high-dimensional, partially observable continuous states, three continuous control inputs, stochastic and delayed effects, and opposing reward dependencies.Its task is multi-criterial and includes state-dependent observation noise.
- Industrial Benchmark: Industrial actions are three-dimensional vectors in [−1, 1]^3 that propose changes to velocity, gain, and shift, each constrained to [0, 100].The scaling factors are dv = 1, dg = 10, and dh = 5.75.
7. Results
Across mountain car, cart-pole balancing, and industrial benchmark experiments, GPRL produced interpretable policies and generally outperformed symbolic-regression policies when evaluated on system dynamics. Results also show that compact policies can match or exceed non-interpretable neural-network baselines while exposing recurring control concepts.
- Mountain Car: Mountain-car GPRL policies achieved a median model penalty of 41.8 for complexities ≥5, while complexity-1 policies still reached the hill.Lower-penalty policies reached the hill in fewer time steps.
- Comparison: Symbolic regression achieved a median regression error of 0.028 when imitating the neural-network policy, but imitation quality did not ensure strong control performance.The regression error suggested good imitation on the sampled policy outputs.
- Mountain Car: On real mountain-car dynamics, GPRL produced the best interpretable policies for all complexities, although symbolic-regression performance was similar.The comparison used Pareto-front policies tested from different start states.
- Cart-pole Balancing: Cart-pole GPRL individuals with complexity ≥13 achieved a median penalty of 27.5 or below, indicating excellent policy suitability.Individuals below complexity 5 performed significantly worse than the neural-network baseline.
- Cart-pole Balancing: Cart-pole Pareto-front solutions repeatedly agreed on relevant state variables and factor ratios, helping experts compare interpretable policies by complexity and performance.The Pareto presentation supports identifying common policy concepts across independent runs.
- Cart-pole Balancing: On true cart-pole dynamics, GPRL had lower median penalties and lower variance than symbolic regression at almost every complexity.The regression median exceeded the neural-network result from complexity 11 onward, while GPRL was generally superior.
- Industrial Benchmark: In the industrial benchmark, one GPRL policy slightly outperformed the neural-network policy despite the latter's greater degrees of freedom.The industrial policies shared recurring lagged-state concepts, including h lags for ∆h and recent v lags for ∆v.
8. Conclusion
The paper concludes that GPRL learns interpretable control policies from existing system trajectories through model-based batch reinforcement learning. Across three benchmarks, it often generalized better than neural-network and symbolic-regression alternatives, supporting its potential relevance to industrial control.
- Contribution: GPRL learns interpretable policies from existing default system trajectories using model-based batch reinforcement learning.Policies can be represented as compact algebraic equations or Boolean logic terms.
- Implications: Interpretable policies may help domain experts assess safety because both the policy and its generalization to state-space regions can be understood.The industrial benchmark particularly indicated potential interest for settings where system data and interpretable algebraic policies are available.
- Evaluation: The complete procedure trains a model, evolves GP policies, and selects a solution from a Pareto front across mountain car, cart-pole balancing, and industrial benchmark.The evaluation compared GPRL with a non-interpretable neural-network policy and symbolic regression.
- Findings: On real system dynamics, even low-complexity interpretable policies could outperform the non-interpretable approach in many cases, suggesting better generalization to new states.Performance on the approximation model could instead be slightly worse than the neural-network result.
- Comparison: Symbolic regression was significantly less suitable than GPRL for producing interpretable policies with adequate performance across the experiments.The method fit an existing non-interpretable neural-network policy rather than directly optimizing control performance.