Source-linked AI summary
PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making
Fangkai Yang, Daoming Lyu, Bo Liu, Steven Gustafson
TL;DR
Reinforcement learning requires extensive interaction, while symbolic planning can be brittle under uncertainty and change. PEORL integrates symbolic planning with hierarchical R-learning so plans guide learning and experience improves planning, yielding rapid policy search and more robust symbolic plans in benchmark domains.
Problem
Reinforcement learning may require an unfeasibly large amount of experience, while manually crafted symbolic knowledge may not be robust to domain uncertainties and changes.
Method
PEORL uses BC and CLINGCON to generate symbolic plans, maps them to stochastic options for R-learning, and feeds learned gain rewards back into planning.
Results
PEORL outperforms RL and HRL through significantly larger cumulative rewards and outperforms planning by discovering extra reward and reducing execution failures.
Takeaways & Limitations
Symbolic planning and hierarchical reinforcement learning can simultaneously improve policy search and symbolic-planning robustness in the evaluated benchmark domains.
Takeaways & Limitations
The convergence properties of option-based hierarchical R-learning remain unknown, so the paper leaves its theoretical study for future work.
Abstract
from arXiv · showhide
Reinforcement learning and symbolic planning have both been used to build intelligent autonomous agents. Reinforcement learning relies on learning from interactions with real world, which often requires an unfeasibly large amount of experience. Symbolic planning relies on manually crafted symbolic knowledge, which may not be robust to domain uncertainties and changes. In this paper we present a unified framework {\em PEORL} that integrates symbolic planning with hierarchical reinforcement learning (HRL) to cope with decision-making in a dynamic environment with uncertainties. Symbolic plans are used to guide the agent's task execution and learning, and the learned experience is fed back to symbolic knowledge to improve planning. This method leads to rapid policy search and robust symbolic plans in complex domains. The framework is tested on benchmark domains of HRL.
1 Introduction
PEORL integrates symbolic planning with hierarchical reinforcement learning to combine planning guidance with experience-based adaptation. The framework uses learned experience to improve symbolic plans and reports stronger cumulative rewards and robustness than separate approaches.
- 1 Introduction: The paper motivates integration because RL can require costly trial-and-error, whereas symbolic plans may be insufficiently robust to uncertainty, change, and learned reward structure.
- 1 Introduction: PEORL integrates symbolic planning with hierarchical R-learning through a Planning–Execution–Observation–Reinforcement-Learning framework.Symbolic plans generate options that guide reinforcement learning, while learned gain rewards improve subsequent symbolic plans.
- 1 Introduction: The framework uses symbolic planning for option discovery in hierarchical reinforcement learning, addressing a capability not covered by earlier symbolic-plan approaches.Earlier approaches treated symbolic actions as corresponding directly to primitive actions and did not dynamically discover new plans and options.
- 1 Introduction: PEORL outperforms RL and HRL agents by returning policies with significantly larger cumulative reward.
- 1 Introduction: PEORL improves planning robustness by using R-learning to discover a rewarding state and reduce execution failures relative to a planning agent.
2 Preliminaries
The preliminaries define action-language representations for symbolic transition systems and explain how answer-set solving produces plans. They also introduce MDPs, hierarchical options, and average-reward R-learning as the components combined by PEORL.
- 2 Preliminaries: Action language BC represents world properties and actions through fluent and action constants, causal laws, and executable or nonexecutable transitions.Dynamic laws specify how executing an action changes fluent values, while inertia captures unchanged values by default.
- 2 Preliminaries: A BC action description defines a transition system in which states are fluent sets and plans are transition paths from an initial state to a goal.Answer sets of the translated program encode states and transition paths that solve the planning problem.
- 2 Preliminaries: An MDP models states, actions, transition probabilities, rewards, and discounting, while RL learns a policy by observing transitions and rewards.
- 2 Preliminaries: R-learning is a model-free value-iteration algorithm for finding policies under average-reward criteria.The paper adopts this average-reward formulation for its planning and learning framework.
- 2 Preliminaries: Hierarchical reinforcement learning introduces temporally extended options above primitive actions to support decision-making across a series of tasks.
3 PEORL Framework
PEORL formalizes a symbolic planning problem, an MDP, and a mapping from symbolic transitions to options. Symbolic plans are executed through sequential options, illustrated in a Grid World where learned options realize actions under uncertain execution.
- 3.1 Symbolic Planning Problem: PEORL combines symbolic planning components with MDP components and a symbolic transition–option mapping in a unified theory.The theory includes initial and goal states, an action description, states, actions, rewards, discounting, and the mapping F_A.
- 3.1 Symbolic Planning Problem: The symbolic planning problem uses BC action descriptions, initial and goal states, and linear plan-quality constraints, while reward is learned rather than assumed as prior planning knowledge.Plans end in the goal state while satisfying the constraint, and PEORL later uses environmental interaction to generate an optimal plan.
- 3.1 Symbolic Planning Problem: PEORL encodes state-action gain rewards in causal laws, initializes unspecified gains to INF, and accumulates them into plan quality.The quality of a plan is represented through the fluent quality and gain-reward facts associated with symbolic transitions.
- 3.2 From Symbolic Transitions to Options: Each symbolic transition maps to an option with an initiation condition and termination condition, and a symbolic plan executes its mapped options sequentially.The mapping is constrained so an option is available at the transition’s source state and terminates at its target state.
- 3.3 Example: Grid World: In the Grid World example, movement is directly realized, while pushing and activating the door use force-specific options to handle multiple execution strategies.The symbolic plan must navigate to the door, activate it, push it open, and then enter the target cell.
4 PEORL Learning
PEORL learning repeatedly obtains symbolic plans, executes their corresponding options, updates hierarchical R-learning values, and feeds learned gain rewards back into planning. The algorithm is empirically evaluated because convergence to an optimum remains theoretically unresolved for finite option sets.
- 4.1 Hierarchical R-learning: PEORL updates R and gain-reward values while each option executes, then performs an additional update when the option terminates.The option-level reward is the cumulative reward accrued during execution of the option mapped from the symbolic action.
- 4.2 PEORL Learning Algorithm: The learning loop alternates planning with option execution, computes plan quality, raises the quality goal, and adds learned gain-reward facts to the planning description.With exploration probability ϵ, CLINGCON generates plans; after execution, the resulting values are used to update subsequent planning.
- 4.3 Theoretical Status: The convergence of option-based hierarchical R-learning to an optimum over a finite number of options remains an open question, so the paper studies effectiveness empirically.The authors leave the theoretical analysis of Algorithm 1 for future work and evaluate it on two benchmark domains.
5 Experiment
Across Taxi and Grid World, PEORL combines symbolic plans with reinforcement learning to improve reward, adapt to changed rewards, and execute plans more reliably under uncertainty.
- 5.1 Taxi Domain: PEORL-agent has smaller cumulative-reward variance than RL-agent and HRL-agent because improper-action penalties do not occur.It initially explores longer plans before converging back to the shortest plans in the deterministic Taxi setting.
- 5.1 Taxi Domain: PEORL-agent outperforms RL-agent and HRL-agent in Taxi, including a changed-reward scenario where it discovers and converges to the optimal alternative.In Scenario 2, visiting (4,4) yields reward 30; PEORL discovers this extra reward while the other agents do not.
- 5.2 Grid World: In Grid World, PEORL achieves optimal behavior by avoiding bumpers and reliably activating and pushing the door, surpassing RL-agent.The setup includes execution failures and penalties for bumper cells, so successful behavior requires learning domain details absent from symbolic knowledge.
- 5.2 Grid World: PEORL improves symbolic-plan robustness in Grid World by reducing execution failures and decreasing the need for replanning.R-learning quickly learns how to execute options mapped from activate and push, whereas P-agent cannot improve execution reliability through learning.
6 Related Work
Related work has integrated symbolic plans or paths with reinforcement learning and has used learning to improve planning, but PEORL targets automatic option discovery through symbolic planning.
- 6 Related Work: Prior approaches use precompiled symbolic plans, finite-state-machine paths, or ASP-generated plans to assist reinforcement learning through one-shot planning.These methods treat symbolic structures as options or guidance for RL rather than automatically discovering options from symbolic planning.
- 6 Related Work: Other work applies clustering, Laplacian Eigenmaps, relational decision trees, weighted averages, or reinforcement learning to discover options or improve planning components.The paper distinguishes PEORL by using symbolic planning for automatic option discovery while also improving symbolic planning through learning.
7 Conclusion and Future Work
The paper concludes that PEORL lets symbolic planning and hierarchical reinforcement learning improve each other, producing rapid policy search and robust symbolic planning.
- 7 Conclusion and Future Work: PEORL uses mutual improvement between symbolic planning and HRL to achieve rapid policy search and robust symbolic planning.Future work includes formal hierarchical R-learning, more complex domains, and integration with deep RL for interpretable end-to-end solutions.