Source-linked AI summary
SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning
Daoming Lyu, Fangkai Yang, Bo Liu, Steven Gustafson
TL;DR
DRL lacks interpretability and can be data-inefficient when learning long-horizon behavior from high-dimensional sensory inputs. The paper proposes SDRL, combining symbolic planning with hierarchical DRL through planner, controller, and meta-controller components. Evaluations report improved task-level interpretability and data efficiency, while the framework assumes expert symbolic structure and a perception module for symbol grounding.
Problem
DRL learns from high-dimensional sensory inputs but is criticized for limited interpretability and data efficiency, especially for long-horizon tasks with sparse or delayed rewards.
Method
SDRL combines symbolic planning with hierarchical DRL: a planner schedules symbolic subtasks, controllers learn their policies, and a meta-controller evaluates performance and proposes intrinsic goals.
Results
The framework demonstrates improved task-level interpretability and data efficiency on benchmark problems, including Taxi and Montezuma’s Revenge.
Takeaways & Limitations
Relating symbolic actions to learned options makes subtasks human-readable while retaining learning from high-dimensional sensory inputs.
Takeaways & Limitations
SDRL assumes human experts provide symbolic structure and a pre-trained perception module maps sensory inputs to symbolic states.
Abstract
from arXiv · showhide
Deep reinforcement learning (DRL) has gained great success by learning directly from high-dimensional sensory inputs, yet is notorious for the lack of interpretability. Interpretability of the subtasks is critical in hierarchical decision-making as it increases the transparency of black-box-style DRL approach and helps the RL practitioners to understand the high-level behavior of the system better. In this paper, we introduce symbolic planning into DRL and propose a framework of Symbolic Deep Reinforcement Learning (SDRL) that can handle both high-dimensional sensory inputs and symbolic planning. The task-level interpretability is enabled by relating symbolic actions to options.This framework features a planner -- controller -- meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively. The three components cross-fertilize each other and eventually converge to an optimal symbolic plan along with the learned subtasks, bringing together the advantages of long-term planning capability with symbolic knowledge and end-to-end reinforcement learning directly from a high-dimensional sensory input. Experimental results validate the interpretability of subtasks, along with improved data efficiency compared with state-of-the-art approaches.
Introduction
The paper addresses DRL’s limited interpretability and data efficiency by combining symbolic planning with hierarchical reinforcement learning. Its planner–controller–meta-controller design schedules, learns, and evaluates interpretable subtasks.
- DRL can learn from high-dimensional sensory inputs but often requires millions of samples and struggles with long-horizon actions under sparse or delayed rewards.
- Symbolic planning supplies human-readable knowledge and high-level reasoning intended to improve both task-level interpretability and DRL data efficiency.
- The planner generates symbolic-action subtasks to achieve intrinsic goals, while controllers learn each subtask’s policy from intrinsic rewards.
- A meta-controller evaluates controller training with extrinsic rewards and proposes new intrinsic goals for the planner.
- The components jointly converge toward an optimal symbolic plan with learned subtasks, and evaluations report improved interpretability and data efficiency.
Preliminaries
The preliminaries define symbolic transition systems, planning problems, MDPs, policy evaluation, and temporally extended options. Together, these concepts provide the formal vocabulary for connecting symbolic planning with hierarchical reinforcement learning.
- Symbolic Planning: Action descriptions use static and dynamic causal laws to represent fluent relationships, action effects, nonexecutability, and inertia.
- Symbolic Planning: A symbolic state is a complete set of fluent atoms, and a planning problem consists of an initial state, goal state, and action description.
- MDPs and RL: An MDP comprises states, actions, transition probabilities, rewards, and a discount factor, while reinforcement learning learns policies from observed transitions and rewards.
- Policy Evaluation: Policy performance can be measured by expected discounted infinite-horizon rewards or expected undiscounted finite-horizon rewards.
- Options: An option is a temporally extended action defined by an initiation set, policy, and stochastic termination condition.
Related Work
Prior work studies interpretability and hierarchical reinforcement learning, but the paper positions SDRL as using symbolic knowledge to make learned subtasks interpretable while automatically selecting and sequencing them.
- Interpretability: Interpretability concerns making system internals understandable, whereas this work targets task-level interpretability through symbolic knowledge.
- Hierarchical Deep Reinforcement Learning: The options framework addresses long-horizon action sequences with sparse rewards but often assumes predefined options whose learned policies can be difficult to interpret.
- Hierarchical Deep Reinforcement Learning: Unlike approaches focused on predefined or eigen-based options, the paper describes automatic selection and sequencing of learnable subtasks.
- Intrinsic Motivation: SDRL uses symbolic planning to measure plan quality with an internal utility function, motivating improvement through larger accumulated rewards.
SDRL Framework
SDRL combines symbolic planning with hierarchical deep reinforcement learning for high-dimensional sensory inputs, using symbolic subtasks linked to learned options. Its planner, controllers, and meta-controller iteratively improve plan quality and subtask policies, with symbolic-level optimality conditioned on R-learning convergence.
- Problem formulation: SDRL models high-dimensional sensory inputs and primitive actions separately from symbolic states and actions, learning subtasks and corresponding sub-policies to maximize cumulative reward.A human-provided symbolic structure captures causal rules describing objects, fluents, and subtask effects.
- Planner and options: The symbolic planner generates subtask sequences for an intrinsic goal, while symbol grounding maps sensory inputs to symbolic states and induces corresponding options.Each symbolic transition can define an option with an initiation set, intra-option policy, and termination condition.
- Symbolic representation: Plan quality is represented through gain rewards and cumulative average-adjusted reward, with optimistic initialization promoting exploration.Intrinsic goals use plan quality rather than a human-specified logical goal condition, enabling model-based exploration by planning.
- Hierarchical learning: Controllers learn each sub-policy with intrinsic rewards, while the meta-controller evaluates subtask competence and environmental return through extrinsic rewards.A subtask success ratio averaged over the previous 100 episodes determines whether a learned sub-policy reliably achieves its subgoal; a threshold of 0.9 identifies unlearnable subtasks.
- Planning and learning loop: The planning-learning loop generates symbolic plans, trains controllers with deep Q-learning, updates meta-controller values using R-learning, and feeds learned rewards back into planning.The loop continues until the symbolic plan cannot be further improved.
- Guarantees: If the meta-controller’s R-learning converges, the algorithm terminates if and only if an optimal symbolic plan exists, and its terminating plan is optimal.The paper therefore guarantees symbolic-level optimality under the stated convergence condition.
Experiment
Experiments in Taxi and Montezuma’s Revenge evaluate SDRL’s intrinsic-goal planning, subtask learning, interpretability, and data efficiency. SDRL adapts plans to changing rewards and learns a high-level symbolic sequence from sensory inputs.
- SDRL is evaluated in the Taxi domain and Montezuma’s Revenge, using Taxi for intrinsically motivated planning and Montezuma’s Revenge for interpretability and data efficiency.
- Taxi Domain: As the Taxi reward structure changes, SDRL and standard PEORL initially learn the coupon-and-drop-off policy, while SDRL later adapts toward the coupon-only policy when that becomes optimal.The experiments report learning curves averaged over 10 runs.
- Montezuma’s Revenge: In Montezuma’s Revenge, SDRL discovers the key-collection plan after 0.5M samples and reaches cumulative external reward 400 around 1.5M samples.The final plan sequences subtasks 1–7; only subtasks 3 and 7 receive external rewards of +100 and +300.
- Montezuma’s Revenge: SDRL reaches the score of 400 around 1.5M samples, whereas hDQN cannot reliably achieve it around 2.5M samples, and SDRL has lower variance.The experiment uses 10 runs, with the figure shadows representing variance.
- Montezuma’s Revenge: Seven of the successfully learned subtasks are selected in the final solution with a success ratio of 100%, while other subtasks are pruned or discarded for poor performance or learning difficulty.Subtasks can be sequenced and reused when their starting-state conditions are satisfied.
Conclusions
The paper proposes SDRL, combining symbolic high-level planning with DRL-based low-level control. Evaluation on benchmark problems validates improved task-level interpretability and data efficiency.
- SDRL uses explicitly represented symbolic knowledge for high-level planning and DRL to learn low-level control policies, improving task-level interpretability and data efficiency.
Proof of Theorem 1
The proof establishes that Algorithm 1 terminates when plan quality is bounded, equivalently when no positive-reward loop exists. This condition is equivalent to the existence of an optimal plan.
- Algorithm 1 terminates if and only if plan quality has an upper bound, equivalently if no plan contains a loop with positive cumulative reward.
Proof of Theorem 2
The proof characterizes the returned plan as optimal by comparing its quality with every alternative plan. The argument uses terminal states with no available options to complete the comparison.
- The returned plan Π∗ is optimal when its quality is at least that of every other plan, with terminal states contributing no further options.