SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning

Introduction

• DRL can learn fine-grained policies from high-dimensional inputs, but often needs millions of samples, struggles with sparse-reward long-horizon tasks, and remains difficult to interpret.The introduction motivates interpretability as important for user trust and decision-making.Source: Introduction · p. 1
• Hierarchical subtasks and object representations have improved data efficiency, but prior approaches have not matched the interpretability of human-like object-based subtask planning.The introduction points to object-based representation and planning as a model for combining learning with interpretable subtasks (Kulkarni et al., 2016; Keramati et al., 2018).Source: Introduction · p. 1
• Symbolic planning uses explicit, human-readable knowledge to plan action sequences, monitor execution, and replan, offering a white-box alternative that needs less trial-and-error than RL.Earlier SP+RL frameworks used symbolic plans to guide meaningful exploration and improve data efficiency in tabular problems (Yang et al., 2018; Lu et al., 2018).Source: Introduction · pp. 1–2
• SDRL combines symbolic planning with hierarchical DRL, using an intrinsic goal based on plan quality to drive a planner–controller–meta-controller architecture.The approach assumes experts can specify high-level, potentially inaccurate dynamics and perception modules can recognize objects and their properties. The planner schedules symbolic subtasks toward intrinsic goals, controllers learn sub-policies from intrinsic rewards, and the meta-controller evaluates controller performance using extrinsic rewards and proposes new goals.Source: Introduction · p. 2
• The components cross-fertilize toward an optimal symbolic plan; the paper proves plan convergence conditional on meta-controller convergence and evaluates interpretability and data efficiency in Taxi and Montezuma’s Revenge.The framework is instantiated with action language BC for planning and R-learning for meta-controller learning; the evaluation uses human-readable subtasks and automatically selected and learned modular control policies.Source: Introduction · p. 2

Planning with Action Language BC

• BC action descriptions represent world properties with fluent atoms and specify causal laws for static conditions, action effects, nonexecutable actions, and inertial persistence.Fluents and action constants are defined over a signature; static laws express state relations, while dynamic laws describe effects of actions.Source: Preliminaries · p. 2
• A planning problem (I, G, D) seeks a transition path from initial state I to goal state G, and CLINGO answer sets encode paths that solve it.States are complete sets of fluent atoms, and transitions consist of a state, an action, and a next state.Source: Preliminaries · p. 2

Reinforcement Learning

• An MDP defines states, actions, probabilistic transitions, rewards, and discounting; reinforcement learning learns a near-optimal policy through observed transitions and rewards.For finite-horizon problems, the paper evaluates policies by expected undiscounted reward.Source: Preliminaries · p. 2

Options

• An option (I, π, β) is available in its initiation set, selects actions through its policy, and terminates stochastically, creating task-level and primitive-action levels.The paper presents options as temporally extended courses of action within hierarchical reinforcement learning.Source: Preliminaries · pp. 2–3

Interpretability

• SDRL uses symbolic knowledge for task-level DRL interpretability and is interpretable by construction, unlike prior program-induction approaches to policy interpretability (Verma et al., 2018).The paper distinguishes task-level interpretability from work explaining deep-network processing (Ribeiro et al., 2016).Source: Related Work · p. 3

Hierarchical Deep Reinforcement Learning

• Unlike predefined or difficult-to-interpret options, SDRL maps symbolic transitions to subtasks and uses an off-the-shelf planner to generate and improve plans.It explicitly represents objects, properties, and high-level transition dynamics, contrasting with the object-oriented MDP abstraction of Keramati et al. (2018).Source: Related Work · p. 3

Integrating Symbolic Planning with Reinforcement Learning

• SDRL extends mostly tabular SP-RL work (Yang et al., 2018; Lu et al., 2018) by adding intrinsic goals and DRL to PEORL (Yang et al., 2018).Its meta-controller bridges symbolic-state planning and pixel-based DRL by learning at the task level from extrinsic rewards derived from DRL training performance; it selects subtasks and improves plans by sequencing learnable subtasks.Source: Related Work · p. 3

Computational Models of Intrinsic Motivation

• SDRL models intrinsic motivation by using symbolic planning’s internal utility to assess plan quality and motivate plan improvement through larger accumulated rewards.This contrasts with prior work using intrinsic motivation to address sparse feedback and delayed rewards (Kulkarni et al., 2016).Source: Related Work · p. 3

Symbolic Representation

• The expert-provided symbolic structure uses causal rules, with a coarse high-level formulation intended to support flexibility under domain change and uncertainty without requiring an accurate, laboriously crafted model.The paper notes that symbolic formulations can be adapted across domains by changing objects or adding a few rules.Source: SDRL Framework · p. 3

SDRL Framework

• The planner proposes symbolic subtasks, controllers learn their policies using intrinsic rewards, and a meta-controller uses competence-based extrinsic rewards to update values that guide new plans.The loop measures plan quality, updates the intrinsic goal, and passes learned reward values back to the symbolic formulation.Source: SDRL Framework · p. 4, Planning and Learning · p. 5

Symbolic Representation

• The symbolic action description represents gain rewards optimistically by default and tracks plan quality as cumulative average-adjusted reward; intrinsic goals constrain plan quality without requiring a logical goal condition.The paper characterizes dropping the logical constraint as enabling model-based exploration by planning.Source: Symbolic Representation · p. 4

From Symbolic Transitions to Options

• Symbolic transitions map to semi-Markov options whose initiation sets and termination conditions are grounded in sensory inputs by a pre-trained oracle.The oracle identifies whether symbolic fluent properties hold in a sensory input; an intra-option policy selects primitive actions.Source: From Symbolic Transitions to Options · p. 4

Rewards

• Subtask competence is measured by success over the previous 100 episodes; below the 0.9 threshold, the extrinsic reward penalizes selection as unlearnable, while reliable policies receive environmental cumulative reward.Intrinsic rewards include pseudo-rewards to encourage subtask achievement.Source: SDRL Framework · p. 4, Rewards · p. 5

Planning and Learning

• If R-learning converges, the algorithm terminates if and only if an optimal symbolic plan exists, and any plan at termination is optimal.The framework also guarantees symbolic-level optimality conditioned on R-learning convergence.Source: Planning and Learning · p. 5

Montezuma’s Revenge.

• SDRL reaches maximal cumulative external reward of 400 around 1.5M samples, while hDQN cannot reliably reach 400 around 2.5M samples.SDRL also has smaller variance than hDQN across runs.Source: Figure 2 · p. 6, Montezuma’s Revenge · p. 7

Taxi Domain

• From Task 8 onward, SDRL learns to collect only the coupon as rewards decline, while standard PEORL and SR retain their previously learned policies.From Tasks 1–7, the optimal policy is to collect the coupon and drop off the passenger; both SDRL and standard PEORL learn it.Source: Taxi Domain · p. 6

Montezuma’s Revenge.

• SDRL’s symbolic planner maps transitions among six predefined locations to 13 subtasks, defined as symbolic transitions rather than hDQN’s object-associated subtasks.This representation gives subtasks initiation sets and termination conditions expressed through logical literals.Source: Montezuma’s Revenge · p. 7
• SDRL discovers the key-collecting plan after 0.5M samples, then sequences subtasks 1–7 to reach maximal reward around 1.5M samples.Only Subtask 3 (collecting the key) and Subtask 7 (opening the right door) receive external rewards, of +100 and +300 respectively.Source: Montezuma’s Revenge · p. 7
• DQNs learn Subtasks 1–10, with seven selected in the final solution at a 100% success ratio; the planner prunes Subtasks 8–13 for poor rewards or learning difficulty.Subgoals are activated when the starting state is satisfied and learned subtasks can be sequenced and reused in other plans.Source: Montezuma’s Revenge · p. 7

• SDRL uses explicitly represented symbolic knowledge for high-level planning based on intrinsic goals and DRL to learn low-level control policies.Source: Conclusions · p. 7
• Benchmark evaluations validate SDRL’s improved task-level interpretability and data efficiency for deep reinforcement learning.Source: Conclusions · p. 7

Proof of Theorem 1

• Algorithm 1 terminates if and only if there is an upper bound on plan quality.The proof states this equivalence directly.Source: Proof of Theorem 1 · p. 9
• A plan-quality upper bound exists if and only if no plan contains a loop whose cumulative ρ_a(s) is positive.This is the loop condition identified as equivalent to termination.Source: Proof of Theorem 1 · p. 9
• By equations (7) and (8), the loop condition can be expressed using rewards and R values at the loop’s endpoint and start.The displayed proof fragments identify the terms as re(s, a), R(s|L|, a), and R(s0, a).Source: Proof of Theorem 1 · p. 9
• Because a loop ends where it starts, its endpoint and start R terms cancel, leaving cumulative re(s, a) no greater than zero.The proof uses s|L| = s0 to obtain this reward condition.Source: Proof of Theorem 1 · p. 9
• The nonpositive cumulative-reward condition is equivalent to every plan having no positive loop.This restates the proof’s criterion in terms of the cumulative reward of a loop.Source: Proof of Theorem 1 · p. 9
• The absence of a positive loop in any plan is equivalent to the existence of an optimal plan.The proof uses this equivalence to conclude Theorem 1.Source: Proof of Theorem 1 · p. 9

Proof of Theorem 2

• Π* returned by Algorithm 1 is a plan for (I, G, D), and the proof considers whether its quality is no greater than that of any plan Π.The argument invokes (Lee et al., 2013, Theorem 2).Source: Proof of Theorem 2 · p. 9
• By (8), the quality inequality is equivalent to a further condition used in the proof.Source: Proof of Theorem 2 · p. 9
• Because s|Π| and s|Π*| are terminal states with no options available, the proof concludes using the condition stated for transitions in Π*.The excerpt does not provide the full condition or its preceding derivation.Source: Proof of Theorem 2 · p. 9