PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Meta-Policies for Neural Decision Making

Raban Emunds, Jannis Blüml, Quentin Delfosse, Kristian Kersting

arXiv:2610.37451v1cs.LG

TL;DR

Neural reinforcement-learning policies are difficult to interpret, while symbolic rules applied directly to low-level actions can become overly complex. NEXUS combines neural skills with symbolic high-level selection, and its evaluations report competitive game rewards alongside stronger game-progress performance and robustness to simplifications than deep RL baselines.

  • Problem

    Neural policies are opaque, while applying symbolic reasoning directly to low-level actions can produce prohibitively complex policies.

    Source: §1 · pp. 1–2

  • Method

    NEXUS combines neural low-level skills with a symbolic meta-policy that filters and selects skills, using object-defined subgoals and LLM-assisted rule and reward design.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    NEXUS achieves competitive game rewards versus DRL baselines and performs strongly on game-progress metrics, with improved robustness to environmental simplifications.

    Source: §5 · p. 8

  • Takeaways & Limitations

    NEXUS provides inspectable, modular skill-level structures and is reported to reduce reward hacking while remaining robust to environment simplifications.

    Source: §7 · p. 11

    LLM-generated rewards and meta-policies remain susceptible to misspecification and subsequent undesirable actions, making human oversight essential for safety-critical applications.

    Source: §7 · p. 11

Abstract

from arXiv · show

Deep Reinforcement Learning remains difficult to interpret and align due to the black-box nature of neural networks. While (neuro-)symbolic approaches improve transparency by incorporating symbolic reasoning, applying them directly to low-level actions results in overly complex, unreadable policies. We introduce NEXUS, a hierarchical RL framework that resolves this tension by combining neural skills with symbolic meta-policies. By shifting transparent reasoning to high-level skills rather than atomic actions, NEXUS drastically reduces symbolic complexity without sacrificing environment performance. Object-centric observations enable high-level reasoning and the design of reward functions and meta-policies by interaction with LLMs. We demonstrate that NEXUS agents learn disentangled skills, are interpretable regarding high-level goals, are better aligned to goals described in the game manual, less prone to reward hacking and more robust to game simplifications than deep RL. Finally, we isolate and evaluate the impact of different meta-policy architectures (neural, symbolic, neuro-symbolic) on environmental return and aligned task performance.

1 Introduction

Neural policies are difficult to interpret, while symbolic reasoning applied directly to low-level actions can become prohibitively complex. NEXUS combines neural skill execution with interpretable high-level selection, aiming to preserve performance while making skills and goals more inspectable.

1 Introduction

  • Symbolic reasoning can improve transparency, but applying it directly to low-level actions risks prohibitive rule complexity and poor scalability.Neural policies are also difficult to interpret, making undesirable behaviors hard to identify and correct.Source: §1 · pp. 1–2
  • NEXUS pairs neural low-level execution with an interpretable meta-policy that selects skills with distinct, object-defined subgoals.LLMs help identify skills, define meta-policy functions, and generate rewards for training the neural controllers.Source: §1 · p. 2
  • NEXUS extends Parallelised Q-Networks to hierarchical skill and meta-policy learning and compares three meta-policy variants with different interpretability levels.The variants and the extension are introduced as contributions of the paper.Source: §1 · p. 2
  • NEXUS is reported to learn disentangled skills with interpretable high-level selection while remaining competitive with deep RL and less susceptible to reward hacking.The paper also reports greater robustness to game simplifications.Source: §1 · p. 2

2 Background

The background introduces deep Q-learning and parallelized Q-networks, then situates NEXUS within hierarchical reinforcement learning. NEXUS assigns distinct rewards to skills and uses a meta-policy to select among them.

2 Background

  • NEXUS combines deep reinforcement learning with hierarchical decomposition, abstracting actions into skills that simplify complex tasks.Parallelised Q-Networks is described as a scalable variant of Deep Q-Networks.Source: §2 · pp. 2–3
  • Deep Q-Networks approximate the action-value function with a neural network when a tabular Q-function becomes infeasible for growing state spaces.DQN stabilizes learning with experience replay and a periodically updated target network.Source: §2 · p. 2
  • NEXUS extracts symbolic state, uses LLM-generated rules to filter candidate skills, and lets a trained Q-function select among the remaining skills.The chosen neural skill produces low-level actions; the meta-policy optimizes environmental reward while each skill uses its own LLM-generated reward.Source: Figure 2 · p. 3
  • Parallelised Q-Networks removes replay buffers and target networks, using many parallel environments and normalization to support faster training while maintaining performance.The passage attributes the speedup to its high degree of parallelization.Source: §2 · p. 3
  • In hierarchical reinforcement learning, a meta-policy selects temporally extended options, while sub-policies may be learned autonomously or guided by designed rewards.Autonomous option discovery can result in entanglement or collapse; NEXUS instead gives each skill an explicit, potentially LLM-generated reward.Source: §2 · p. 3

3 Neural Execution Under Symbols

NEXUS combines object-centric observations, neural skill policies, and a high-level meta-policy to make skill selection interpretable. It offers neural, symbolic, and neuro-symbolic variants, with the latter using learned preferences to resolve situations where multiple skills are eligible.

3 Neural Execution Under Symbols

  • NEXUS separates object-centric state processing, high-level skill selection, and low-level neural skill execution; object-centric states support symbolic rules and skill-specific reward design.The paper also says these representations allow LLMs to generate rules and reward functions.Source: §3 · p. 3

3.1 Hierarchical PQN

  • Each skill has a distinct reward and Q-function, while the meta-level selects skills using environment rewards; off-policy learning lets skills learn from trajectories generated by the active skill.The hierarchical setup uses separate low-level Q-functions and a meta-level Q-network.Source: §3.1 · p. 4

3.2 Interpretable Meta-Policy Function

  • The symbolic meta-policy maps object-centric states to skills through human-readable rules, whose pairwise-disjoint activation conditions make each selection traceable to one rule.The rule-based representation is intended to be simple to understand and modify.Source: §3.2 · p. 4

3.3 Neuro-symbolic Meta-Policy Function

  • The neuro-symbolic variant permits multiple symbolic conditions to be active, then uses learned Q-values to select the highest-valued eligible skill, adapting when several skills apply.The symbolic variant instead prioritizes mutually exclusive skills, which can require increasingly complex hard-coded rules in ambiguous situations.Source: §3.3 · p. 5

4 Experimental Evaluation

The evaluation tests whether NEXUS learns disentangled skills, makes high-level skill selection interpretable, competes with deep methods, and generalizes to simplified games. NEXUS learns target skills, remains competitive on game rewards while outperforming baselines on manual-defined goals, and shows robustness that varies by meta-policy variant.

4 Experimental Evaluation

  • The evaluation asks whether NEXUS learns meaningful disentangled skills, enables interpretable high-level selection, competes with deep methods, and improves robustness to game simplifications.The stated objective is an object-centric pipeline that solves complex environments while maintaining interpretability of high-level goals.Source: §4 · p. 5

4.1 Experimental Setup

  • The evaluation uses Seaquest and Kangaroo in JAXAtari and Crafter, compares three NEXUS meta-policy variants with PQN and other baselines, and aggregates results over six seeds.Agents receive symbolic environment states directly, and generalization is tested on simplified game variants.Source: §4.1 · pp. 5–6

4.2 Interpretability Results

  • In Seaquest and Kangaroo, NEXUS learns target skills more evenly than baselines, which tend to focus on a single environment-reward-maximizing skill.NEXUS learns most target skills from a single source of off-policy data, so a skill can learn from another skill's actions without being activated.Source: §4.2 · p. 6, Figure 3 · p. 6
  • On Crafter, NEXUS learns all five LLM-proposed skills simultaneously, while the PQN baseline focuses on only some skills.NEXUS approaches often converge faster on the proposed skills than the baseline.Source: Figure 4 · p. 6, §4.2 · p. 7
  • NEXUS's high-level rules make skill selection transparent, but when several conditions hold, the learned Q-function chooses among admissible skills, trading interpretability for flexibility.The symbolic Seaquest policy gives surfacing precedence when oxygen is low or all divers are collected.Source: §4.2 · p. 7

Policy complexity

  • On Seaquest, NUDGE needs 16 rules, 46 literals, and ADPL 23.56, while NEXUS uses 2–4 rules, 3–4 predicates, and ADPL 2.5–3.These metrics cover symbolic decision logic only; NEXUS's low-level skills remain neural.Source: Policy complexity · pp. 7–8, Table 1 · p. 8

4.3 Performance Results

  • Across games, NEXUS is competitive on game rewards and notably outperforms baselines on manual-defined goals, while Figure 6 reports Seaquest and Kangaroo results.The authors interpret the manual-goal results as evidence of baseline reward hacking and suggest that NEXUS mitigates reward misalignment while retaining good overall performance.Source: §4.3 · p. 8, Figure 6 · p. 8
  • Under unseen game simplifications, symbolic and neuro-symbolic NEXUS variants show improved Atari robustness, while the fully neural variant shows limited robustness.In Crafter, all NEXUS variants maintain or improve performance; the authors attribute robustness to symbolic steering that deactivates unnecessary skills.Source: §4.3 · p. 8

5 Discussion

NEXUS remains robust to game simplifications, with symbolic and neuro-symbolic variants showing different trade-offs in performance, interpretability, and flexibility. Its interpretability is scoped to skill selection and objectives, while the underlying action execution remains neural.

5 Discussion

  • NEXUS variants remain practical on simplified games: symbolic and neuro-symbolic improve on modified Seaquest, neuro-symbolic remains effective on modified Kangaroo, and all versions maintain or improve Crafter performance.Results are aggregated over six seeds with ±1 standard deviation; baseline performance on aligned metrics remains low.Source: §5 · p. 8, Figure 7 · p. 9
  • The symbolic meta-policy is recommended for high interpretability and alignment without sacrificing performance, while the neuro-symbolic variant suits settings with frequent ambiguity.The neural variant has strong raw game reward but limited interpretability and degraded performance on simplified games; defining symbolic rules adds overhead that LLMs can largely automate.Source: §5 · p. 9
  • Interpretability covers skill selection and explicit skill objectives, not low-level execution; action selection within each skill remains neural.In the neuro-symbolic variant, the learned Q-function makes the final choice among rule-admissible skills in ambiguous states.Source: §5 · p. 9
  • NEXUS assumes accurate object-centric scene decoding, although the authors also evaluate performance under noisy detections.The approach additionally requires a task description such as a game manual, and generated code and semantics require human verification and refinement.Source: §5 · pp. 9–10
  • NEXUS uses Q-learning and is therefore limited to discrete action spaces; the authors describe continuous-action extension as viable.They suggest an off-policy actor-critic in place of ϵ-greedy action selection.Source: §5 · p. 10

6 Related Work

Related work explores symbolic policies, hierarchical decompositions, and structured reward specifications. NEXUS combines semantic skill separation with a repeatedly reselected neuro-symbolic meta-policy, distinguishing it from methods that plan fixed skill sequences.

6 Related Work

  • Interpretable RL policies use decision trees [Delfosse et al., 2024b], logic rules [Delfosse et al., 2023a], parametric functions, or programmatic policies.Prior work also combines neural and logical policies and uses hierarchical policies with annotated task sketches or symbolic planning.Source: §6 · p. 10
  • Reward shaping and reward machines provide structured reward specifications, but the cited approaches typically learn one value function over flat actions and had not been evaluated on visually complex Atari or Crafter domains.NEXUS instead learns a separate Q-function per skill and uses object-centric reward specifications for semantic disentanglement.Source: §6 · p. 10
  • Unlike SCALAR’s fixed planned skill order and skill-by-skill PPO curriculum, NEXUS reselects skills at every step and learns them concurrently from shared off-policy experience.The paper describes the directions as complementary and identifies SCALAR’s trajectory-based refinement as a possible way to automate NEXUS specification verification.Source: §6 · p. 10
  • NEXUS resembles the Options framework but selects options at every time step, jointly determining their activation and termination through its rule set.The object-centric encoding enables the logical rule set to serve as a generalized initiation set.Source: §6 · pp. 10–11

7 Conclusion

NEXUS combines interpretable high-level skill selection with neural low-level execution. The conclusion reports disentangled skills, reduced reward hacking, and robustness to simplifications, while the motivation highlights the interpretability challenges of existing approaches.

7 Conclusion

  • NEXUS combines high-level interpretability at the meta-policy level with neural low-level action execution.The conclusion reports identifiable, disentangled skills, modular structures for inspection and intervention, reduced reward-hacking, and robustness to environment simplifications where standard deep RL agents fail.Source: §7 · p. 11
  • The authors report that LLMs can generate skill decompositions, reward functions, and symbolic meta-policies, supporting human-in-the-loop control at the skill level.They propose future work on detecting when new skills or meta-policy updates are needed, including through causal world models.Source: §7 · p. 11

A Motivation - Shortcomings of existing approaches

  • The motivation frames existing approaches as trading off interpretability: neural agents are black boxes, hierarchical agents can have entangled skills, and symbolic policies grow with complexity.The figure contrasts these approaches rather than reporting a quantitative result.Source: Figure 8 · p. 16

A.1 Interpretable Policy Sizes

The appendix contrasts the size of existing low-level interpretable policies with NEXUS’s compact high-level policies and points readers to the hierarchical PQN algorithm.

A.1 Interpretable Policy Sizes

  • Existing interpretable policies can be transparent at the low-level action selection yet difficult for humans to interpret because of their massive size; NEXUS uses hierarchical abstraction for compact high-level policies.The appendix visualizes policies from existing interpretable RL methods and gives examples from SCoBots, NUDGE, and INSIGHT.Source: §A.1 · pp. 16, 18, 19
  • The appendix identifies the illustrated Pong policies as examples from SCoBots, NUDGE, and INSIGHT.These listings provide concrete policy examples for the policy-size discussion.Source: §A.1 · pp. 18–19

B Hierarchical PQN Algorithm

  • The appendix provides the complete algorithm for hierarchical PQN with a neural meta-policy.The supplied passage identifies Algorithm 2 but does not state further algorithmic steps.Source: B Hierarchical PQN Algorithm · p. 19

C Implementation Details

The implementation adapts established RL codebases to object-centric inputs, and evaluates games selected for their multi-skill demands, including reward-hacking settings and open-ended play.

C Implementation Details

  • The implementations adapt CleanRL and PureJaxRL to object-centric inputs with lightweight MLP feature extractors; exploration was tuned by a brief sweep using final test return.Hyperparameters largely follow the original implementations, with more parallel environments enabled by JAX.Source: C Implementation Details · p. 19
  • Experiments used six random seeds, reported standard deviations, and ran on a single NVIDIA Tesla V100 GPU.The authors state that the repository includes code and configuration files to reproduce the experiment results.Source: C Implementation Details · p. 20

C.1 Environments

  • Crafter, Seaquest, and Kangaroo were selected because their gameplay naturally requires combining low-level skills.Seaquest and Kangaroo exhibit reward hacking; Crafter adds open-ended navigation, combat, resource collection, and crafting.Source: §C.1 · p. 20, Figure 9 · p. 20

C.2 Evaluation Metrics

The evaluation combines performance and goal-alignment measures with rule-based proxies for policy complexity, then describes how those proxies apply to NUDGE and NEXUS policies.

C.2 Evaluation Metrics

  • The evaluation uses skill returns, Human-Normalized Score, and aligned environment goal scores to assess skill learning, relative performance, and task progress.HNS uses human and random baselines; 1.0 indicates human-level performance, above 1.0 superhuman performance, and below 0 sub-random performance.Source: §C.2 · pp. 20–21
  • Aligned goal scores track divers retrieved in Seaquest and platforms reached in Kangaroo, grounding evaluation in the games’ manual-stated objectives.These measures address cases where environment rewards may not match intended goals or may permit reward hacking.Source: §C.2 · p. 21

C.3 Interpretability Proxies - Detailed Computation

  • Rule size, total rule length, and Average Decision Path Length (ADPL) measure policy branches, rule conditions, and expected decision checks.ADPL is defined over a uniform distribution of execution paths; each path’s cumulative evaluations contribute to the measure.Source: §C.3 · p. 21, §1 · p. 22

2. NUDGE Policy Computations

3. NEXUS (symbolic) Computations

4. NEXUS (nesy) Computations

B. Average Decision Path Length (ADPLNEXUS-2)

Reward-shaping baselines acquire some isolated skills but fall short of NEXUS overall, while the appendix also describes the decision-check structure of one NEXUS variant.

B. Average Decision Path Length (ADPLNEXUS-2)

  • In NEXUS (nesy), both conditional blocks run each forward pass; the second averages 1.5 checks under the stated condition frequency.The first block always requires one check.Source: B. Average Decision Path Length… · p. 23

D.1 Comparative evaluation against reward shaping baselines

  • The comparison includes dynamic symbolic reward selection, uniform fixed weighting, and LLM-generated fixed reward weights.The symbolic strategy uses the same LLM-designed meta-policy as NEXUS (symbolic) to select the active skill reward at each step.Source: §D.1 · p. 24
  • All three reward-shaping baselines fall significantly short of NEXUS across both environments, despite acquiring some localized behaviors.The baselines fail to coordinate skills effectively, whereas NEXUS’s neuro-symbolic architecture supports balanced skill learning and coordination.Source: §D.1 · p. 24, Figure 10 · p. 24

D.2 Analysis of neuro-symbolic meta-policy impact and skill activation dynamics

The neuro-symbolic meta-policy’s choices affect returns and aligned scores; additional-game evaluation and comparisons with interpretable baselines show the scope and trade-offs of NEXUS’s high-level policy design.

D.2 Analysis of neuro-symbolic meta-policy impact and skill activation dynamics

  • Replacing Meta-Q-value maximization with random valid-skill choices lowers overall returns and aligned metrics in both Seaquest and Kangaroo, more strongly in Seaquest.Default skills dominate usage, while alternatives are selected under specific conditions; Figure 11 reports activation frequencies for these skills.Source: §D.2 · p. 24, Figure 11 · p. 25
  • In Seaquest, collect_divers runs on 20% of steps and 30% of eligible steps; in Kangaroo, collect_fruits runs on 13% and 46%, respectively.upward accounts for 7% of Seaquest steps and 70% of eligible steps; handle_enemies accounts for 9% and 40% in Kangaroo.Source: §D.2 · p. 25

D.3 Additional environments

  • NEXUS learns skills in Pong, Breakout, and Freeway and achieves performance comparable to baseline methods across the three evaluated environments.These games are less complex than Kangaroo and Seaquest, and their skills are not strictly necessary for reaching environment goals.Source: §D.3 · p. 25, Figure 12 · p. 26, Figure 13 · p. 26

D.4 Neuro-symbolic RL baselines.

  • Unlike NUDGE, BlendRL, and SCoBots, NEXUS defines policies over skills and active goals, yielding more concise meta-policies and avoiding manual valuation functions.The paper describes a trade-off: action-level transparency and performance in prior methods rely on extensive manual inductive biases, while NEXUS targets high-level interpretability with less reliance on them.Source: §D.4 · pp. 25–27

E Noisy Detections

Under noisy detections, NEXUS approaches retained comparatively stable performance when noise affected both training and evaluation, though evaluation-only noise hurt performance. The appendix also describes LLM-assisted skill and meta-policy generation, followed by human refinement and game-specific skill examples.

E Noisy Detections

  • With noise during training and evaluation, NEXUS approaches had only small drops in Seaquest and small drops or improvements in Kangaroo; evaluation-only noise caused significant losses across methods.Under evaluation-only noise in Seaquest, symbolic and neuro-symbolic agents still rescued relatively many divers. The authors hypothesize that noise increased exploration and reduced overfitting in Kangaroo.Source: E Noisy Detections · p. 27, Figure 17 · p. 28, Figure 16 · p. 28

F LLM Interaction

  • The LLM receives the game manual and object-observation types, proposes skills and rewards, then proposes a fixed meta-policy; authors refine the outputs before implementation.Crafter skill rewards were written manually; the authors also made framework and semantic changes, and derived neuro-symbolic rules manually from symbolic policies.Source: F LLM Interaction · p. 28

1. Obstacle Avoidance

  • In Kangaroo, the fixed skill-selection priority chooses obstacle avoidance when an apple or monkey is nearby.The prompt-derived skill description says to dodge or punch to avoid contact with apples or monkeys.Source: §1 · pp. 30–31

1. Avoid Obstacles (Survival Navigation)

  • Seaquest’s avoidance skill targets collisions with sharks, enemy submarines, missiles, and the patrol submarine at the surface.The skill description covers navigation both underwater and while surfacing.Source: §1 · p. 32

2. Collect Divers

  • Seaquest’s diver-collection reward is positive when the nearby-diver count decreases between observations.The reward compares the count in the previous and current observations.Source: §2 · p. 32

3. Surface Strategically

  • Seaquest’s surfacing skill rewards surfacing with six divers more than surfacing when oxygen is low.The described conditions are six collected divers or low oxygen; the reward is 2.0 for six divers and 0.5 otherwise.Source: §3 · p. 32

4. Eliminate Threats (Combat)

  • Seaquest combat rewards reducing enemy counts, while Crafter’s generated policy prioritizes survival, combat, gathering and crafting, then exploration.The Crafter skill list identifies five skills as sufficient building blocks and includes gathering available materials.Source: §4 · pp. 32, 35, Figure 24 · p. 35

Related papers

Loading 2610.37451v1…