Source-linked AI summary
Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agents
Norbert Oswald, Fabian Deuser, Thomas Bräunl
TL;DR
The paper asks where symbolic action-precondition knowledge should enter RL, since poor placement can leave agents vulnerable to hallucinated preconditions. It compares three placements of a precondition Bayesian network; across tasks, the gains differ by setting, spanning solution quality, sample efficiency, and routing generalisation.
Problem
Where symbolic knowledge about action preconditions should enter the RL loop is less studied than what knowledge to represent, despite the safety and reliability risks of hallucinated preconditions in changing environments.
Source: §1 · p. 2Method
The paper compares an inference-only verifier, a training-time enforcer, and a network-integrated learner using a deterministic precondition Bayesian network.
Source: §3 · p. 3, §3.2 · p. 4Results
Across tasks, gains shift from MiniGrid solution quality (SE 98.2% vs PPO+RND 88.8%) to Fetch efficiency (SE and SL about 2× sooner) and routing generalisation (SV and SE near-optimal; SL behind).
Source: §4.2 · p. 6, §4.3 · p. 7, §5 · p. 8Takeaways & Limitations
With a valid BN and correctly grounded predicates, external SV and SE gate structural actions and make each firing traceable to its permitting condition.
Source: §5 · p. 8The framework requires a sound precondition BN, and structure alone does not make complex RL tasks easy.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Humans carry behaviour knowledge of how to act in familiar situations into every new task rather than relearning it from scratch. There is no reason a Reinforcement Learning (RL) agent shouldn't do the same: known behaviour patterns need not be learned, only applied. Neuro-symbolic RL bridges prior knowledge and RL by injecting symbolic knowledge alongside a learned policy. The point at which this knowledge is integrated is critical: a poor choice can produce, for instance, hallucinated preconditions, which surface as safety and reliability problems in agents acting in changing environments. We formalise this behavioural knowledge as a precondition Bayesian network (BN) over the agent's \emph{structural actions} - the actions whose legality depends on preconditions, such as picking up a key, grasping a block, toggling a door, or dropping an object. The BN restricts when these actions may fire, and we inject it into the RL loop at three placements: (1) a \emph{symbolic verifier}, consulted only at inference, that fires a structural action once its preconditions hold; (2) a \emph{symbolic enforcer}, active during both training and inference, that governs structural-action use throughout learning; and (3) a \emph{symbolic learner}, which folds the knowledge into the network and learns the restriction and use of structural actions itself. To test the three variants we run experiments on two benchmarks with opposite regimes: one built on long, ordered planning chains, the other on continuous manipulation. We compare against strong baselines on solution quality, sample efficiency, and traceability. The payoff is substantial. On MiniGrid, all three placements improve the \emph{solution quality} over the PPO+RND baseline, the symbolic enforcer leading at $98.2\%$ against the baseline's $88.8\%$. On Fetch, $\dots$
1 Introduction
Prior action knowledge can spare RL agents from rediscovering how to act, but where that knowledge enters the learning loop remains underexplored. This paper compares three injection placements for a precondition Bayesian network across embodied-RL benchmarks.
1 Introduction
- Prior knowledge of legal actions and their effects can reduce rediscovery, helping agents explore less and learn faster and better, especially in complex environments.The paper frames where knowledge enters the RL loop as less studied than what knowledge an agent needs or how to represent it.Source: §1 · p. 2
- In changing environments, hallucinated preconditions or brittle domain models can create safety and reliability problems when structural actions fire illegally.The paper focuses on settings where agents must reason over goals and constraints, execute long action sequences, and recover when assumptions fail.Source: §1 · p. 2
- The paper formalises behavioural knowledge as a precondition Bayesian network over states, predicates, subgoals, and actions.Behavioural knowledge means prior knowledge about action preconditions and effects and the behaviour patterns that follow, whether hand-specified or learned from data.Source: §1 · p. 2
- The study compares inference-time, training-time, and network-level injection on MiniGrid and Fetch, and tests whether their ordering extends to street-network routing.It evaluates solution quality, sample efficiency, and traceability using the same precondition specification.Source: §1 · p. 2
2 Related Work
Prior work represents behavioural knowledge in several forms and usually studies a single point of entry into RL. This paper positions inference-time, training-time, and network-level injection as alternatives to compare on one specification.
2 Related Work
- Prior work spans graph, Bayesian-network, learned-operator, predicate, abstraction, and PDDL representations, with knowledge either hand-written or learned from data.Examples include subtask-precondition graphs [31], Q-Cogni's Bayesian network [9], and PDDL model synthesis with an LLM [13].Source: §2 · p. 2
- Inference-time shielding wraps a trained policy to veto or correct unsafe actions, while hierarchical methods divide decisions between high- and low-level layers.Examples include shielding [1] and hierarchical frameworks such as Options, HAM, MAXQ [11], HAC, and Feudal Nets [33].Source: §2 · p. 2
- External symbolic structure can also shape learning through fixed controllers or rewards, while a third placement compiles symbolic knowledge into the network.SDRL uses a STRIPS/PDDL planner to sequence subgoals; PDDL-guided RL and reward machines inject structure through rewards.Source: §2 · p. 3
- Prior work usually studies a shield, hierarchy, or in-network module in isolation, whereas this paper compares three injection points on the same specification.The comparison characterises when each placement wins.Source: §2 · p. 3
3 Framework: Symbolic Placement Strategies
The framework represents action preconditions as a Bayesian network (BN) and compares three placements for using that knowledge in an RL loop. The placements differ in whether the BN gates actions at inference, governs action use during training, or is distilled into a learned network head.
3 Framework: Symbolic Placement Strategies
- A precondition BN formalizes when structural actions are legal, with Boolean conditionals making the legality mask a direct Boolean evaluation.The paper notes that probabilistic conditionals could replace the Boolean mask with a soft probability mask without changing the framework.Source: §3 · p. 3
3.1 Precondition Bayesian Networks
- The BN represents state quantities, predicates, task-progress subgoals, actions, and precondition functions that determine structural-action legality.Subgoals mark ordered task stages, and each structural action's precondition depends on its parent predicates and subgoals.Source: §3.1 · pp. 3–4
- For any state, the BN produces a legality mask whose bit indicates whether the corresponding structural action is legal.Navigation actions change the agent's pose, while structural actions change the world state.Source: §3.1 · p. 4
3.2.1 SV: Symbolic Verifier
- The Symbolic Verifier (SV) applies the BN only at inference, blocking illegal structural actions and auto-firing a unique legal one when the policy does not propose one.SV is bidirectional: it can reject an illegal firing or trigger an unambiguous legal action.Source: §3.2.1 · p. 4
3.2.2 SE: Symbolic Enforcer
- The Symbolic Enforcer (SE) applies the legality mask during training and inference, leaving the policy to supply navigation while the BN decides structural actions.Its high-level BN rules have zero learnable parameters, making the approach cheap and stable but dependent on a correct precondition specification.Source: §3.2.2 · p. 5
3.2.3 SL: Symbolic Learner
- The Symbolic Learner (SL) feeds predicate and subgoal vectors to a learned head that selects structural actions, without applying an external mask at inference.Unlike SV and SE, SL can select a structural action whose preconditions do not hold.Source: §3.2.3 · p. 5
4 Experiments
The experiments compare three placements of precondition knowledge across discrete planning, continuous control, and real-street routing, evaluating task success, learning speed, and traceability.
4 Experiments
- The benchmarks contrast MiniGrid’s discrete, partially observed, long-horizon planning with Fetch’s continuous control; their structural actions include pickup, toggle, drop, and grasp.MiniGrid uses a 7 × 7 egocentric view, while Fetch has a continuous 25-dimensional observation space and 7-DoF arm.Source: §4 · pp. 5–6
4.1 Evaluation
- Evaluation measures episode-level task success using each benchmark’s native metric, sample efficiency by steps to 80% of baseline final performance, and traceability qualitatively.Tasks are resampled each episode, so the scores reflect generalization across each environment’s configuration distribution.Source: §4.1 · p. 6
4.2 MiniGrid ObstructedMaze
- On MiniGrid, SE reaches 98.2% success versus 88.8% for PPO+RND, while SL reaches 93.8% and SV reaches 90.0%.The baseline and all placements use the same PPO+RND navigation policy; Table 2 presents the placement results.Source: Table 2 · p. 6, §4.2 · p. 6
- For SV and SE, BN gating makes each structural-action firing traceable to grounded predicates, whereas SL’s learned rule can drift from the true precondition.SL does not consult the BN at inference.Source: §4.2 · p. 7
4.3 FetchPickAndPlace
- On Fetch, SE and SL reach SAC+HER’s converged success about 2× sooner, at roughly 100k versus 250k environment steps, while SV does not speed learning.All variants reach the same success rate; SE auto-fires grasp during training and SL supervises it, leaving the policy to learn reaching.Source: Figure 3 · p. 7, §4.3 · p. 7
4.4 Experiments beyond embodied tasks
- On 5,000 held-out Astoria trips, SV and SE achieve path ratios of 1.03 and 1.04 and SPL of 0.70 and 0.67, respectively.Held-out endpoints were never training goals; SL has the lowest success rate despite 6× the training budget.Source: §4.4 · p. 8
5 Discussion and Conclusion
Across three domains, applying known preconditions during learning generally improves learning outcomes, while external placements keep structural actions checkable; sound BNs and ordinary RL tuning remain necessary.
5 Discussion and Conclusion
- Across all investigated domains, applying correct behavioural knowledge as the agent learns is more effective than using network capacity to re-derive it.The placement of knowledge, rather than the underlying learner or architecture, drove the results across the learners tested.Source: §5 · p. 8
- On Fetch, SE and SL reach SAC+HER’s solution-quality ceiling about 2× sooner, while SV merely matches it.On MiniGrid, all placements exceed baseline solution quality, with SE highest; on held-out routing, SV and SE are near-optimal while SL falls behind.Source: §5 · p. 8
- With a valid BN and correctly grounded predicates, SV and SE guarantee that covered structural actions fire only when their preconditions hold; SL provides no such guarantee.External gating also traces each firing to the condition that permitted it; SL’s learned rule can drift from the true precondition.Source: §5 · p. 8
- The framework requires a sound BN, and structure alone does not eliminate the need for policy and parameter selection, reward shaping, or careful exploration handling.The BN reduces how much of this usual RL apparatus is needed, but does not remove it.Source: §5 · p. 8
- Current BNs are small and flat, fitting a handful of predicates, subgoals, and actions; scaling raises injection-cost and legal-action-selection pressures.A hierarchical precondition BN is proposed as a natural way to address both pressures, with longer-horizon loco-manipulation tasks suggested for future comparison.Source: §5 · pp. 8–9