PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

Christophe D. Hounwanou, John Emeka Eze, Yaé U. Gaba

arXiv:2608.18008v1cs.LGcs.AI

TL;DR

Hybrid LLM-RL agents raise open questions about whether added task structure improves learning over pure RL. This paper formalizes LLM progress scores as bounded potentials and shows that potential-based shaping preserves optimal policies despite inaccurate scores, with numerical verification on a small MDP.

  • Problem

    Whether hybrid LLM-RL agents learn faster or achieve better sample efficiency and final performance than pure RL remains open.

  • Method

    The paper models the hybrid system on an augmented state space and applies bounded-potential reward shaping to LLM-derived per-state progress scores.

  • Results

    Potential-based shaping preserves the optimal policy set under inaccurate LLM scores and is numerically verified across four configurations, including one scaled 20× the base reward magnitude.

  • Takeaways & Limitations

    Under boundedness, LLM feedback can provide policy-invariant shaping rather than changing which policies are optimal.

  • Takeaways & Limitations

    The empirical validation is scoped and does not settle comparative benchmark performance, while full-scale evaluation remains future work.

Abstract

from arXiv · show

Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.

1 Introduction

The paper separates hybrid RL–LLM soundness from empirical learning speed, proving that bounded-potential shaping preserves optimal policies without requiring accurate LLM scores. It validates this guarantee numerically while leaving large-scale performance comparisons for future work.

  • Motivation: The architecture combines an LLM planner’s task decomposition with an RL controller’s low-level execution under actual dynamics.Pure RL faces sparse rewards and long horizons, while pure LLM planning lacks action-level dynamics knowledge, recovery ability, and calibrated scene-state uncertainty.
  • Motivation: The paper distinguishes whether the hybrid converges to the right policy from whether it learns faster than pure RL in practice.The first is a fixed-point/soundness question, whereas the second is an empirical convergence-rate question requiring large-scale experiments.
  • Theoretical contribution: Using the LLM’s per-state progress score as a bounded potential leaves the augmented MDP’s set of optimal policies unchanged, even when scores are inaccurate.This guarantee is stronger than those available from more general LLM-as-reward approaches such as Text2Reward and Eureka.
  • Numerical verification: The guarantee was verified on a 3-state MDP under four potential configurations, including one scaled to 20× the base reward magnitude in absolute value.The numerical test is reported as Section 4.4’s verification of the policy-preservation result.
  • Scope and limitations: The paper does not claim benchmark superiority over published baselines; its claim is that the theoretical fixed point is correct by construction if the hybrid works empirically.The pipeline validation is not presented as evidence of superiority, and a larger comparative study is planned as follow-up work.

2 Related work

Related work spans LLM-based embodied-agent planning, language-augmented reinforcement learning, and hierarchical RL with language subgoals. The paper builds on classical potential-based shaping to provide a principled interface between LLM guidance and value-based control.

  • LLM planners for embodied agents: LLM planners support embodied agents through task decomposition, affordance grounding, textual feedback, and related planning frameworks.SayCan combines LLM task decomposition with learned affordance value functions, while Inner Monologue feeds textual environment feedback back into the LLM.
  • LLM-augmented RL and reward from LLM: LLMs have been used to generate training goals, proxy rewards, dense reward code, intrinsic rewards, and online policies for reinforcement learning.Examples include ELLM, Kwon et al., Text2Reward, Eureka, Motif, and GLAM.
  • Hierarchical RL with language subgoals: Hierarchical RL used language-conditioned policy sketches and low-level controllers as high-level abstractions before and during the LLM era.DECKARD extends this paradigm through a Dream/Wake decomposition.
  • Policy-invariant shaping: The paper instantiates the classical potential-based shaping theorem for LLM-derived potentials to preserve optimal policies and guide principled RL design.It argues that RL systems incorporating LLM feedback should adopt this construction or explicitly justify deviations from it.

3 Preliminaries

The section formalizes the hybrid planner–controller system as a Goal-Augmented MDP and defines LLM-derived potential-based reward shaping. It establishes the interfaces, assumptions, and policy-invariance basis used later.

  • MDP formulation: The underlying model is a discounted infinite-horizon MDP with states, actions, transition kernel P, rewards R, initial distribution ρ0, and policies inducing returns.
  • Planner and subgoals: A task string and state caption feed a planner that emits a finite ordered list of natural-language subgoals.The planner may be stochastic, with output distribution P(ℓ, ϕ(s)).
  • Planner and subgoals: The Done-oracle determines whether a subgoal is satisfied and may be implemented by a symbolic checker, learned classifier, or LLM prompt.The analysis allows the oracle to be imperfect and considers mismatches between planner language and completion semantics.
  • Goal-Augmented MDP: The Goal-Augmented MDP augments each environment state with the current subgoal, while transitions update environment and subgoal components through P, the planner, or Done.Its reward matches the environment reward unless shaping is applied, and its initial distribution samples an initial planner-generated subgoal.
  • Potential-based shaping: LLM scoring defines a potential Φ over state–subgoal pairs, typically normalized to [0, 1], and adds αFΦ to the environment reward.Under standard boundedness assumptions, classical potential-based shaping preserves the underlying MDP’s set of optimal policies; the paper instantiates Φ with LLM feedback.

4 The Goal-Augmented MDP framework … 4.4 Potential-based reward shaping from LLM feedback

The framework augments the MDP with natural-language subgoals, uses an LLM planner and goal-conditioned RL controller, and adds bounded-potential shaping. This shaping preserves the optimal policy set despite inaccurate LLM scores, while potentially affecting learning speed.

  • 4 The Goal-Augmented MDP framework: The framework routes states and captions to an LLM planner, combines active subgoals with RL actions, and adds potential-based shaping to environment rewards.The shaping term is F = γΦ(s′, g) − Φ(s, g), with plan revisions triggered periodically.
  • 4.1 Goal-augmented MDP: A subgoal is a finite-length natural-language string, and the LLM planner’s output distribution determines the subgoal grammar G.The framework does not require a formal specification language.
  • 4.1 Goal-augmented MDP: The goal-augmented MDP uses augmented states (s, g), transitions combining environment dynamics with a subgoal transition kernel, and rewards R + F.The subgoal kernel is induced by a caller-specified completion oracle Done : S × G → {0, 1}.
  • 4.2 LLM as high-level planner: The LLM samples a plan of at most Kmax subgoals conditioned on the task and initial-state caption, while the active pointer advances when the current subgoal is completed.Completion can use environment event flags, a learned MLP classifier, or the LLM itself; this choice is a design axis.
  • 4.3 Low-level policy: The low-level RL policy conditions on both state and active subgoal, encoding the subgoal with a frozen MiniLM-L6-v2 sentence embedder before PPO training with GAE.The default subgoal embedding dimension is dg = 384.
  • 4.4 Potential-based reward shaping from LLM feedback: The LLM produces a bounded score sLLM(s, g) ∈ [0, 1], which defines the potential used for shaping.The potential is applied within the goal-augmented MDP framework.
  • 4.4 Potential-based reward shaping from LLM feedback: Every optimal policy of the goal-augmented MDP projects to an optimal policy of the original MDP for any measurable subgoal-scheduling rule, even when the bounded potential is inaccurate.The guarantee follows because the shaping term telescopes, leaving only a vanishing tail when γ < 1.
  • 4.4 Potential-based reward shaping from LLM feedback: The LLM score can be arbitrarily wrong at individual states without changing optimal policies; a poor potential can slow learning but cannot corrupt the fixed point.The numerical verification also reports policy preservation for four potential configurations, including values 10–20× the base reward magnitude.

4.5 Replanning and full algorithm · 4.6 Reference framework implementation · 4.7 Empirical validation

The framework defines replanning triggers, provides a reference implementation aligned with its formalism, and validates the pipeline through planner-isolation and CPU-scale MiniGrid studies. The reported evaluations characterize planner behavior and implementation correctness rather than testing performance against published baselines.

  • 4.5 Replanning and full algorithm: Replanning occurs every H environment steps or when Done fails to fire within the per-subgoal budget B.Algorithm 1 specifies the full inference procedure.
  • 4.6 Reference framework implementation: The reference implementation provides typed Planner, PotentialShaper, SubgoalScheduler, and ReplanTrigger interfaces, with direct correspondence to the Section 4 formalism and Algorithm 1.It maps each formal definition to a module.
  • 4.6 Reference framework implementation: The implementation supports Ollama-served Qwen and Llama models, pluggable cloud backends, and three Done-oracle variants.The Done oracles use environment events, a learned MLP, or an LLM completion prompt.
  • 4.6 Reference framework implementation: The package includes 26 unit tests, YAML configurations for standard ablations, and CPU-runnable end-to-end validation.GPU is required only for scale, not validation.
  • 4.7 Empirical validation: The full 8-baseline study on BabyAI, ALFWorld, and Crafter remains planned follow-up work requiring GPU and cloud-LLM compute.The paper instead presents two smaller-scale validations focused on framework operation and planner behavior in isolation.
  • 4.7.1 Planner-in-isolation audit: Qwen-2.5:14b produced parseable output on every task and covered just over half of ground-truth subgoals on average across 20 MiniGrid tasks.A match required at least 50% of ground-truth content tokens to appear in an LLM subgoal, with coverage measured in order.
  • 4.7.1 Planner-in-isolation audit: Plans averaged 8.0 subgoals versus approximately 3.6 in the ground truth, with roughly five extraneous subgoals per plan.The extraneous steps were often verbose lower-level phrasings such as turning to face a wall instead of going to the key.
  • 4.7.2 Pipeline validation on MiniGrid-DoorKey-6x6: The pipeline validation used PPO and a hybrid subgoal-conditioned PPO configuration on MiniGrid-DoorKey-6x6-v0 for 30,000 environment steps per seed across three seeds.The hybrid used locally served Qwen-2.5:14b, cached the initial plan, replanned periodically at H = 1000, disabled failure triggering with B = 0, and disabled shaping with c = 0.

5 Results

The pilot validates an end-to-end CPU implementation but shows no hybrid advantage at the tested budget. The central failure is a plan-vocabulary mismatch that prevents subgoal progression and leaves the scheduler stuck on its first subgoal.

  • Pipeline validation: The pilot does not show a hybrid advantage: PPO’s higher IQM is driven by one high-variance seed, and confidence intervals overlap.The comparison is reported for MiniGrid-DoorKey-6x6 with PPO and a Qwen-2.5:14b hybrid agent, each run with n = 3 seeds.
  • Failure diagnosis: The scheduler remained on the first subgoal for each episode, leaving the RL policy conditioned on a stale string.The resulting subgoal-conditioning signal collapsed at the source, matching the anticipated systematic plan-vocabulary mismatch failure mode.
  • What the pilot validates: The reference implementation ran end-to-end on CPU without exceptions across 90,000 environment steps.MiniLM-L6-v2 subgoal encoding integrated cleanly with PPO, while mean episode length remained within the pure-PPO baseline’s confidence interval.
  • What the pilot validates: The plan-cache design limited LLM inference to approximately 30 calls per seed, providing a concrete cost calibration.The broader pilot used 92 Qwen-2.5:14b calls and 4,199 output tokens across all runs.

6 Discussion

Potential-based shaping preserves the augmented MDP’s optimal-policy set, but not convergence speed or automatically all off-policy estimation settings. The framework therefore recommends potential constructions while highlighting boundedness, subgoal-detection, and replanning-cost failure modes.

  • Theoretical scope: The theorem preserves the augmented MDP’s optimal-policy set, but makes no convergence-rate claim.A bad potential can slow learning arbitrarily or direct exploration away from productive states.
  • Theoretical scope: Shaped-reward consistency at training and target time is required for the invariance argument to transfer to bootstrapped value estimation.The reference implementation injects F before the rollout collector reads the reward.
  • Design implication: General non-potential dense rewards, including Text2Reward and Eureka, are not covered because inaccurate LLM outputs can change the optimal policy.The recommendation is to use the potential construction or justify the need for a general dense form.
  • Failure modes: Out-of-range or malformed LLM scores above ∼1% can violate strict potential-based shaping despite prompting for [0, 1] and clamping.The paper recommends monitoring this bounded-Φ requirement.
  • Failure modes: Subgoal-detection failures can stall the scheduler and eliminate the subgoal channel’s signal.LLMDone or a learned classifier is suggested when LLM plan grammar drifts.
  • Failure modes: Failure-triggered replanning on every subgoal makes LLM cost per episode grow linearly with horizon instead of remaining a small constant.Instrument the LLM call count per episode.

7 Limitations •

The empirical validation is scoped and does not settle comparative benchmarking. The guarantee also depends on bounded, well-defined LLM potentials, while prompt design affects potential values and may permit gaming.

  • 7 Limitations: The empirical validation demonstrates end-to-end operation, planner isolation, and one integration failure, but does not settle comparative benchmarking.Full-scale evaluation is planned follow-up work.
  • 7 Limitations: Potential-based shaping requires a bounded, well-defined Φ; unbounded or improperly noisy LLM scores invalidate the proof.The guarantee depends on the LLM score satisfying boundedness.
  • 7 Limitations: The scoring prompt affects the distribution of Φ values and can be gamed by prompt-sensitive LLMs.The exact prompt shape is a design choice and represents a prompt-engineering surface rather than a theoretical concern.

8 Conclusion

The paper formalizes LLM-planner/RL-controller systems as Goal-Augmented MDPs and establishes that potential-based LLM reward shaping preserves optimal policies despite inaccurate state-level scores. Numerical, CPU pipeline, and MiniGrid validation support the framework while identifying one integration failure involving Done-oracle vocabulary mismatch.

  • The architecture is formalized as a Goal-Augmented MDP.
  • Potential-based shaping from LLM feedback preserves the set of optimal policies even when state-level LLM scores are inaccurate.
  • Numerical verification on a small MDP and an end-to-end CPU reference pipeline validate the framework.
  • MiniGrid-DoorKey-6x6 validation confirms the specified behavior but reveals a Done-oracle vocabulary mismatch with LLM-generated plans.

Reproducibility statement

The paper provides deterministic, fully specified procedures and repository materials enabling reproduction of all numerical results across its verification, audit, and validation experiments.

  • All numerical results are reproducible from deterministic, fully specified procedures in the accompanying repository.The repository covers Proposition 4.1 verification, the planner-in-isolation audit, and MiniGrid-DoorKey-6x6 pipeline validation.
  • Each experiment reports CPU runtime, LLM-call counts, raw outputs, logs, per-seed results, confidence intervals, and aggregation steps.The reported aggregation procedures include IQM and bootstrap.

A Prompt templates

The reference implementation uses separate prompts for planning, progress scoring, completion decisions, and failure-triggered replanning. These templates constrain outputs to structured formats, including bounded scores, binary completion tokens, and capped subgoal lists.

  • A Prompt templates: The appendix provides verbatim prompt templates for the reference implementation.The templates cover planning, potential-function scoring, completion-oracle decisions, and failure-triggered replanning.
  • P1 - Planning prompt: The planning prompt asks for at most 12 subgoals, each a short imperative English clause of at most 32 tokens.It identifies the agent’s environment and requires completing a natural-language task through sequential subgoals.
  • P1 - Planning prompt: The planning prompt supplies the task string and current-state caption before requesting the plan.The user template ends with an explicit “Output:” field.
  • P2 - Scoring prompt (potential function Φ): The scoring prompt instructs the estimator to output only one number in [0, 1] representing progress toward a subgoal.The system prompt defines this score as an estimate of closeness to completing the subgoal from the current state.
  • P3 - Completion oracle prompt (LLM-as-Done): The completion oracle must output exactly one token: DONE when the subgoal is satisfied and CONTINUE otherwise.Its user template provides the subgoal and current-state caption before the answer field.
  • P4 - Failure-triggered replanning prompt: When a subgoal exceeds its step budget, the replanning prompt requests a revised plan of at most 12 remaining subgoals.The input includes the original task, current state, failed subgoal, and budget B in steps.
Loading 2608.18008v1…