PaperlayerView exact revision on arXiv

Source-linked AI summary

PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

Hang Yu, James Staley, Cheng Xi Tsou, Xiujin Liu, Wenchang Gao, Jindan Huang, Shijie Fang, Zhegong Shangguan, Angelo Cangelosi, Reuben Aronson, Elaine Short

arXiv:2609.31855v1cs.ROcs.AI

TL;DR

Human demonstrations and feedback each offer useful but limited guidance for reward learning. PHIRL aligns rewards inferred from demonstrations with progress feedback, and significantly outperforms IRL baselines with only 20% of demonstrations annotated offline.

  • Problem

    Demonstrations can lack local precision, while feedback is sparse; preference feedback can also be uninformative when all sampled trajectories are undesirable.

    Source: §I · p. 1

  • Method

    PHIRL iteratively infers rewards from demonstrations and aligns them with progress annotations over four dimensions.

    Source: §I · p. 1

  • Results

    Across two simulated robot tasks and one real robot task, PHIRL significantly outperformed IRL baselines with only 20% of demonstrations annotated offline.

    Source: §I · p. 1

  • Takeaways & Limitations

    Annotations can be collected before or during learning, and VLM-generated progress could enhance PHIRL’s converged performance.

    Source: §VI · p. 7

    PHIRL assumes a finite horizon or complete status, making it unsuitable for infinite-horizon tasks or cyclic observations.

    Source: §V · p. 7

Abstract

from arXiv · show

Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.

I. INTRODUCTION

PHIRL combines demonstrations with progress feedback to learn robust rewards while reducing human engagement, addressing limitations in reward learning from demonstrations and feedback alone.

I. INTRODUCTION

  • Demonstrations offer dense policy guidance but can lack local precision, while human feedback is accurate yet sparse and high-quality demonstrations are difficult to provide.Preference feedback can be uninformative when every sampled trajectory is undesirable, as when a robot never grasps an object.Source: §I · p. 1
  • PHIRL iteratively learns rewards from demonstrations and aligns them with cumulative task-completion progress across temporal delta, magnitudes, potential, and completion.The framework can also use a fine-tuned vision-language model to provide online progress feedback.Source: §I · p. 1
  • PHIRL significantly outperforms IRL baselines across simulated and real robot tasks with only 20% of demonstrations annotated offline.Its shaped rewards were also resilient to two reward-hacking behaviors in qualitative analysis.Source: §I · p. 1

II. BACKGROUND

  • Reward learning from demonstrations can support transfer and rapid re-optimization, but learning an accurate reward from demonstrations alone is challenging and often requires high-quality examples.Combining demonstrations and feedback has been reported to improve learning efficiency and reduce demonstration-quality requirements.Source: §I · p. 1, §II · p. 2
  • Prior work combines demonstrations and feedback, but PHIRL aligns rewards with progress across multiple dimensions without assuming demonstrated behaviors always contribute to task completion.Its progress annotations can be collected before learning, reducing human-in-the-loop effort.Source: §II · p. 2

III. METHODOLOGY

PHIRL’s methodology centers on making rewards inferred from demonstrations consistent with feedback on those same demonstrations.

III. METHODOLOGY

  • The method aims to learn robust reward functions by combining human demonstrations with human feedback.Source: §III · p. 2
  • PHIRL aligns feedback with rewards inferred from the same demonstrations.Source: §III · p. 2
  • The guiding insight is that inferred rewards should be consistent with feedback on the demonstrations used to infer them.Source: §III · p. 2

A. Progress

PHIRL uses progress as feedback for reward alignment and learns an initial reward model from demonstrations with AIRL.

A. Progress

  • Progress is numerical feedback describing cumulative task completion, from fully incomplete to fully complete.In this work, labels range from 0 to 100 and are based only on the current observation.Source: A. Progress · pp. 2–3

B. Reward Learning from Demonstrations

  • The IRL objective is to infer a reward function from human demonstrations, interpreting the demonstrations as generated by an optimal policy for maximum-likelihood learning.Source: B. Reward Learning from Demonstrations · p. 3
  • PHIRL uses AIRL to learn the initial reward distribution, with the discriminator formed from fθ and the policy.The method selects AIRL because it infers state-only reward and value functions, while progress is path-independent.Source: B. Reward Learning from Demonstrations · p. 3

C. Reward Function Shaping

PHIRL shapes learned rewards to reflect four properties of progress: temporal change, change magnitude, potential, and task completion.

C. Reward Function Shaping

  • Positive progress changes should correspond to positive incremental rewards, encouraging transitions toward task completion.The temporal-delta alignment uses a strictly increasing monotonic hinge function.Source: C. Reward Function Shaping · p. 3
  • Larger progress improvements should receive larger rewards because steps differ in how much they advance the task.The passage contrasts picking up the target object with merely moving closer to it.Source: C. Reward Function Shaping · p. 3
  • States with higher progress should have higher learned potential; PHIRL sets the potential function Φ(s) to V(s).The potential-based shaping signal is F(st, at, st+1) = γΦ(st+1) − Φ(st).Source: C. Reward Function Shaping · p. 3
  • A successful demonstration should have greater total reward than a failed one, regardless of demonstration horizon.In this work, a demonstration ending below progress 90 is classified as unsuccessful.Source: C. Reward Function Shaping · p. 3

D. PHIRL

PHIRL alternates AIRL reward learning with progress alignment, minimizing a sum of shaping components over demonstrations and an annotated subset. Its framework can extend beyond AIRL, including methods without a potential component.

D. PHIRL

  • PHIRL alternates AIRL reward updates with alignment to progress labels on an annotated subset of demonstrations.The reward and potential are updated by minimizing the combined alignment loss.Source: D. PHIRL · p. 4
  • The alignment objective combines four shaping components, with the completion-shaping weight set to 1/|D| and the other three weights set to 1.Offline PHIRL collects demonstrations and annotations before learning begins.Source: D. PHIRL · p. 4
  • The learning loop samples demonstration and policy trajectories, updates the AIRL discriminator, then uses labeled trajectories to update the reward and potential against progress.VLM-generated progress labels can optionally be added to the annotated dataset during training.Source: D. PHIRL · p. 4
  • PHIRL is not inherently tied to AIRL; methods without a potential component can use the remaining three alignments.The framework’s central idea is to shape learned rewards to align with evaluation information specified by progress.Source: D. PHIRL · p. 4

IV. EXPERIMENTS AND RESULTS

The experiments evaluate PHIRL against four baselines on one real-robot and two Robosuite tasks. They also assess whether its learned rewards resist reward hacking in two representative scenarios.

IV. EXPERIMENTS AND RESULTS

  • The evaluation uses four baselines across one real-robot task and two Robosuite tasks to test PHIRL’s performance and claimed contributions.The Robosuite tasks are Lift and PickPlaceCan; the real-robot task is animal-brushing.Source: §IV · p. 4
  • The experiments also examine whether PHIRL’s learned reward functions are robust to reward hacking in two representative scenarios.Source: §IV · p. 4
  • The section frames the experiments as validation of PHIRL’s claimed contributions.Source: §IV · p. 4

A. Setups

The evaluation combines two Robosuite simulation domains with a real-robot animal-brushing task, using high- and mixed-quality demonstrations where specified. Four reward-learning baselines are compared with PHIRL, which uses 20% annotated demonstrations.

A. Setups

  • The study evaluates Lift and PickPlaceCan in Robosuite and animal-brushing with a Kinova Gen 3 Lite arm.The real-robot goal is to brush pom-poms off a stuffed animal.Source: A. Setups · p. 4
  • Each Robosuite high-quality dataset contains 200 demonstrations from one proficient operator; each mixed-quality dataset contains 300 from six operators.The six mixed-quality operators comprised two each rated worse, okay, and better.Source: A. Setups · p. 4
  • The real-robot mixed-quality dataset contains 60 demonstrations from six participants, two each rated worse, okay, and better.Source: A. Setups · p. 4
  • PHIRL is compared with AIRL, Pref, AIRL-Pref, and Rank2Reward, and the standard PHIRL model uses 20% annotated demonstrations.PPO serves as the adversarial learner and trajectory generator.Source: A. Setups · p. 4
  • Training runs for 500 episodes with 10 parallel environments and a horizon of 256; hyperparameters are selected by search.Source: A. Setups · p. 4
  • Progress annotation assigns one 0–100 label to each of 10 sub-trajectories per demonstration, and the VLM uses fine-tuned Gemma 3 12B.For preference feedback, annotators select the preferred sub-trajectory from a pair.Source: A. Setups · p. 4

B. PHIRL Outperforms Baselines

PHIRL outperforms baselines in simulation and on the real robot, with larger advantages in the more challenging PickPlaceCan and mixed-quality settings. Online progress labels further improve performance, while reward-hacking scenarios illustrate differences in the rewards assigned to undesirable behavior.

B. PHIRL Outperforms Baselines

  • 91.3% higher returns than the best baseline in PickPlaceCan, with 2.3 times more return rewards in Can-MH, where demonstration quality is lower.In Lift, PHIRL achieved mean environmental rewards of 188.12 in PH and 167.16 in MH, respectively 5.6% and 17.1% above the best baseline.Source: Figure 2 · p. 5, B. PHIRL Outperforms Baselines · p. 5
  • Fig. 2 compares performance and ablations in Lift and PickPlaceCan; among shaping components, reward shaping is most effective overall, whereas completion shaping is most critical when demonstrations include failures.Source: Figure 2 · p. 5
  • PHIRL outperformed AIRL (39), AIRL-Pref (62), Pref (58), and Rank2Reward (54) on the real-robot mixed-quality demonstrations.The task measured pom-poms brushed off the stuffed animal; PHIRL-online achieved the highest figure result.Source: B. PHIRL Outperforms Baselines · pp. 5–6, Figure 3 · p. 5
  • In the Lift-Drop Loop, AIRL rewards repeated lifting and dropping, whereas PHIRL punishes dropping and rewards lifting.In Touch-And-Stay, PHIRL gives a negative but near-zero reward for poking without picking and placing, indicating no progress.Source: Figure 4 · p. 6
  • Online progress labels improved Lift-PH performance from M = 188.12 to M = 205.08 and real-robot results from 73 to 84.The Lift-PH difference was statistically significant (p < 0.05).Source: B. PHIRL Outperforms Baselines · p. 6
  • In Can-MH, online labels increased returns from M = 116.99 to M = 147.28 (p < 0.001).The passage reports a stronger impact of online labels in noisier demonstrations or harder tasks.Source: B. PHIRL Outperforms Baselines · p. 6

C. Ablation Study and Sensitivity Analysis

PHIRL’s reward-shaping components generally contribute to performance, and increasing annotation coverage generally improves results. Annotation gains are larger for lower-quality datasets and usually saturate at 50% coverage.

Sensitivity

  • Increasing annotation coverage generally improves PHIRL performance, with gains more significant on lower-quality datasets and usually saturated by 50% coverage.Standard PHIRL uses 20% of available annotations.Source: C. Ablation Study and Sensitivity… · p. 6

Ablation

  • Potential shaping is generally influential across conditions, while other reward-shaping ablations also significantly decrease performance.The completion-shaping term is the exception in most conditions, where demonstrations are marked fully completed (progress > 90 for the last frame).Source: C. Ablation Study and Sensitivity… · p. 6

Additional results

  • Further details, statistical analyses, learning curves, success rates, annotation examples, and additional simulation domains are available in the appendix.The paper points readers to the PHIRL appendix at github.com/PHIRL2026/PHIRL.Source: C. Ablation Study and Sensitivity… · p. 6

D. PHIRL Learns More Robust Reward Functions

PHIRL’s reward functions resist two reward-hacking behaviors, and annotations generated by an unfine-tuned VLM can improve learning in most cases, though less than real human feedback. The discussion also identifies limits in annotation sampling, task scope, and feedback modality.

D. PHIRL Learns More Robust Reward Functions

  • PHIRL rewards repeated reaching and lifting but punishes dropping in Lift-Drop-Loop, while PHIRL and AIRL both negatively reward poking in Touch-And-Stay.AIRL’s reward allows repeated lifting and dropping to increase cumulative reward; PHIRL gives poking a small negative reward because it is not harmful to task completion.Source: D. PHIRL Learns More Robust Reward… · p. 6

V. DISCUSSION AND LIMITATION

  • The study randomly sampled demonstrations for annotation rather than preferentially selecting lower-quality examples, and PHIRL currently relies on progress feedback.The authors expect selecting lower-quality demonstrations to improve sample efficiency, especially with limited feedback labels, and propose blending progress with language instructions and structured rubrics.Source: §V · p. 7
  • Unfine-tuned Gemma 3 12B annotations improved PHIRL learning in most cases, but performance remained below real human feedback.The formative study used ReWiND-style language instructions to specify task subgoals and annotate demonstrations; generated-annotation gains were less significant and saturated faster.Source: §V · p. 7, Figure 5 · p. 7
  • PHIRL was evaluated in one training run, assumes finite-horizon tasks or a complete status, and is unsuitable for infinite-horizon or cyclic observations.The authors report that repeating every condition with multiple seeds was beyond their budget.Source: §V · p. 7

VI. CONCLUSIONS

PHIRL alternately infers rewards and aligns them with progress annotations. The paper reports baseline outperformance with 20% of demonstrations annotated, alongside improved performance from VLM-generated progress and robustness in two reward-hacking scenarios.

VI. CONCLUSIONS

  • PHIRL alternately infers a reward function and aligns it with progress annotations over annotated demonstrations.Annotations can be collected before or during learning, allowing varying levels of human participation.Source: §VI · p. 7
  • PHIRL significantly outperforms baselines with 20% of demonstrations annotated before learning.Source: §VI · p. 7
  • VLM-generated progress enhanced PHIRL’s converged performance, and PHIRL showed robust performance in Lift-Drop-Loop and Touch-And-Stay.Source: §VI · p. 7

Related papers

Loading 2609.31855v1…