PaperlayerView exact revision on arXiv

Source-linked AI summary

Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

Yang chen, Yitan Zhang, Michael Witbrock, Shuyue Hu

arXiv:2609.38955v1cs.LGcs.AI

TL;DR

IRL seeks rewards that explain expert demonstrations, but conventional methods often rely on computationally burdensome and unstable reward-policy loops. LFIRL uses diffusion-policy scores to recover values and rewards in sequential offline stages. Across benchmark tasks, it reports 2-3x efficiency gains in most settings while matching or surpassing strong baselines in reward recovery.

  • Problem

    Many IRL methods alternate reward learning and policy optimization because they cannot extract reward-supervision signals directly from demonstrations, creating computational burden and training instability.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    LFIRL uses diffusion-policy action scores to recover soft Q and value functions, calibrates them with a state-dependent offset, and extracts rewards through Bellman consistency in a sequential offline pipeline.

    Source: §1 · p. 2, §4 · p. 4, §4.2 · p. 6

  • Results

    LFIRL achieves 2-3x training-time efficiency gains in most settings while matching or surpassing strong online and offline baselines in reward recovery.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Diffusion-derived local action-score information supports staged reward recovery without the conventional reward-policy loop.

    Source: §1 · p. 2

    The state-dependent offset characterization assumes a soft-optimal expert policy and an exact learned diffusion-policy score.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Inverse Reinforcement Learning (IRL) aims to recover a reward function that explains expert demonstrations. Existing IRL methods typically rely on a bi-level optimization procedure that alternates between reward learning and policy optimization, leading to substantial computational burden and training instability. In this work, we introduce a different route that eliminates policy optimization entirely by leveraging diffusion policies. Our key insight is that a diffusion policy encodes the action-gradient structure of the optimal soft Q function, enabling reward learning to be cast as a sequence of value recovery problems, thereby allowing us to bypass reward-policy loops inherent in prior IRL methods. Specifically, our method proceeds in three stages: (I) recovering the optimal soft Q function via action-gradient matching and estimating the corresponding soft value function (LogSumExp of Q values) in a way inspired by Gumbel regression; (II) calibrating these soft values by inferring a state-dependent offset; (III) extracting the reward by enforcing Bellman consistency. This leads to Loop-Free Inverse Reinforcement Learning (LFIRL), a fully offline algorithm that operates in a simple, loop-free, and sequential manner. LFIRL is simple to implement and significantly improves training efficiency while maintaining strong reward recovery performance. Empirically, across Maze, Franka Kitchen, Adroit Hand Pen, and Push-T benchmarks, LFIRL achieves 2-3x speedup over the fastest baselines, while matching or surpassing state-of-the-art methods in reward recovery quality.

1 Introduction

Conventional IRL’s alternating reward and policy updates create computational burden and training instability, motivating the question of whether rewards can be recovered without that loop. LFIRL uses diffusion-policy action scores for sequential value recovery and reports strong reward recovery with faster training.

1 Introduction

  • IRL’s repeated reward-policy updates are computationally expensive and make training harder to stabilize.The paper asks whether a useful reward can be recovered without this loop.Source: §1 · pp. 1–2
  • Diffusion-policy scores encode the action gradient of the optimal soft Q function, allowing Q recovery directly from demonstrations.This avoids relying on learner-policy information from iterated reward-policy updates.Source: §1 · p. 2
  • LFIRL sequentially recovers soft Q and value functions, calibrates them with a state-dependent offset, then extracts reward through Bellman consistency.The pipeline operates offline on static trajectories.Source: §1 · p. 2
  • 2-3x training-time efficiency gains in most settings accompany reward recovery that matches or surpasses strong online and offline baselines.Evaluations cover Maze, Franka Kitchen, Adroit Hand Pen, and Push-T.Source: §1 · p. 2

2 Related Work

Prior IRL and imitation-learning work includes looped and modified reward-policy optimization, as well as diffusion methods used for imitation and reward learning. LFIRL differs by using diffusion-derived local action scores to recover soft values and then rewards in a staged pipeline.

2 Related Work

  • Many adversarial IRL methods, including GAIL [16], AIRL [10], DAC [22], and GAIfO [39], use nested reward-learning and policy-optimization loops.Other approaches modify the loop, including single-loop updates and methods that avoid a separate RL optimization procedure.Source: §2 · pp. 2–3
  • Diffusion has been used for imitation, reward learning, policy optimization, and adversarial imitation, but LFIRL uses it as a local soft-Q score estimator.It then recovers value and reward sequentially rather than using diffusion as the final policy, extracting trajectory-level relative rewards, or embedding diffusion in an adversarial loop.Source: §2 · p. 3

3 Preliminaries

The preliminaries define the MDP and maximum-entropy IRL setting, then connect diffusion-policy scores to action-gradient supervision for soft-Q recovery. The method relies on the expert policy being soft-optimal for that connection.

3 Preliminaries

  • An MDP comprises state and action spaces, absorbing states, a transition kernel, a discount factor, and a reward function.The preliminaries define its state-action density under a policy and transition model.Source: §3 · p. 3

3.1 Maximum-Entropy RL and IRL

  • Generalized maximum-entropy RL adds relative entropy to a reference policy to the expected-reward objective; with a uniform reference, it recovers standard MaxEnt RL up to a constant.Under its optimality condition, the optimal policy and soft Q function have the same action gradient after accounting for the reference policy.Source: §3.1 · p. 3
  • MaxEnt IRL seeks a reward under which the expert policy is optimal, while its min-max structure induces reward updates against learner behavior and learner-policy optimization.The resulting reward-policy loops bring computational burden and instability, although some methods use cheaper single-loop updates.Source: §3.1 · p. 4

3.2 Diffusion Models and Diffusion Policy

  • A diffusion policy’s learned score indicates how to adjust noisy actions toward higher likelihood under the expert action distribution.When the expert is soft-optimal, this score supplies action-gradient supervision for recovering the optimal soft Q function.Source: §3.2 · p. 4

4 Methods

LFIRL recovers rewards in three sequential stages: infer soft Q and value functions, calibrate state-dependent offsets, then enforce Bellman consistency. Its algorithm is fully offline, with each stage run once using frozen outputs from preceding stages.

4.1 Stage I: Recover the Soft Q Function and Value function

  • Diffusion-policy scores supervise action-gradient matching for soft Q, while a margin anchors expert actions above nearby perturbations without globally restricting action diversity.The reference-policy score is corrected during gradient matching, and Sobolev-style training combines function values with derivatives.Source: §4.1 · pp. 4–5
  • A Gumbel-regression-inspired moment condition estimates the soft value without explicitly computing LogSumExp; with Q frozen, the value objective has a unique minimizer.The derivation follows the deterministic LogSumExp characterization under the reference policy rather than a stochastic model for Q values.Source: §4.1 · pp. 5–6

4.2 Stage II: Infer the State-Dependent Offset and Calibrate Soft Values

  • Because action gradients leave state-dependent shifts unidentified, LFIRL learns an offset and regularizes Bellman-implied rewards to keep shifts coherent across transitions.Absorbing states provide additional direct supervision for the offset.Source: §4.2 · p. 6
  • After offset calibration, LFIRL refits the soft value against the calibrated Q function to keep the two functions consistent.The base Q and value estimates are fixed while the offset is updated.Source: §4.2 · p. 6

4.3 Stage III: Recover the Reward Function

  • With calibrated Q and value functions, reward recovery fits a reward network to the Bellman-implied target Q(s_t, a_t) − γV(s_{t+1}) using Bellman residual errors.Reward targets are clipped and normalized for training stability.Source: §4.3 · p. 6

4.4 Algorithm Summary

  • The algorithm runs each stage once on frozen outputs and uses an offline dataset, requiring neither environment interactions nor learned-dynamics rollouts during training.The dataset is collected from the reference policy.Source: §4.4 · p. 7

5 Experiments

The experiments assess reward recovery quality and training efficiency across several environments and evaluation protocols. LFIRL performs competitively across tasks, reduces training time, and remains competitive with fewer demonstrations, though performance declines as demonstrations are reduced.

5.1 Experimental Setup

  • The evaluation covers PointMaze, Franka Kitchen, Adroit Hand Pen, and Push-T, averaging results over five runs with matched environment-step budgets and demonstration counts.PointMaze uses success rate, Kitchen uses completed subtasks, and Pen and Push-T use trajectory-discrimination accuracy.Source: §5.1 · p. 7, Table 1 · p. 8

5.2 Reward Recovery

  • LFIRL supports competitive downstream control and trajectory discrimination, achieving the best or near-best final performance in most settings.PointMaze and Franka Kitchen are assessed through downstream control, while Push-T and Adroit Hand Pen use trajectory-quality discrimination.Source: §5.2 · pp. 7–8

5.3 Average Running Time

  • About 2–3x less runtime than the fastest baseline across tasks, with roughly 2–5x less when loading a pre-trained diffusion policy.Training time is compared under the same step budget.Source: §5.3 · p. 8, Table 2 · p. 9

5.4 Correlation with Ground-Truth Rewards

  • LFIRL achieves the highest PCC on five of six tasks and the highest SCC on five of six tasks.PCC measures linear association and SCC measures rank agreement between recovered and ground-truth rewards; Table 3 reports both on the same state-action samples.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 Ablation Study

  • Removing the state-dependent offset b(s) consistently degrades performance across evaluated tasks.The ablation indicates that offset inference and soft-value calibration matter for recovering a well-aligned value structure.Source: §5.5 · p. 9, Figure 5 · p. 9

5.6 Reduced-Data Experiments

  • Performance decreases as demonstrations are reduced, but LFIRL remains competitive in reduced-data settings.Sparser expert state-action anchors weaken Q recovery and consequently affect downstream value and reward recovery.Source: §5.6 · pp. 9–10, Figure 6 · p. 10

6 Discussion

LFIRL’s time efficiency is attributed to its sequential pipeline, which avoids repeated coupling between reward updates and policy or value optimization. The method uses a DDPM-based diffusion policy for action-gradient supervision; a Flow Matching variant performs more weakly under the same epoch budget.

6 Discussion

  • LFIRL’s speed comes mainly from its linear pipeline, where each stage uses the frozen output of the preceding stage rather than repeating reward-policy coupling.Offline training alone does not ensure speed: offline IRL methods can remain slow when reward updates depend repeatedly on policy or value optimization.Source: §6 · p. 10
  • LFIRL uses a conditional DDPM diffusion policy, but any state-conditioned generative model that provides a reliable score or action-gradient signal could supply Stage I supervision.The paper also instantiates a Flow Matching version.Source: §6 · p. 10
  • Under the same number of training epochs, the Flow Matching variant yields weaker reward recovery than the DDPM-based version.The authors suggest differing convergence behavior may contribute; they note Flow Matching action-generation quality may be more sensitive to integration accuracy, schedule design, and training budget.Source: §6 · p. 10

7 Conclusion

The conclusion presents LFIRL as a staged IRL framework that replaces reward-policy optimization loops with sequential value recovery. Across four benchmark families, it matches or surpasses strong baselines while reducing training time, but accurate recovery depends on sufficient local anchors around expert actions.

7 Conclusion

  • LFIRL removes the conventional reward-policy loop by turning reward recovery into sequential value-recovery problems.The fully offline pipeline recovers Q from diffusion-policy score signals, estimates and calibrates soft values, then enforces Bellman consistency to recover reward.Source: §7 · p. 10
  • Experiments on PointMaze, Franka Kitchen, Adroit Hand Pen, and Push-T show reward recovery that matches or surpasses strong baselines while achieving 2–3x efficiency gains in most settings.Source: §7 · p. 10
  • Q recovery depends on sufficient local anchors around expert actions; sparse anchor coverage may require more demonstrations for accurate reconstruction.Source: §7 · p. 10
  • The conclusion identifies broader generative-model classes and more sample-efficient recovery from limited demonstrations as future directions.Source: §7 · p. 10

Appendices

The appendix derives how generative-policy scores provide local action-gradient supervision for Q recovery, then documents value fitting and a Flow-Matching alternative.

A.1 Derivation of Diffusion Scores and Their Relation to Soft Q-Gradients

  • Near expert actions, the diffusion score minus the reference-policy score supplies action-gradient supervision for recovering the soft Q function.The derivation relies on low-noise score approximation and, for the exact identity, idealized assumptions including a soft-optimal expert policy and exact diffusion score.Source: §A.1 · p. 14

A.3 Derivations for Stage II: Offset Calibration and Value Re-estimation

  • Matching action derivatives recovers Q’s action-dependent geometry but leaves an undetermined state-dependent offset.The appendix establishes this relation on a sampled action neighborhood under idealized assumptions.Source: §A.2 · p. 15, §A.3 · p. 15

A.2 Derivations for Stage I: Q-Recovery and Initial Value Fitting

  • Gumbel-regression-style fitting of the frozen Q estimate produces an initial value estimate for use in Stage II.The value network is trained with gradients only with respect to its parameters while the Q estimate remains fixed.Source: §A.2 · p. 15

B.1 Flow-Matching Policy and Score Estimation

  • Flow Matching can replace DDPM score estimation in Stage I, using scores of intermediate actions near the expert-action endpoint.Its score expression applies for flow times below 1; the endpoint itself has a vanishing denominator.Source: §B.1 · p. 16
  • Figure 7 compares DDPM- and Flow-Matching-based pretraining with the same subsequent reward-recovery pipeline; higher is better.The variants differ in how the pretrained generative policy is obtained.Source: Figure 7 · p. 17

B.2 Additional Results: DDPM vs. Flow Matching Pretraining

With the downstream LFIRL pipeline held fixed, the Flow-Matching variant generally performs worse than DDPM under the same pretraining epochs, suggesting a less effective score signal for Q recovery.

B.2 Additional Results: DDPM vs. Flow Matching Pretraining

  • With the same pretraining epochs, Flow Matching generally performs worse than DDPM, suggesting DDPM provides a more effective score signal for Q recovery.The comparison changes the pretrained generative policy while keeping the remaining LFIRL pipeline unchanged.Source: §B.2 · p. 17

C Sensitivity to the Reference Policy

On MuJoCo tasks, LFIRL’s performance depends on the specified reference policy: task-level results vary by policy, while the method does not infer an unknown reference policy from demonstrations.

C Sensitivity to the Reference Policy

  • Uniform references perform better on Swimmer and Walker2d, while Gaussian references perform better on Hopper; LFIRL is not invariant to the reference policy.The reference-policy variants use the same demonstrations and training configuration; Table 4 reports episodic returns as mean ± standard deviation.Source: C Sensitivity to the Reference Policy · pp. 17–18, Table 4 · p. 18
  • The Gaussian-reference variant remains effective, so recovery is not restricted to the uniform-reference simplification.The comparison evaluates these variants on MuJoCo tasks using expert demonstrations sampled from pretrained SAC experts.Source: C Sensitivity to the Reference Policy · pp. 17–18
  • The experiment assumes a specified reference policy and does not infer it from demonstrations; jointly inferring an unknown reference and reward is beyond scope.The passage notes that demonstrations alone do not uniquely separate an unknown reference policy from the reward without additional assumptions.Source: C Sensitivity to the Reference Policy · p. 18

D Sensitivity to Stage-I Hyperparameters

Stage-I performance depends most on diffusion-noise magnitude and the intended margin, while the anchoring coefficient is comparatively stable once anchoring is sufficient.

D Sensitivity to Stage-I Hyperparameters

  • Performance decreases as diffusion noise increases, with a larger decline at high noise across all three PointMaze environments.The proposed explanation is that noisier actions move farther from the expert-action distribution, weakening the local score approximation.Source: D Sensitivity to Stage-I Hyperparameters · p. 18
  • Performance is comparatively stable for λ = 1 and λ = 2, while λ = 0.5 weakens anchoring and lowers performance, particularly on Large.The authors identify λ = 1 as a stable choice across the three environments.Source: D Sensitivity to Stage-I Hyperparameters · p. 18
  • ξ = 0.5 and ξ = 2 both underperform ξ = 1 on every environment, making the intended margin the most consequential tested parameter.Too small a margin provides insufficient separation; too large a margin may conflict with gradient matching or suppress near-optimal actions.Source: D Sensitivity to Stage-I Hyperparameters · pp. 18–19
  • Noise magnitude and anchoring margin matter more than λ once the value-anchoring coefficient is large enough to provide effective anchoring.These results identify noise magnitude and margin as the more consequential Stage-I settings.Source: D Sensitivity to Stage-I Hyperparameters · p. 19

E.1 Implementation Details and Hyperparameter Setup

The appendix points to the overall LFIRL training procedure and summarizes environment-specific network and hyperparameter settings, expert-demonstration sources, and experimental hardware.

E.1 Implementation Details and Hyperparameter Setup

E.2 Expert Demonstrations Sources

  • Maze and Kitchen use Minari expert trajectories, while Push-T uses demonstrations from the Diffusion Policy dataset.Source: §E.2 · p. 19, Table 8 · p. 19

E.3 Hardware Information

Related papers

Loading 2609.38955v1…