PaperlayerView exact revision on arXiv

Source-linked AI summary

PRICE the Action Chunks: Physical Relational Credit Assignment for Embodied Reinforcement Learning

Yangang Zou, Jiajun Lu, Weitao Zhou, Haibao Yu, Bozhou Zhang, Jiawei Wang, Honglong Tian, Minglei Li, Li Zhang

arXiv:2609.38890v1cs.RO

TL;DR

Outcome-based RL gives every action chunk the same trajectory-level advantage, leaving local progress unresolved, while finer-grained approaches often require evaluators or task-specific supervision. PRICE estimates success potentials at physically corresponding chunk boundaries from current and historical outcomes, then confidence-gates potential changes as local credit. It improves over outcome-based baselines across simulation and real-robot experiments, and independent continuation evaluation finds retained credits aligned with local progress.

  • Problem

    Outcome-based RL assigns the same trajectory-level advantage to every action chunk, leaving unresolved where progress was made or lost in long-horizon manipulation.

    Source: §1 · p. 1

  • Method

    PRICE pools current and historical terminal outcomes at physically corresponding chunk boundaries to estimate success potentials, then confidence-gates their changes as action-chunk credit.

    Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 6

  • Results

    Across LIBERO, RoboTwin 2.0, and real robots, PRICE improves over outcome-based baselines; on six RoboTwin 2.0 tasks, average success is 90.0% versus 31.9% for OpenVLA-OFT (SFT).

    Source: §5.2 · p. 8, §6 · p. 11

  • Takeaways & Limitations

    Independent continuation evaluation shows that retained PRICE credits align with local progress measured by changes in node-averaged success.

    Source: §1 · p. 2

    The oracle-alignment analysis assumes task and initial-context distributions, environment transitions, and terminal-outcome laws do not depend on the policy parameters.

    Source: §4 · p. 5

Abstract

from arXiv · show

Outcome-based reinforcement learning (RL) post-trains vision--language--action policies using terminal success signals, but assigns the same trajectory-level advantage to every action chunk. A failed episode can thus penalize useful early actions as if they caused the failure. Existing approaches seek finer-grained feedback through learned evaluators, adding task-specific supervision or additional model training. We explore, for the first time to our knowledge, whether physical relations across trajectories can provide action-chunk credit in embodied RL from terminal outcomes alone, without an auxiliary evaluator. The key insight is that rollouts reaching corresponding physical situations can serve as references for one another: their terminal outcomes provide evidence for assessing local progress. We introduce Physical Relations for Inferring Credit from Episodes(PRICE), with two components: (i) a physical relational graph that pools current and historical outcomes at corresponding chunk boundaries to estimate success potentials; and (ii) confidence-gated credit assignment that uses changes in these potentials to refine trajectory-level supervision. Our analysis connects oracle potential changes to the terminal-success objective and provides a finite-sample directional bound for outcome-independent evidence pools. Independent continuation tests show that PRICE's retained credits align with local progress, while experiments on LIBERO, RoboTwin 2.0, and real robots demonstrate improved task success over outcome-based baselines and faster learning.

1 INTRODUCTION

Outcome-based RL supplies terminal feedback but leaves unresolved which action chunks made progress or caused failure. PRICE uses physical correspondences across rollouts to infer local credit from terminal outcomes, without an auxiliary evaluator.

1 INTRODUCTION

  • Trajectory-level advantages assign the same credit to every chunk, so a failed episode can penalize useful actions as if they caused the failure.A robot may grasp and transport an object successfully, then fail during placement.Source: §1 · p. 1
  • Existing finer-grained approaches use designed process rewards or learned evaluators, motivating credit assignment from terminal outcomes alone.PRICE addresses this question without an auxiliary evaluator.Source: §1 · pp. 1–2
  • Rollouts that share physical situations can use one another’s terminal outcomes as evidence about which intermediate situations favor eventual success.The correspondence challenge is to distinguish task-relevant physical conditions, such as object pose or contact, from mere visual similarity.Source: §1 · p. 2
  • PRICE compares physically corresponding situations across rollouts, pools current and historical outcomes into success potentials, and confidence-gates potential changes as action-chunk credit.Its analysis connects oracle potential changes to the terminal-success objective and gives a finite-sample directional bound for outcome-independent evidence pools.Source: §1 · p. 2
  • PRICE outperforms evaluator-free baselines, learns faster than SimpleVLA-RL, remains competitive with SRPO, and its retained credits align with independently measured local progress.Controlled ablations identify contributions from historical evidence and confidence gating; benefits also extend to real-robot manipulation.Source: §1 · p. 2

2 RELATED WORK

Prior embodied-RL work uses terminal outcomes for post-training, while finer-grained feedback has relied on task-specific supervision, learned evaluators, or rollout-level comparisons. Cross-trajectory methods motivate local credit, but embodied control requires reliable physical correspondence across continuous observations.

2.1 RL POST-TRAINING FOR EMBODIED POLICIES

  • Outcome-based VLA post-training uses binary success signals, but typically assigns the same episode result to every decision and does not identify where success probability changed.Other post-training work spans policy architectures and optimization frameworks; Q-Chunking instead uses multi-step temporal-difference backups for sparse-reward manipulation.Source: §2.1 · p. 3

2.2 FINE-GRAINED FEEDBACK IN EMBODIED REINFORCEMENT LEARNING

  • Finer-feedback approaches use task-dependent process rewards, learned reward or value models, world models, or task-stage predicates; SRPO compares failures with a success set at rollout level.These approaches leave open whether grouped rollouts’ terminal outcomes can provide action-chunk credit without task-stage predicates or an auxiliary evaluator.Source: §2.2 · p. 3

2.3 CREDIT ASSIGNMENT FROM CROSS-TRAJECTORY RELATIONS

  • Cross-trajectory credit has been studied through return redistribution, potential changes, repeated states, shared edges, and neighboring-rollout value estimates.These studies show that cross-trajectory relations can provide local credit without predefined progress labels.Source: §2.3 · p. 3
  • Embodied relational credit depends on reliable physical correspondence: visually similar frames can differ in contact or pose, distorting node values and their differences.PRICE compares endpoint success potentials across trajectories and gates credits using endpoint differences and evidence counts with a Hoeffding-style uncertainty threshold.Source: §2.3 · p. 4

3 PRELIMINARIES

The preliminaries define post-training as maximizing expected terminal success from policy-generated trajectories. They then describe outcome-based GRPO, which normalizes terminal outcomes within groups but gives every chunk in a trajectory the same advantage.

Problem setup

  • The post-training objective is expected terminal success, with each task trajectory receiving only a binary terminal-success indicator.The setup represents policy input as history-derived and pads early termination with absorbing histories and policy-independent dummy actions for analysis.Source: §3 · p. 4

Outcome-based GRPO

  • Outcome-based GRPO samples K trajectories from the same initial context under a frozen collection policy and assigns group-normalized terminal advantages.When all outcomes in a group agree, every advantage is zero.Source: §3 · p. 4
  • The clipped GRPO surrogate operates over valid action tokens, but every chunk in a trajectory receives the same advantage, leaving its local contribution unresolved.The token count normalizes over valid tokens, and the objective may include a reference-policy KL penalty.Source: §3 · p. 4

4 METHOD

PRICE estimates local progress by comparing success potentials at physically matched action-chunk boundaries, then adds confidence-gated potential changes to the trajectory-level GRPO advantage. Its graph pools terminal outcomes across current and eligible historical rollouts, and its oracle analysis connects potential changes to the terminal-success objective.

4 METHOD

  • Oracle potential changes recover the gradient of the terminal-success objective in expectation when used to weight policy scores.PRICE uses confidence gating to select potential differences relative to their estimated uncertainty.Source: §4 · p. 5

4.1 PHYSICAL RELATIONAL GRAPH CONSTRUCTION

  • PRICE represents action-chunk boundaries as physical interaction descriptors, matches them to graph nodes, and links consecutive nodes with directed action-chunk edges.The descriptor combines visual observations and proprioception; node matching uses cosine similarity to prototypes and a threshold.Source: §4.1 · pp. 5–6
  • Node success potentials pool terminal outcomes from trajectories visiting the same node, counting each trajectory once at its first visit and excluding the query trajectory from current evidence.Historical summaries add evidence only when their cumulative adjacent-policy KL remains within the reuse limit.Source: §4.1 · p. 6

4.2 CONFIDENCE-GATED ACTION-CHUNK CREDIT

  • PRICE assigns candidate chunk credit from the difference between destination and source potentials, retaining it only when the propagated uncertainty interval excludes zero.A wrong nonzero credit requires endpoint estimation error to exceed the gate radius; the final chunk retains the base GRPO advantage.Source: §4.2 · pp. 6–7

4.3 OPTIMIZATION OBJECTIVE

  • The optimization objective adds confidence-gated local credit, scaled by λ, to the trajectory-level GRPO advantage for valid action tokens.The method specifies assigning the resulting advantage to every valid action token in each chunk.Source: §4.3 · p. 7

5 EXPERIMENTS

Experiments evaluate PRICE across simulated and real-robot tasks, finding strong benchmark performance, faster training, and locally aligned credits. Analyses test the roles of historical evidence and confidence gating.

5.1 EXPERIMENTAL SETUP

  • Evaluation spans LIBERO suites, six RoboTwin 2.0 tasks, and two HiFi-UMI real-robot tasks; measures include task success, training steps to target success, and independent MC credit metrics.The main simulated results report mean success rates over three independent training runs.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • On LIBERO, PRICE reaches 99.0% average success versus SRPO’s 99.2%; on six RoboTwin 2.0 tasks, its 90.0% average exceeds SFT initialization’s 31.9% by 58.1 percentage points.On LIBERO-Long, PRICE reaches 97.6%, exceeding the strongest evaluator-free baseline by 3.6 percentage points.Source: §5.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
  • At 90% success, PRICE reduces training steps by 34.5% on Goal and 37.1% on Spatial, with GPU-hour savings of 38.3% and 32.8%, respectively.PRICE also reaches both 90% and 95% success with fewer steps and lower GPU-hour costs than SimpleVLA-RL; its average success is comparable to SRPO under similar budgets.Source: §5.2 · p. 8, Figure 3 · p. 9

5.3 REAL-WORLD EXPERIMENTS

  • PRICE-AWR reaches 55.0% success on Remote Insertion and 67.5% on Produce Sorting, exceeding GRPO-AWR by 12.5 and 15.0 percentage points, respectively.Macro-average success is 61.25% for PRICE-AWR, 47.5% for GRPO-AWR, and 41.25% for initialization.Source: Figure 4 · p. 9, §5.3 · p. 9

5.4 ANALYSIS OF PRICE

  • On LIBERO-Spatial, retained PRICE credits agree in sign with 97.5% of non-tied independent reference differences, with RankCorr 0.709 and AlignedGap +28.47 pp.The independent reference estimates progress by continuing a frozen policy from saved chunk-source and destination evidence pools.Source: §5.4 · pp. 9–10, Figure 5 · p. 10
  • Historical evidence yields distinct endpoint potentials when rollout groups share outcomes, while confidence gating raises DirAcc from 77.2% before gating to 97.5% among retained credits.The comparison evaluates pre-gate contrasts with and without history at fixed nodes and then evaluates retained and rejected credits against independent MC progress.Source: §5.4 · p. 10, Figure 5 · p. 10
  • Removing historical evidence lowers average success from 98.3% to 94.0%, while removing the gate lowers it to 96.9%.With current-group evidence alone, no credit passes the gate under these settings, so the update reduces to GRPO; without gating, uncertain directions enter policy updates.Source: §5.4 · p. 10, Table 3 · p. 10

6 CONCLUSION

The conclusion presents PRICE as a way to refine trajectory-level supervision using pooled outcome evidence and confidence gating, and reports theoretical and empirical support. The endpoint-value analysis states the assumptions under which its policy-gradient argument is developed.

6 CONCLUSION

  • PRICE derives action-chunk credit from terminal outcomes through physical relations across trajectories, using current and historical evidence with confidence gating.The conclusion also reports gains over outcome-based baselines on LIBERO, RoboTwin 2.0, and real robots, plus improved simulated training efficiency and local-progress alignment.Source: §6 · p. 11

A ENDPOINT VALUE CHANGES AND THE TERMINAL OBJECTIVE

B POOLED ENDPOINT CONCENTRATION

The pooled endpoint concentration analysis gives sample-size requirements for resolving potential contrasts and explains why evidence from a single small rollout group may be insufficient under the default gate.

B POOLED ENDPOINT CONCENTRATION

  • The concentration setup fixes endpoint-pool membership and sample size independently of terminal labels and assumes conditional Bernoulli outcomes.Source: B POOLED ENDPOINT CONCENTRATION · p. 15
  • Contrasts of 1.0, 0.4, and 0.3 require at least 7, 42, and 73 supporting trajectories per endpoint, respectively, under the stated settings.The settings use δedge = 0.15.Source: B POOLED ENDPOINT CONCENTRATION · p. 16
  • With K = 8, within-group evidence cannot yield nonzero credit under the default gate: passing endpoints require seven peers and then have identical means.Without historical outcome summaries, the augmented advantage reduces to the GRPO advantage.Source: B POOLED ENDPOINT CONCENTRATION · p. 16

C POOLED FINITE-SAMPLE EDGE DIRECTION

The section states a finite-sample direction guarantee for fixed outcome-independent evidence pools, then describes factors that affect how pooled evidence estimates current-policy progress and how PRICE training and ablations are configured.

C POOLED FINITE-SAMPLE EDGE DIRECTION

  • For fixed endpoint pools meeting the stated conditional Bernoulli assumptions, the true edge contrast lies in its interval with probability at least 1 − δedge.When the interval excludes zero, an accepted nonzero credit has the same direction as the empirical and true contrasts; the incorrect-credit probability is at most δedge.Source: C POOLED FINITE-SAMPLE EDGE DIRECTION · p. 16

C.1 FROM POOLED EVIDENCE TO CURRENT-POLICY PROGRESS

  • The estimation error separates into sampling error, collection-policy mismatch, and state-abstraction error.Physical matching and the historical KL screen restrict contributing histories, and these choices together with the sampling interval govern local-credit reliability.Source: §C.1 · p. 17

D.1 PRICE TRAINING ALGORITHM

  • Each PRICE round computes credit from a frozen archive, updates the policy, and then commits evidence from raw rollout groups for the next round.The final chunk receives zero local credit, and advantages are not renormalized after credit addition.Source: §D.1 · pp. 17–18

D.2 TRAINING SETTINGS

  • Table 4 covers simulation training settings and PRICE credit parameters, while reporting real-robot AWR weights separately.Source: Table 4 · p. 18, §D.2 · p. 18
  • The ablations remove historical evidence by setting Chist = 0, remove gating by using raw potential differences, or set λ = 0 for GRPO.Source: §D.2 · p. 18

D.3 CREDIT EVALUATION PROTOCOL

The section specifies the protocols for auditing PRICE credits, comparing history and gating choices, and measuring alignment with independent continuation estimates.

D.3 CREDIT EVALUATION PROTOCOL

  • The main credit audit uses a frozen OpenVLA-OFT policy on LIBERO-Spatial, with five rollout rounds, 320 groups, and 20,480 valid non-terminal chunks.It evaluates 160 chunks across 16 windows selected before observing MC outcomes.Source: §D.3 · p. 18
  • The controlled heatmaps use locked windows and endpoint assignments; panel (d) uses current-group peer outcomes, while panel (e) pools potentials before gating.Panel (b) shows the gated result; the training history ablation retains the gate.Source: §D.3 · pp. 18–19
  • Independent continuation evaluation restores states from Figure 5(e)’s evidence pools and continues the same frozen policy to estimate endpoint success rates.The main audit uses 12,695 continuations across 9,878 states in 159 endpoint contexts, with equal weight per state within each endpoint pool.Source: §D.3 · p. 19
  • DirAcc measures sign agreement on non-tied comparisons, RankCorr measures rank agreement, and AlignedGap averages credit-aligned MC differences.The evaluation protocol includes MC ties in RankCorr and AlignedGap.Source: §D.3 · p. 19
  • The online archive audit samples 400 supported chunks per run at rounds 16, 32, and 63, then estimates retained-credit progress using 512 fresh continuations per endpoint pool.It audits three PRICE runs on LIBERO-Spatial.Source: §D.3 · p. 19

E.1 ROBOT OBSERVATIONS AND TASK DEFINITIONS

The section defines the real-robot tasks and offline adaptation setup, then describes the weighted flow-matching objective and reports training-cost savings at success milestones.

E.1 ROBOT OBSERVATIONS AND TASK DEFINITIONS

  • Remote Insertion places a remote in a storage box, while Produce Sorting requires placing both specified items in their category trays; the policy receives camera images and 20-dimensional proprioception.Action prediction horizons are 50 for Remote Insertion and 20 for Produce Sorting.Source: §E.1 · p. 19

E.2 OFFLINE ADAPTATION PROTOCOL

  • Offline adaptation uses 60 task-specific interaction trajectories from the initialized policy as a fixed dataset of terminal outcomes and physical relations, with no additional online rollouts during optimization.The initialization is an OpenPI-π0.5 checkpoint post-trained with 400 HiFi-UMI trajectories.Source: §E.2 · p. 20

E.3 OFFLINE RELATIONAL CREDIT AND WEIGHTED FLOW MATCHING

  • PRICE-AWR adds action-chunk relation credit to the trajectory-level terminal advantage before mapping it to a bounded AWR weight; GRPO-AWR uses only the trajectory-level advantage.Both methods use the same flow-matching objective, and the offline relational contrasts are fixed during adaptation.Source: §E.2 · p. 20, §E.3 · pp. 20–21

F.2 TRAINING EFFICIENCY

  • 272.8 GPU hours on Goal and 221.2 on Spatial at 90% success represent 38.3% and 32.8% lower costs than SimpleVLA-RL, respectively.At 95%, the reductions are 17.3% and 23.8%.Source: Table 6 · p. 21, §F.2 · p. 21

F.1 RUN-TO-RUN VARIATION IN MAIN RESULTS

  • Main PRICE results report mean success rates and sample standard deviations across three independent training runs, with SD computed from each run’s average success rate.The supplied passage describes the reporting procedure but does not provide the table’s individual values.Source: §F.1 · p. 21

F.3 MECHANISM ANALYSIS

PRICE credits align with independent local-progress measures, and controlled comparisons show that confidence gating selects more reliable credits. Credit quality is robust across descriptor choices, and relational credit also improves results with Fast-WAM on LIBERO-10.

F.3 MECHANISM ANALYSIS

  • Retained credits exceed 94% DirAcc online; at equal coverage of 45 candidates, confidence gating achieves 97.5% DirAcc and +28.5 pp AlignedGap.Online alignment remains positive across all three training stages.Source: §F.3 · pp. 21–22, Table 7 · p. 22, Table 8 · p. 22
  • Retained candidates show +28.47 pp AlignedGap versus +3.81 pp for rejected candidates, and 73.6% of credited chunks fall in interaction-related intervals.The interaction-related intervals include approach, contact or manipulation, proximity to the target, and object movement.Source: §F.3 · p. 22, Figure 6 · p. 23

F.4 DESCRIPTOR ROBUSTNESS

  • Fusion yields 93.9% DirAcc, 0.758 RankCorr, and +33.7 pp AlignedGap; vision has the highest coverage at 4.2%.All three tested descriptors achieve DirAcc above 87% and positive AlignedGap.Source: §F.4 · p. 22, Table 9 · p. 23

F.5 HYPERPARAMETER SENSITIVITY

  • λ=0.2 reaches 90% success in 22 steps versus 29 for λ=0.4, while their 63-step success rates are similar at 97.0% and 96.9%.The results support λ=0.2 as the default for faster learning at comparable final success.Source: Table 10 · p. 23, §F.5 · p. 23
  • Stricter δedge settings retain 25 or 30 credits at 100% DirAcc, while η=0.93 yields 777 credited chunks across 484 nodes.At δedge=0.30, 46 credits are retained with 95.1% DirAcc and +27.3 pp AlignedGap; higher tested matching thresholds increase archive size while reducing coverage.Source: §F.5 · p. 23, Table 11 · p. 24, Table 12 · p. 24

G EXTENSION TO WORLD ACTION MODELS

  • PRICE reaches 97.2% success on LIBERO-10 with Fast-WAM, versus 92.8% for matched GRPO, an improvement of 4.4 percentage points.The comparison shares the initial checkpoint, rollout grouping, optimizer settings, training budget, and evaluation protocol.Source: G EXTENSION TO WORLD ACTION MODELS · p. 24, Table 13 · p. 25

H.1 SIMULATED MANIPULATION

  • LIBERO-Spatial examples show successful bowl transfers from three starting locations and an unsuccessful transfer that reaches the plate region without completing the task.The figure presents chronological frames from four recorded OpenVLA-OFT rollouts.Source: §H.1 · p. 25, Figure 7 · p. 26

H.2 REAL-ROBOT MANIPULATION

  • Additional robot episodes show failed insertion leaving the remote near the box and failed sorting leaving the fruit held near its tray after the vegetable placement.The successful sorting example shows both placements, extending the fruit-only view in the main figure.Source: Figure 8 · p. 27, §H.2 · p. 27

I LIMITATIONS

PRICE’s credit quality depends on correspondence between physical states and on historical outcomes remaining representative of the current policy; coverage and learning efficiency also have limits.

I LIMITATIONS

  • Credit quality depends on descriptors preserving success-relevant distinctions and historical outcomes remaining representative under the current policy.The paper frames this as a trade-off between evidence reuse, collection-policy mismatch, and state-abstraction error.Source: I LIMITATIONS · p. 28
  • Local credit requires repeated visits to corresponding states, limiting coverage early in training or when trajectories have little overlap.The paper identifies improved exploration and evidence reuse as ways that could expand coverage.Source: I LIMITATIONS · p. 28
  • As success approaches saturation, further gains beyond 90–95% require additional interaction, while uniform terminal outcomes reduce new evidence diversity.The paper suggests sampling difficult tasks or initial conditions more frequently to focus interaction on remaining failures.Source: I LIMITATIONS · p. 28

Related papers

Loading 2609.38890v1…