PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang, Bing Zheng, Zhaolin Wang, Yuanwei Liu

arXiv:2610.32431v1cs.AI

TL;DR

A failed VLA rollout can penalize recovery actions along with errors under trajectory-level GRPO. MART-GRPO uses a VLM to select event-guided failure-feedback boundaries, and improves success rates over Full GRPO across multiple tasks.

  • Problem

    In GRPO, a failed rollout’s negative advantage can penalize recovery attempts as well as the error, although not every action contributed to failure.

    Source: §1 · p. 1

  • Method

    MART-GRPO uses a fine-tuned, frozen VLM to identify the last visible error without observed recovery in complete rollouts and truncate subsequent failure feedback.

    Source: §1 · p. 2

  • Results

    MART-GRPO improves success rates over Full GRPO across multiple tasks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    At ideal stops, truncation preserves the reference group-gradient expectation without increasing covariance.

    Source: §1 · p. 2

    Retrospective event boundaries can change estimator expectation; the ideal projection guarantee requires ηB = 0, beyond visible-event localization.

    Source: §2.4 · p. 5, §3.3 · p. 7

Abstract

from arXiv · show

A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.

1 INTRODUCTION

The introduction frames how trajectory-level GRPO can penalize recovery actions alongside errors, and proposes martingale-guided event truncation. It reports higher mean success than Full GRPO in seven-task OpenVLA-OFT comparisons on RoboTwin 2.0.

1 INTRODUCTION

  • GRPO applies trajectory-level advantages across action chunks, so a failed rollout can give an error and subsequent recovery attempts the same negative advantage.Source: §1 · p. 1, Figure 1 · p. 2
  • Ideal stops use remaining conditional credit to preserve the reference group-gradient expectation without increasing covariance; retrospective boundaries are analyzed through residual and bias–variance effects.Source: §1 · p. 2
  • MART-GRPO uses a fine-tuned, frozen VLM to localize the last visible error without observed recovery in complete rollouts and mask later failure feedback.Successful-rollout loss terms, group advantages, and original loss normalization are preserved.Source: §1 · p. 2
  • Seven-task OpenVLA-OFT comparisons on RoboTwin 2.0 show higher mean success than Full GRPO.Source: §1 · p. 2
  • Two-task ablations favor event-guided truncation over uniform penalty scaling and random prefixes.Source: §1 · p. 3

2 A MARTINGALE ANALYSIS OF FAILURE FEEDBACK

The analysis characterizes when truncating failure feedback preserves a GRPO group-gradient estimator’s expectation without increasing covariance. It then examines how retrospective boundary selection can shift expectation and trade off against variance reduction.

2.3 CONDITIONAL PROJECTION OF A COMPLETE GRPO GROUP

  • Ideal stopping preserves the reference GRPO group-gradient estimator’s expectation and does not increase covariance when the removed contribution has zero conditional expectation given retained information.The theorem accommodates dependence among group advantages when coefficients and token counts are measurable from the retained information.Source: §2 · p. 3, §2.3 · p. 4

2.2 REMAINING CONDITIONAL CREDIT

  • Remaining conditional credit is the expected policy-gradient contribution of future actions given observed history, and it equals continuation-success sensitivity.Contributions can cancel, so zero credit does not require every later action advantage to vanish.Source: §2 · p. 3, §2.2 · p. 3

2.1 GRPO WITH A FEEDBACK MASK

  • The GRPO feedback mask changes the temporal support of a trajectory-level advantage while retaining the original valid-token denominator.At collection parameters, the full-feedback objective weights each chunk by AiZi,k/Ni.Source: §2.1 · p. 3

2.2 REMAINING CONDITIONAL CREDIT

  • An action inside a failed rollout can still carry positive local credit because its effect on success probability can be positive.The local action advantage records how an action moved the success probability, distinct from the group coefficient Ai.Source: §2.2 · p. 3

2.3 CONDITIONAL PROJECTION OF A COMPLETE GRPO GROUP

  • If conditional success probability is zero at a stopping time, that failure-certifying stop has zero remaining credit and preserves the ideal projection result.For stops with zero remaining credit, earlier stopping has no larger covariance for the same coefficients and denominators; certification is sufficient but not necessary.Source: §2.3 · p. 4

2.4 FROM IDEAL STOPS TO RETROSPECTIVE BOUNDARIES

  • For retrospective boundaries, strict MSE improvement requires variance reduction ΔV to exceed n times the squared expectation shift.Complete-rollout selection can change expectation because the selected boundary may convey information beyond the retained prefix.Source: §2.4 · pp. 4–5, Figure 2 · p. 5

3 MARTINGALE GRPO

MART-GRPO uses a frozen VLM to identify unrecovered visible errors in complete failed rollouts and mask failure feedback after a selected boundary. It preserves successful-rollout terms and the original token normalization, while retrospective boundary choice is analyzed separately from ideal stopping guarantees.

3 MARTINGALE GRPO

  • A frozen VLM localizes the last visible error without observed recovery in complete failed rollouts, and its interval’s right endpoint sets the feedback boundary.Feedback is retained through that boundary and the failed-rollout suffix is masked.Source: §3 · p. 6, Figure 3 · p. 6

3.1 LEARNING TO LOCALIZE FAILURE EVENTS

  • The event-localization targets combine synchronized camera views, proprioception, task instructions, frame alignment, visible outcomes, error types, intervals, supporting frames, and recovery labels.The annotated corpus contains 1,024 failed rollout videos, with training, validation, and test partitions disjoint by task–scene identity.Source: §3.1 · p. 6
  • A LoRA adapter is trained with the Qwen3.8-27B vision-language backbone frozen; the validation-selected checkpoint is then frozen during RL.Selection uses validation task-macro interval IoU, with response loss and earlier update as tie-breakers.Source: §3.1 · pp. 6–7

3.2 SELECTING THE FEEDBACK BOUNDARY

  • Truncation requires a predicted failure, specific error type, non-null interval, and explicit no-recovery report; otherwise semantic fallback retains full feedback.For qualifying intervals [ℓ_i, u_i], feedback is retained through B_i = u_i, including every candidate onset chunk.Source: §3.2 · p. 7

3.3 POLICY OPTIMIZATION

  • Masks apply to new failed rollouts, while successes retain all loss terms and the original valid-token counts remain in the PPO update.For retrospective VLM boundaries, the ideal projection guarantee additionally requires η_B = 0.Source: §3.3 · p. 7

4 EVALUATION

The evaluation measures VLM event localization, seven-task policy performance, and feedback ablations on Hammer and Handover. MART-GRPO improves mean success over Full GRPO at update 40, while localization scores and two-task ablations assess event selection and feedback placement.

4 EVALUATION

  • The policy comparison uses matched task-specific SFT starts, 64 accepted groups of eight rollouts per update, and greedy evaluation on seen and held-out seeds.The primary endpoint is update 40, with evaluations every five updates.Source: §4 · p. 7, §4.2 · p. 8

4.1 VLM EVENT LOCALIZATION

  • 72.1% task-macro IoU, 68.1% exact endpoint agreement, and 91.8% within-one-chunk agreement are reported for the selected VLM checkpoint.The checkpoint is validation-selected; localization scores average within tasks and then across tasks.Source: Figure 4 · p. 8, §4.1 · p. 8

4.2 VLA POLICY PERFORMANCE

  • 65.01% to 68.64% (+3.63 points) mean success at update 40: MART-GRPO exceeds Full GRPO across seven RoboTwin 2.0 tasks.Six tasks improve; held-out-seed gains are 6.03 points, compared with 1.23 points on seen seeds.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 FEEDBACK ABLATIONS

  • MART-GRPO exceeds uniform scaling by 3.52 points on Hammer and 3.12 on Handover, and random prefixes trail it by 9.38 and 2.34 points, respectively.The two-task ablations preserve success weights and original token denominators.Source: Table 2 · p. 9, §4.3 · p. 9

5 CONCLUSION AND LIMITATIONS

The paper characterizes ideal failure-feedback stops through remaining conditional credit and uses MART-GRPO to select event-guided boundaries. It concludes that visible events may not track remaining credit reliably as the policy evolves.

5 CONCLUSION AND LIMITATIONS

  • Policy evaluations show success-rate improvements over Full GRPO across multiple tasks; VLM-guided truncation also beats uniform penalty scaling and random prefix selection on both ablation tasks.The authors identify how reliably visible events track remaining credit across policy stages as an open question.Source: §5 · p. 9
  • The conclusion states that future work will test how reliably visible events track remaining credit across policy stages and develop prefix-based event predictors.Source: §5 · p. 9

AI USE STATEMENT

  • The authors report AI assistance with implementation, planning, analysis, language polishing, figure preparation, and proof checks, while taking responsibility for the manuscript.Source: AI USE STATEMENT · p. 10

A RELATED WORK

  • Related methods reconstruct stage-relative advantages, allocate computation by phase divergence, use matched corrective targets, or mask updates with importance-ratio triggers.MART-GRPO instead selects which existing trajectory-level advantage reaches actions, while retaining the policy architecture and environment reward.Source: A RELATED WORK · pp. 14–15

B.1 NOTATION AND REGULARITY

  • The analysis assumes a fixed policy, finite horizon, binary terminal outcome, valid differentiation under expectation, and square-integrable scores.Continuous-action policies are assumed to have differentiable densities with common support; terminal padding uses parameter-independent zero-score dummy actions.Source: §B.1 · p. 15

B.2 STANDARD MARTINGALE IDENTITIES

  • The success probability conditioned on the analytical history is a bounded martingale, and its increments connect action-conditioned advantage to local action credit.The policy's observable input may be only part of the analytical history used for conditioning.Source: §B.1 · p. 15, §B.2 · pp. 15–16

B.3 REMAINING CONDITIONAL CREDIT

  • Remaining conditional credit is the conditional covariance between terminal success and the remaining score sum, so later action credits can cancel.At the initial time, conditioning on the scene gives a gradient whose average is the population gradient.Source: §B.3 · pp. 16–17

B.4 PROOF OF PROPOSITION 1

For ideal local stopping rules, the retained record makes the truncated score a conditional projection of the full score. The group result extends this projection to dependent GRPO advantages under stated information and normalization conditions.

B.4 PROOF OF PROPOSITION 1

  • When the stopped history is retained on failure and the complete trajectory on success, zero conditional mean of the removed score makes the truncated score a conditional projection.The proposition's necessity result concerns the unweighted score; unconditional expectation preservation alone can arise through conditional shifts canceling across histories.Source: §B.4 · p. 17

B.5 PROOF OF THEOREM 2

  • For GRPO groups, retaining every outcome makes group advantages measurable, allowing the group-gradient projection despite dependent advantages.The result requires measurable token denominators; a future-dependent failure count adds information that requires recentering the removed score.Source: §B.5 · p. 18

B.6 PROOF OF COROLLARY 3

A stop at zero conditional success probability is an ideal failure stop. Among ideal rules with matching coefficients and denominators, earlier stops have no greater covariance and can attain minimum covariance under the stated measurability condition.

B.6 PROOF OF COROLLARY 3

  • When conditional success probability is zero before the horizon, failure is already determined and the remaining conditional credit is zero.At the horizon the suffix is empty, so the theorem's projection result applies to this stop.Source: §B.6 · p. 18

B.7 ORDERING IDEAL STOPPING RULES

  • Among ideal stopping rules with the same coefficients and denominators, an earlier stop yields no greater covariance; the earliest zero-credit stop attains minimum covariance when denominators are measurable in its retained information.Zero remaining credit can also occur before the outcome is determined, as in the paper's independent-fair-coin example.Source: §B.6 · p. 18, §B.7 · p. 19

B.8 PROOF OF PROPOSITION 4

For retrospective boundaries, the paper measures departure from the ideal conditional projection and analyzes how bias and variance trade off under truncation. It also compares complete-video boundaries with prefix rules and considers partial suffix removal.

B.8 PROOF OF PROPOSITION 4

  • A retrospective boundary's projection residual measures its departure from the ideal conditional-projection property.The bound applies under a fixed unfiltered or accepted-group law.Source: §B.8 · p. 19

B.9 EXACT TRUNCATION TRADEOFF

  • Under ideal projection, the removed contribution has zero mean and is uncorrelated with the retained estimate, so truncation reduces variance; for a general rule, MSE improvement depends on bias, variance, and batch size.For nonzero fixed expectation shift d and positive variance reduction ΔV, strict MSE improvement holds when n < ΔV / ||d||².Source: §B.9 · p. 20

B.10 APPROXIMATE STOPPING AND CONDITIONAL BIAS

  • For adapted early stops, the conditional expectation shift from failure-only suffix removal is bounded using remaining score variation and outcome uncertainty.The bound is zero at the horizon and at most Lv√η on the early-stop event under the stated conditional norm and success-probability conditions.Source: §B.10 · pp. 20–21

B.11 COMPARING RETROSPECTIVE AND PREFIX BOUNDARIES

  • Complete-video and prefix-rule removed contributions agree unless at least one failed trajectory has differing boundaries; the hindsight decomposition bounds their expectation and variance differences.The complete-video rule selects the endpoint of the last unrecovered error, whereas the prefix rule stops at the first qualifying observed error.Source: §B.11 · pp. 21–22

B.12 PARTIAL SUFFIX REMOVAL

  • Under exact projection, complete suffix removal minimizes MSE within the continuous family of suffix-removal strengths.The family varies suffix removal while preserving the prefix weight, successful-rollout terms, and original denominator.Source: §B.12 · p. 22

C EXAMPLES CLARIFYING THE PROJECTION CONDITIONS

The examples distinguish low remaining credit from uncertain success and show why boundary selection must respect observed history. They also demonstrate that cancellation across scenes can hide a failure of conditional projection, while rare-success masking can increase MSE.

Rare success and expectation shift

Zero credit with uncertain outcomes

  • 1/16 to 1/32: with uncertain success but zero remaining credit, masking failures preserves zero expectation and halves variance.At p = b = 1/2, the full and masked estimator variances are 1/16 and 1/32, respectively.Source: C EXAMPLES CLARIFYING THE PROJECTION… · p. 23

Future-dependent boundary selection

  • A boundary chosen using an unobserved future action shifts expectation from zero to 1/16, despite zero remaining credit at each fixed candidate time.The selected failure-only mask removes an event with probability 1/4 and contribution −1/4.Source: C EXAMPLES CLARIFYING THE PROJECTION… · p. 23

Cancellation across initial scenes

  • Equal global expectations across scenes can mask failed conditional projection: on failure, the retained scene identifies G = ±1/4 while the masked estimator is zero.Averaging across scenes gives both estimators expectation zero and variances 1/16 and 1/32.Source: C EXAMPLES CLARIFYING THE PROJECTION… · p. 24

D POLICY TRAINING AND EVALUATION

The evaluation uses OpenVLA-OFT on seven RoboTwin 2.0 tasks, with task-specific SFT checkpoints and shared training settings for the RL methods. The section also describes feedback normalization, event-model inputs and boundary mapping, and the annotated video corpus.

D POLICY TRAINING AND EVALUATION

  • RoboTwin 2.0 evaluation covers seven randomized-scene tasks spanning tool use, handover, lifting, placement, and stacking, with binary terminal rewards.Rollouts end at task success or when the execution budget is exhausted.Source: D POLICY TRAINING AND EVALUATION · p. 24

D.1 TRAINING AND EVALUATION SETTINGS

  • Both RL methods start from the same task-specific SFT checkpoint; updates use 64 accepted groups of eight rollouts and a global PPO minibatch of 128.The scheduled Full GRPO evaluations run from update 0 to 40, with update 0 supplying the SFT evaluation.Source: §D.1 · p. 24

D.2 FEEDBACK ACCOUNTING AND NORMALIZATION

  • Chunk scores sum token scores, and the unclipped full-feedback objective weights each chunk by AiZi,k/Ni using the original valid-token denominator.The implementation retains the valid-token mask and denominator when suffix advantages are set to zero; earlier termination reduces the valid-token count before truncation.Source: §D.2 · p. 24

E.1 VLM INPUTS, PROMPTING, AND BOUNDARY MAPPING

  • The event model reviews completed-rollout video and robot observations; for a localized failure with no observed recovery, the upper error-interval endpoint sets the feedback boundary.Successful rollouts bypass event inference and retain all loss terms; uncertain outcomes, unclear errors, missing localization, or reported or uncertain recovery retain full feedback.Source: §E.1 · pp. 25–27

E.2 ANNOTATED VIDEO CORPUS

  • The annotated corpus contains 1,151 videos, split by task–scene identity with zero identity overlap across partitions and no held-out policy-evaluation scene seeds.Annotations include outcome, error type and interval, observed recovery, and evidence frames; null intervals are allowed when an error cannot be localized.Source: §E.2 · p. 27

E.3 EVENT MODEL TRAINING

The event adapter is selected using validation localization metrics, and test diagnostics measure agreement with reference intervals on eligible failures. The examples show predictions matching the reference interval across later checkpoints.

E.3 EVENT MODEL TRAINING

  • 78.2% validation task-macro IoU and the lowest response NLL both occur at update 304 (epoch 8), the retained checkpoint.Four checks without the required IoU improvement trigger stopping at update 437 (11.5 epochs).Source: §E.3 · p. 28
  • Test localization diagnostics compare intervals with reference labels on 220 localizable, unrecovered test failures across the seven policy-comparison tasks.Eligibility comes from reference annotations independently of model predictions; the reported checkpoint is selected by validation IoU.Source: §E.3 · p. 28, Table 8 · p. 29
  • At epochs 6 and 8, both predicted endpoints match the reference interval [4, 5] for the Handover rollout.The base model predicts [6, 8], while epoch 3 predicts [5, 5].Source: §E.3 · p. 28

F POLICY RESULTS AND FEEDBACK ABLATIONS

F.1 SEVEN-TASK POLICY RESULTS

The section reports seven-task policy comparisons at update 40 and each run’s best scheduled score, alongside the evaluation protocol and event-mask feedback statistics.

F.1 SEVEN-TASK POLICY RESULTS

  • Table 1 reports seven-task policy comparisons at update 40 and each run’s best scheduled score.The event model is the validation-selected epoch-8 adapter from update 304.Source: §F.1 · p. 29
  • The fixed-endpoint evaluation uses 128 seen-seed and 128 held-out-seed episodes with greedy decoding; best scores are maxima over evaluations at updates 5 through 40.The SFT column uses the Full-GRPO run’s initial evaluation, and separate simulator executions can yield different initial success rates.Source: §F.1 · pp. 29–30
  • Table 10 summarizes feedback retained by event masks, pooling event records and loss-mask coefficients across all 40 MART updates.The retained coefficient mass is the ratio of post-mask to pre-mask summed mass; success weights and valid-token denominators are preserved.Source: §F.1 · pp. 29–30

F.2 FEEDBACK ABLATIONS

The ablations compare event-guided truncation with uniform scaling and random-prefix feedback on Hammer and Handover. Under similar feedback-weight budgets, event-guided truncation achieves higher success on both tasks.

F.2 FEEDBACK ABLATIONS

  • Uniform scaling matches the aggregate coefficient ratio, while random prefix samples accepted-failure boundary indices and intersects each prefix with valid rollout tokens.The frozen scaling factors are γ = 0.4920678764 for Hammer and γ = 0.3268904598 for Handover; both ablations preserve success weights, original token denominators, and terminal rewards, and use no VLM inference.Source: §F.2 · pp. 30–31
  • Table 10 reports feedback-mask statistics pooled across 40 MART updates, including accepted failed trajectories, shortened boundaries, retained negative coefficient mass, and full-feedback fallbacks.These statistics quantify loss weights.Source: Table 10 · p. 31
  • Both ablations use the same SFT checkpoints, 64 groups of eight rollouts per update, global PPO minibatch 128, and update-40 evaluation on seen and held-out seeds.Each evaluation split contains 128 episodes; ablation statistics are frozen before training, and realized coefficient-mass ratios are measured on each run’s own rollout groups.Source: §F.2 · p. 31
  • At update 40, uniform scaling scores 199/256 on Hammer and 104/256 on Handover; random prefix scores 184/256 and 106/256, respectively.For uniform scaling, seen-seed/held-out-seed scores are 100/128 and 99/128 for Hammer, and 50/128 and 54/128 for Handover; for random prefix they are 96/128 and 88/128, and 46/128 and 60/128.Source: §F.2 · p. 31
  • Under similar feedback-weight budgets, event-guided truncation achieves higher success than uniform scaling and random-prefix feedback on both Hammer and Handover.Figure 9 reports update-40 success alongside each run’s retained negative coefficient mass pooled over 40 updates.Source: Figure 9 · p. 31, §F.2 · p. 31
Loading 2610.32431v1…