PaperlayerView exact revision on arXiv

Source-linked AI summary

SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models

Senyu Fei, Siyin Wang, Li Ji, Ao Li, Shiduo Zhang, Liming Liu, Jinlong Hou, Jingjing Gong, Xianzhong Zhao, Xipeng Qiu

arXiv:2511.15605v2cs.ROcs.CLcs.CV

TL;DR

VLA models rely heavily on expert demonstrations, and sparse binary rewards make it difficult to learn effective policies from failed trajectories. SRPO uses successful trajectories from the current batch as self-references and latent world representations to reward progress in failures, reaching 99.2% success from a 48.9% supervised baseline in 200 RL steps—a 103% relative improvement.

  • Problem

    VLA models rely heavily on expert demonstrations, while sparse binary rewards in VLA reinforcement learning make effective policy learning difficult.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    SRPO uses successful trajectories from the current batch as self-references and latent world representations to assign progress-wise rewards to failed attempts.

    Source: §1 · p. 2

  • Results

    99.2% success from a 48.9% supervised baseline after 200 reinforcement learning steps, a 103% relative improvement, on LIBERO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Further analysis reports that world-progress reward modeling can be applied to real-world robotic tasks.

    Source: §6 · p. 11

    The method assumes the agent does not directly access the environment state and instead relies on observations.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations, leading to demonstration bias and limiting performance. Reinforcement learning (RL) is a vital post-training strategy to overcome these limits, yet current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity. Relying on binary success indicators wastes valuable information in failed trajectories, resulting in low training efficiency. To solve this, we propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework. SRPO eliminates the need for external demonstrations or manual reward engineering by leveraging the model's own successful trajectories, generated within the current training batch, as a self-reference. This allows us to assign a progress-wise reward to failed attempts. A core innovation is the use of latent world representations to measure behavioral progress robustly. Instead of relying on raw pixels or requiring domain-specific fine-tuning, we utilize the compressed, transferable encodings from a world model's latent space. These representations naturally capture progress patterns across environments, enabling accurate, generalized trajectory comparison. Empirical evaluations on the LIBERO benchmark demonstrate SRPO's efficiency and effectiveness. Starting from a supervised baseline with 48.9% success, SRPO achieves a new state-of-the-art success rate of 99.2% in just 200 RL steps, representing a 103% relative improvement without any extra supervision. Furthermore, SRPO shows substantial robustness, achieving a 167% performance improvement on the LIBERO-Plus benchmark.

1 Introduction

SRPO addresses sparse rewards in VLA reinforcement learning by using successful trajectories from the current batch as references for rewarding failed attempts, comparing behavior through transferable world-model latent representations.

1 Introduction

  • VLA reinforcement learning can improve performance, but sparse outcome rewards waste information in failed trajectories, while process-supervision approaches often rely on expert demonstrations or hand-crafted task decompositions.Multi-turn trajectory rollouts are computationally costly, making inefficient use of failed attempts particularly consequential.Source: §1 · p. 2
  • Latent world-model representations capture transferable behavioral progress patterns, enabling robust trajectory comparison without precise environmental reconstruction or domain-specific training.This avoids the poor cross-domain generalization or extensive task-specific fine-tuning associated with traditional pixel-level world models.Source: §1 · pp. 2–3
  • SRPO uses successful trajectories within each training batch as behavioral references, assigning failed trajectories progress-wise rewards based on their alignment with successful patterns.It favors trajectory-level rewards over fine-grained hand-crafted shaping, which the authors say can steer policies toward suboptimal solutions.Source: §1 · p. 2
  • 99.2% success after 200 RL steps, versus the 48.9% one-demonstration-per-task supervised baseline, is a 103% relative improvement on LIBERO without extra demonstrations or manual reward engineering.The paper reports this as state-of-the-art LIBERO performance.Source: §1 · pp. 2–3
  • 167% performance improvement on LIBERO-Plus demonstrates robustness, and the paper reports reward effectiveness in both simulation and real-world experiments.Source: §1 · p. 3

2 Related Work

Related work increasingly applies RL to VLAs to improve exploration and robustness, but sparse rewards and reliance on demonstrations or task-specific priors limit efficient online learning. SRPO addresses these gaps with self-referential progress rewards from task-agnostic latent world representations.

2.1 Vision-Language-Action Models

  • VLA research is shifting from supervised post-training toward RL to improve exploration and robustness, but sparse rewards limit the use of failure trajectories and training efficiency.Prior work has trained both autoregressive (Tan et al., 2025) and diffusion-based VLAs (Chen et al., 2025a).Source: §2.1 · p. 3

2.2 Reinforcement Learning

  • VLA policies optimized with GRPO and binary success rewards (Li et al., 2025; Zang et al., 2025; Tan et al., 2025; Liu et al., 2025) face sparse feedback, while handcrafted rewards can require expert demonstrations or task-specific priors (Chen et al., 2025b; Lu et al., 2025; Shu et al., 2025).The cited passage identifies those requirements as difficult to scale in online RL settings that encourage autonomous learning.Source: §2.2 · p. 3
  • SRPO uses successful trajectories as self-references and task-agnostic latent world representations to assign progress-wise rewards to failures, using embedding L2 distances for policy optimization.A pretrained world model encodes trajectories; the resulting rewards support advantage estimation and policy optimization under KL regularization (Assran et al., 2025).Source: §2.2 · p. 3, Figure 2 · p. 4

3 Methods

SRPO addresses sparse terminal rewards by using successful trajectories in the current batch as references for world-model-based progress rewards on failures, then optimizing VLA policies with trajectory-level relative advantages. Its formulation operates from partial observations and goal descriptions rather than directly accessible environment states.

3.1 Problem Formulation and Key Components

  • The policy selects actions from the current observation and goal description, while the agent cannot directly access environment states and may receive only sparse terminal rewards.The method uses a world model to shape rewards for failed trajectories.Source: §3.1 · p. 4

3.2 World progress reward modeling

  • Failed trajectories receive progress-wise rewards by comparing their world-model encodings with clustered representations of successful trajectories, with smaller distances yielding larger rewards.DBSCAN identifies representative centers; successful trajectories receive reward 1.0, while failed-trajectory distances are transformed using their mean and standard deviation.Source: §3.2 · pp. 4–5

3.3 Self-Referential Policy Optimization

  • SRPO optimizes VLA policies with self-referential, trajectory-level advantage estimates derived from latent world representations and relative performance within trajectory groups.Its objective uses a clipped surrogate and a KL-divergence term to maintain policy stability.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate SRPO on LIBERO and LIBERO-Plus. The reported results show state-of-the-art overall performance, advantages from its self-reference reward design, and stronger generalization than full-shot SFT baselines.

4.1 Experimental Setup

  • Evaluation spans LIBERO’s four 10-task suites and seven LIBERO-Plus perturbation dimensions, using OpenVLA* policies initialized by one-shot SFT and V-JEPA 2 latent representations for SRPO.OpenVLA* is the modified OpenVLA model with action chunking and parallel decoding.Source: §4.1 · pp. 6–7

4.2 Main Results

  • SRPO achieves state-of-the-art overall average success across LIBERO’s four suites, improving the one-shot SFT baseline through online post-training.The supplied passages report the overall comparison but provide no numerical results from Table 1.Source: §4.2 · p. 8
  • SRPO outperforms sparse-reward methods and TGRPO, supporting self-reference rewards from pre-trained world models over sparse outcomes and manually tuned task-specific progress rewards.The passages attribute the advantage to denser process supervision without heuristic stage partitioning or reward tuning.Source: §4.2 · p. 8
  • Using only third-person visual observations and language instructions, SRPO outperforms baselines that use multiple views, proprioception, or 3D data.The reported result indicates that SRPO’s state-of-the-art performance does not require those additional input modalities.Source: §4.2 · p. 8

4.3 Generalization Performance

  • SRPO surpasses full-shot SFT after LIBERO training and, under perturbations, also exceeds full-shot OpenVLA-OFT; Table 2 reports gains over one-shot SFT across all seven LIBERO-Plus dimensions.The passages attribute generalization gains to broader trajectory exploration during online interaction.Source: Table 2 · p. 7, §4.3 · p. 8

5 Analysis

Across the analyses, SRPO’s latent world-progress rewards produce more informative progress signals, support efficient learning from near-successful trajectories, and enable broader action exploration. The reward shaping also improves results across two VLA backbones on real-world manipulation tasks.

5.1 Can Latent World Representation Improve Reward Shaping?

  • Table 3 reports that SRPO outperforms the baseline on all five progress-reward indicators, while its progress curves are smoother and more monotonic than pixel-level and ImageBind signals.The evaluation uses 700 success and 300 failure trajectories; trajectory-level latent representations capture physical regularities, particularly for long-horizon tasks with repeated sub-tasks.Source: Table 3 · p. 8, §5.1 · pp. 8–9, Figure 3 · p. 9

5.2 Can SRPO Improve Training Efficiency?

  • SRPO reaches the four LIBERO suites in 79 Spatial, 59 Object, 103 Goal, and 219 Long steps, versus tens of thousands of steps for SFT.Compared with GRPO, SRPO has a steeper efficiency slope, especially on long-horizon tasks, by extracting learning signals from productive segments of near-successful episodes.Source: §5.2 · pp. 9–10, Figure 5 · p. 10

5.3 Can SRPO Motivate Novel Trajectory Exploration?

  • In LIBERO-Spatial, the online RL-trained policy explores previously unreachable regions and generates more dispersed end-effector trajectories than the SFT baseline.The comparison uses 10 trajectory episodes per task and indicates spatial exploration beyond specific demonstration paths.Source: §5.3 · pp. 10–11, Figure 6 · p. 10

5.4 Can world progress rewarding modeling extend to real-world?

  • π0 and π0-FAST show average gains of +66.8% and +86.7%, respectively, with progress-aware reward shaping on real-world manipulation tasks.The offline RL experiments use five tasks on an X-ARM 7 robot and compare both backbones with supervised fine-tuning baselines.Source: §5.4 · p. 11

6 Conclusion · A Progress Reward Benchmark

SRPO uses task-agnostic latent world representations for progress-wise rewards, using self-referential learning to exploit failed trajectories without expert demonstrations or task-specific engineering. The accompanying benchmark evaluates progress reward functions using curated trajectories across task domains and perturbations.

6 Conclusion

  • SRPO uses task-agnostic latent world representations to provide progress-wise rewards for VLA reinforcement learning.Source: §6 · p. 11
  • Self-referential learning removes reliance on expert demonstrations and task-specific engineering while utilizing information from failure trajectories.Source: §6 · p. 11
  • The paper reports improved performance and efficiency, with further analysis showing better reward shaping and motivation for novel trajectory exploration.Source: §6 · p. 11
  • Pre-trained world model latent representations provide a generic substrate for progress assessment, and the reward modeling can apply to real-world robotic tasks.Source: §6 · p. 11

A Progress Reward Benchmark

  • The benchmark uses 700 human-annotated successful and 300 failure trajectories across diverse task domains to assess progress reward function quality.Successful trajectories are manually selected for approximately linear forward progression without significant regressions or backtracking.Source: A Progress Reward Benchmark · p. 16
  • The benchmark includes viewpoint, lighting, object, sensor-noise, and background perturbations, and evaluates progress signals with five core metrics.Source: A Progress Reward Benchmark · p. 16

A.1 Core Evaluation Metrics · A.2 Evaluating Progress Reward Quality

The sections define complementary measures of progress-reward quality: temporal consistency within trajectories and separation between successful and failed trajectories. They describe how these measures assess reward progression and outcome discriminability.

A.1 Core Evaluation Metrics

  • Temporal Correlation measures the Spearman rank correlation between progress values and frame numbers across tasks; higher absolute values indicate stronger monotonic relationships.The metric is defined across N tasks and their trajectory lengths.Source: §A.1 · p. 16
  • Temporal Monotonicity measures the average percentage of steps at which progress increases across tasks, with values closer to 100 indicating stronger monotonic progression.It uses an indicator function over progress at each step.Source: §A.1 · p. 16
  • Maximum Mean Discrepancy (MMD) measures average separation between success and failure trajectories across tasks using their final progress values.The calculation uses a feature map in a reproducing kernel Hilbert space.Source: §A.1 · p. 16
  • Jensen-Shannon Divergence measures average distributional divergence between success and failure trajectories across tasks.The definition uses the mixture distribution and Kullback-Leibler divergence.Source: §A.1 · p. 16
  • Standardized Mean Difference measures average separation effect size between success and failure trajectories, using pooled standard deviation.The pooled standard deviation is defined for each task.Source: §A.1 · p. 16
  • Figure 9 depicts a smoothly rising reward curve for a successful wiping trajectory and a stagnant signal for a failed mug-placement trajectory.The success curve dips slightly during a grasping pause; the failed trajectory does not reach a high reward value after the model fails to locate the second mug.Source: Figure 9 · p. 17

A.2 Evaluating Progress Reward Quality

  • Temporal consistency is indicated by correlation values approaching 1.0 and monotonicity scores near 100%, while distribution discriminability reflects separation between successful and failed trajectories.Larger MMD and absolute SMD indicate better separation; JSD values closer to ln 2 suggest maximal discriminability.Source: §A.2 · p. 17
  • Progress-reward quality is assessed through internal consistency within successful trajectories and external discriminability between success and failure outcomes.The framework combines complementary metrics to evaluate both properties.Source: §A.2 · pp. 17–18

B Reward Analysis Details · C Ablations

SRPO’s reward curves provide more reasonable and stable progress estimates than pixel-level rewards and ImageBind, while its reward construction uses normalized visual-distance measures for successful and failed trajectories.

B Reward Analysis Details

  • For successful trajectories, SRPO and ImageBind use cumulative sliding-window video embeddings and normalize distance to the full-video embedding from 0 at maximum distance to 1 at minimum distance.The windows progress from frames 0–10 through frames 1–12 and onward, ending with frames 1 to the penultimate frame.Source: B Reward Analysis Details · p. 18
  • For successful trajectories, RLVR measures each frame’s L1 distance to the final frame, from frame 10 through the penultimate frame, and applies the same 0-to-1 normalization.This is the pixel-level reward construction.Source: B Reward Analysis Details · p. 18
  • For failed trajectories, SRPO measures each segment’s minimum L2 distance to successful-trajectory cluster centers and assigns 1 to the closest segment and 0 to the most distant.The cluster centers serve as reference points for progress reward computation.Source: B Reward Analysis Details · p. 18
  • SRPO produces more reasonable and stable progress estimates than pixel-level rewards and ImageBind in the analyzed trajectories.The reward-curve analysis includes randomly selected successful and failed trajectories.Source: B Reward Analysis Details · p. 18
  • Pixel-level rewards struggle with long-horizon tasks containing multiple subtasks, often showing sharp progress increases only in the final frames.The analysis compares pixel-level rewards with SRPO and ImageBind.Source: B Reward Analysis Details · p. 18
  • ImageBind captures trajectory-level features, but its rewards can oscillate with sudden spikes or drops that are unfriendly to reinforcement learning.The analysis attributes this limitation to the features’ lack of physical intuition.Source: B Reward Analysis Details · p. 18

C.1 Ablation on Self-Referential Mechanism · C.2 Ablation on Success Clustering

The ablations show that within-batch self-reference avoids the performance ceiling of fixed expert references, while clustering successful trajectories becomes increasingly valuable as exploration diversifies. Together, they support progress rewards based on the policy’s evolving successes and robust strategy-level comparisons.

C.1 Ablation on Self-Referential Mechanism

  • The fixed-reference variant initially trains slower than full SRPO, still significantly outperforms GRPO, but plateaus and needs nearly 1.4 times the training steps for suboptimal results.Figure 10 also reports a significant performance drop when the referential component is removed.Source: §C.1 · p. 18, Figure 10 · p. 18
  • External progress-wise rewards provide a stronger learning signal than sparse binary rewards early, but fixed references eventually let GRPO catch up and constrain open-ended exploration.As the policy evolves, static trajectories cannot provide nuanced step-by-step progress assessments for its diverse rollouts, creating a lower performance ceiling than self-reference.Source: §C.1 · pp. 18–19

C.2 Ablation on Success Clustering

  • Tasks can have multiple successful strategies, so a failure trajectory should be compared with the nearest successful strategy rather than an arbitrary one.For example, an object-placement task may succeed by placing object A before B or vice versa.Source: §C.2 · p. 19
  • Using a cluster centroid instead of one successful trajectory reduces sensitivity to noisy or suboptimal segments that could otherwise inflate a failure trajectory’s progress score.A centroid represents a cleaner, more prototypical version of a successful strategy.Source: §C.2 · p. 19
  • The nearest-success-trajectory variant has initial learning efficiency comparable to full SRPO, but its performance gains diminish significantly as training progresses.Figure 10 reports that removing clustering slows convergence.Source: Figure 10 · p. 18, §C.2 · p. 19
  • Clustering matters more later because successful strategies diversify and multiply, making prototypical strategies and robust progress signals increasingly important.Early in training, the limited repertoire of successful strategies and small number of success trajectories reduce clustering’s advantage.Source: §C.2 · p. 19

D Hyperparameter Analysis … F.1 Supervised Fine-Tuning (SFT) Stage

The hyperparameter analysis identifies α = 0.8 as the best balance between progress guidance and task completion. The pixel-level world-model case study reports poor zero-shot video consistency and contrasts costly, demonstration-dependent fine-tuning with SRPO’s latent-representation approach.

D Hyperparameter Analysis

  • The reward uses a sigmoid to map trajectories into (0, 1), with α scaling progress rewards while correct answers receive full credit (1.0).The evaluated α values were 0, 0.3, 0.5, 0.8, and 1.0.Source: D Hyperparameter Analysis · p. 19
  • α = 0 performed worst, while α = 0.3 and 0.5 showed gradual improvement, indicating that progress rewards enhance learning efficiency.The paper attributes the poor α = 0 result to purely outcome-based rewards being insufficient for complex tasks requiring sequential reasoning.Source: D Hyperparameter Analysis · p. 19
  • The α ranking was 0.8 > 1.0 > 0.5 > 0.3 > 0, with α = 0.8 achieving the best performance.The analysis describes this as a balance between progress awareness and outcome correctness.Source: Figure 11 · p. 19, D Hyperparameter Analysis · p. 19
  • α = 1.0 performed better than lower values but below α = 0.8, suggesting excessive emphasis on progress can distract from the final objective.The analysis concludes that α = 0.8 outperforms both purely outcome-based rewards and excessively progress-focused rewards.Source: D Hyperparameter Analysis · pp. 19–20

E What if we use a pixel-level world model for reward shaping?

  • Cosmos-Predict2-14B generated reference videos with poor scene consistency in zero-shot LIBERO tasks; task-specific SFT is costly and depends heavily on expert demonstrations.The paper presents SRPO’s latent-world-representation reward modeling as more cost-effective and generalizable.Source: E What if we use a pixel-level world… · p. 20

F.1 Supervised Fine-Tuning (SFT) Stage

  • The one-shot SFT phase used an OpenVLA action-chunking and parallel-decoding checkpoint, 8 × A100 GPUs, up to 150,005 steps, and learning-rate decay after 100,000 steps.Image augmentation was enabled; inputs were text instructions and single-image observations, with 8 action chunks.Source: §F.1 · p. 20

F.2 SRPO Post-Training Stage … G.2 Rewarding Validation

The real-world experiments combine progress-aware RL with diverse manipulation tasks and report gains over SFT; a separate validation evaluates whether latent progress rewards track task advancement across five robot tasks.

F.2 SRPO Post-Training Stage

  • SRPO post-training uses GRPO-based advantage estimation, 8 samples per group, a 0.8 progress reward weight, and 50 trials per task.The RL phase is initialized from the one-shot SFT model.Source: §F.2 · p. 20

F.3 Model Details

  • The model retains Llama 2’s discrete action-token output rather than continuous action heads, enabling action log-probabilities for policy gradients at a potential cost to action precision.The passage contrasts this design with OpenVLA-OFT’s continuous action heads.Source: §F.3 · p. 21

G.1 Detailed Setup

  • For safety and reset-cost reasons, physical-robot training used offline RL combining AWR with SRPO’s self-referential progress-wise rewards and advantages.Per-step progress is the difference between successive cumulative rewards, and advantages normalize these increments using their mean and standard deviation across trajectories. The policy is trained from scratch using pretrained weights and compared with the SFT model.Source: §G.1 · p. 21
  • The real-world task suite tests fruit selection under distractors and position swaps, towel folding, whiteboard cleaning, and semantic selection of a specified poker card.The fruit tasks are putting an apple or pear into a plate; the poker task selects the specified card from five cards.Source: §G.1 · p. 21
  • Both VLA policy backbones improved substantially over their SFT counterparts across all five real-world manipulation tasks.The passage refers to Figure 8, but no Figure 8 visual or numerical results are supplied here.Source: §G.1 · p. 21

G.2 Rewarding Validation

  • Reward validation evaluates the pretrained progress-reward function on five real-robot tasks, each with 30 successful and 20 failure trajectories.Source: §G.2 · p. 22
  • Across five real-robot tasks, latent progress rewards achieved near-perfect Spearman correlation and monotonicity, with strong MMD/SMD separation despite simulation-to-reality shifts.Table 4 is identified as the Progress Reward Benchmark results table; the supplied passages do not provide its cell values.Source: Table 4 · p. 21, §G.2 · p. 22

G.3 Case Study

Real-world case studies show the approach adapting to moved targets, deformable objects, and semantic choices, alongside examples of progress changes and partial or failed two-step tasks.

G.3 Case Study

  • In real-world cases, the policy replans when a pear’s target plate moves, handles towel folding, and identifies a Joker card among multiple options.The paper presents these cases as evidence of robustness and generalization across diverse scenarios.Source: §G.3 · p. 22
  • In board wiping, the eraser was dropped and picked up again, while pausing during grasping could decrease progress.The passage does not specify what the model was doing when progress decreased.Source: §G.3 · p. 24
  • Across four two-step tasks, book placement and bowl placement failed at step 1, while mug placement and the stove-and-moka-pot task finished step 1.The tasks were picking up a book and placing it in the back compartment; placing a white mug on the left plate and a yellow mug on the right; turning on the stove and putting a moka pot on it; and placing a black bowl from a ramekin on top of a cabinet.Source: §G.3 · p. 25

Related papers

Loading 2511.15605v2…