PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Correcting VLA: Online Action Refinement via Sparse World Imagination

Chenyv Liu, Wentao Tan, Lei Zhu, Fengling Li, Jingjing Li, Guoli Yang, Heng Tao Shen

arXiv:2602.21633v1cs.ROcs.AIcs.CV

TL;DR

VLA models often rely on static data priors, while reinforcement learning commonly uses external rewards and world action models lack explicit self-improvement mechanisms. SC-VLA combines sparse world imagination with online action refinement to guide actions through internally predicted progress and future states. It reports state-of-the-art performance, including the highest throughput, fewer execution steps, higher success, and gains in real-world experiments.

  • Problem

    VLA models lack robust physical-dynamics understanding, external rewards can be disconnected from internal states, and world action models lack explicit self-improvement mechanisms.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    SC-VLA predicts task progress and future trajectory trends with sparse world imagination, then uses online action refinement to reshape dense rewards and adjust actions.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    SC-VLA demonstrates state-of-the-art performance with the highest task throughput across four manipulation tasks and improves task throughput in extensive simulation and real-world experiments.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    Structured predictive priors can guide physically consistent actions while avoiding explicit world models and manual reward engineering.

    Source: §6 · pp. 8–9

    Existing world action models remain limited by implicit future representations that lack interpretable physical semantics and direct policy-improvement signals.

    Source: §2 · p. 3

Abstract

from arXiv · show

Standard vision-language-action (VLA) models rely on fitting statistical data priors, limiting their robust understanding of underlying physical dynamics. Reinforcement learning enhances physical grounding through exploration yet typically relies on external reward signals that remain isolated from the agent's internal states. World action models have emerged as a promising paradigm that integrates imagination and control to enable predictive planning. However, they rely on implicit context modeling, lacking explicit mechanisms for self-improvement. To solve these problems, we propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination. We first design sparse world imagination by integrating auxiliary predictive heads to forecast current task progress and future trajectory trends, thereby constraining the policy to encode short-term physical evolution. Then we introduce the online action refinement module to reshape progress-dependent dense rewards, adjusting trajectory orientation based on the predicted sparse future states. Evaluations on challenging robot manipulation tasks from simulation benchmarks and real-world settings demonstrate that SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines, alongside a 14% gain in real-world experiments. Code is available at https://github.com/Kisaragi0/SC-VLA.

1. Introduction

SC-VLA addresses the limits of static VLA priors, external rewards, and implicit future representations by combining sparse world imagination with online action refinement. It achieves strong throughput and success across simulation and real-world manipulation tasks.

1. Introduction

  • Existing VLA policies mainly fit statistical data priors rather than robustly modeling physical dynamics.Large-scale imitation learning enables direct action generation but does not ensure physical grounding.Source: §1 · p. 1
  • Reinforcement-learning approaches often depend on external reward signals that can remain disconnected from the model’s internal states.This motivates self-improvement strategies based on native adaptation.Source: §1 · pp. 1–2
  • World action models support action generation and future prediction, but existing methods lack explicit mechanisms to use predicted states for action refinement.Their future signals therefore do not directly realize internal-state-aligned self-improvement.Source: §1 · p. 2
  • Sparse World Imagination forecasts task progress and future trajectory orientation, constraining the policy to encode short-term physical evolution before generating actions.Auxiliary predictive heads provide sparse future-state signals.Source: §1 · p. 2
  • Online Action Refinement uses predicted future states to reshape progress-dependent dense rewards and adjust trajectory orientation through residual reinforcement learning.This intrinsic guidance avoids reliance on external reward models.Source: §1 · p. 2
  • SC-VLA achieves the best task throughput and success rate across four simulation and real-world manipulation tasks.The evaluated tasks are StackCube, PlaceSphere, LiftPegUpright, and PegInsertion.Source: §1 · p. 2

2. Related Works

Prior VLA research combines imitation learning, reinforcement learning, and world-action modeling, but reward construction and future-state representations remain limited. SC-VLA is positioned as an explicit, internally guided alternative.

Vision-Language-Action Models

  • VLA models such as RT-2, OpenVLA, GR00T, and π0 improve semantic transfer, adaptation, deployment, or continuous action generation while lacking explicit physical-evolution modeling.Their reliance on offline imitation and current-step semantic alignment leaves physical evolution under-modeled.Source: §2 · pp. 2–3

Reinforcement Learning for VLA

  • Reinforcement-learning methods for VLA synthesize rewards from semantic reasoning, rules, auxiliary predictions, trajectory similarity, or world-model deviations.These approaches address sparse feedback through several external reward-design strategies.Source: §2 · p. 3
  • These reward-based methods generally depend on external inference, handcrafted rules, or similarity computations that are decoupled from policy representations and add system complexity.Scalar return prediction alone also struggles to provide fine-grained spatiotemporal geometric constraints.Source: §2 · p. 3

World Action Models

  • World action models jointly predict actions and future evolution to provide intrinsic guidance without explicit external rewards.Examples use imagined images, latent future representations, physical tokens, or autoregressive future visual states.Source: §2 · p. 3
  • Existing world action models encode future signals implicitly, limiting interpretable physical semantics and fine-grained short-horizon policy correction.They also lack direct policy-improvement signals comparable to reward-based methods.Source: §2 · p. 3

3. Preliminary

The preliminary method uses conditional flow matching for continuous robot action generation and Soft Actor-Critic for stable reinforcement-learning optimization. Together, they provide the policy and optimization foundations for later predictive constraints.

3.1. Basic Robot Policy

  • Conditional flow matching transforms prior noise into an observation-conditioned target action distribution using a learned vector field.Training matches the predicted vector field to the target velocity between noise and ground-truth action samples.Source: §3.1 · p. 3
  • At inference, the flow-matching policy generates actions by solving its ordinary differential equation with a numerical integrator such as Euler stepping.This formulation supports continuous action generation from a prior noise sample.Source: §3.1 · p. 3
  • The policy formulation is extended with structured predictive constraints to improve short-term physical-evolution modeling.These constraints are introduced after the basic flow-matching formulation.Source: §3.1 · p. 3

3.2. Soft Actor-Critic (SAC)

  • Soft Actor-Critic optimizes the continuous-action policy with an off-policy maximum-entropy objective balancing expected return and exploration.Entropy regularization is controlled by an automatically tuned temperature parameter α.Source: §3.2 · p. 4
  • SAC uses reparameterized action sampling to enable low-variance gradient backpropagation through the stochastic policy.This supports stable policy optimization in continuous action spaces.Source: §3.2 · p. 4

4. Self-Correcting VLA

SC-VLA combines sparse world imagination with residual online action refinement in a two-stage framework, using internally derived predictive signals to guide physical action adaptation.

4. Self-Correcting VLA

  • SC-VLA combines a flow-matching base policy guided by sparse imagination with a residual reinforcement-learning module optimized using intrinsic predictive rewards.The framework derives rewards from internal imagination consistency rather than external dense supervision.Source: §4 · p. 4, Figure 2 · p. 5

4.1. Sparse World Imagination

  • Sparse world imagination injects explicit progress and short-horizon state-change targets into the query sequence, constraining action generation to encode physical evolution.The model predicts task progress and relative state changes from intermediate DiT features while jointly optimizing flow matching and auxiliary physical supervision.Source: §4.1 · pp. 4–5

4.2. Online Action Refinement

  • Residual refinement adds minimal online corrections to frozen base actions, using sparse world-imagination observations to preserve base-policy intent while adapting to perturbations and fine-grained contacts.The residual action is scaled and combined with the frozen base policy, avoiding unconstrained exploration in raw observation space.Source: §4.2 · p. 5
  • The directional dense reward aligns executed end-effector displacement with the short-term evolution direction predicted by the base policy.It uses the predicted translational state change and provides continuous directional feedback after residual actions.Source: §4.2 · pp. 5–6
  • Progress-dependent reward scheduling emphasizes predictive guidance early and gradually shifts toward autonomous exploration as task progress increases.The final reward combines environment and predictive guidance rewards, balancing early exploration efficiency with late-stage sensitivity to real dynamics feedback.Source: §4.2 · p. 6

5. Experiments

SC-VLA is evaluated across simulation and real-world manipulation tasks, with experiments targeting success rate, throughput, component contributions, and transferability. It outperforms the listed baselines while ablations identify complementary guidance and reward mechanisms.

5.1. Simulation Setup and Baselines

  • On PegInsertion, SC-VLA (SPI) improves success rate by 28% over π0 and 10% over GR00T N1.5, while SC-VLA (SPI, OAR) reaches 86%.The comparison concerns pretrained baselines and the addition of residual reinforcement learning.Source: §5.1 · p. 7, Table 1 · p. 7
  • SC-VLA outperforms existing baselines across four ManiSkill manipulation tasks, achieving superior success rates and the shortest average completion length of 157 steps.This corresponds to a 43% reduction versus π0 and an 8% reduction versus Diffusion Policy.Source: §5.1 · pp. 6–7, Table 2 · p. 7

5.2. Ablation Study

  • Removing progress guidance lowers average success from 82% to 80%, mainly affecting staged tasks such as StackCube and LiftPegUpright.The ablation associates progress guidance with improved temporal consistency during multi-stage execution.Source: §5.2 · p. 7, Table 3 · p. 7
  • Removing state guidance lowers average success from 82% to 78%, with larger drops on PegInsertion and StackCube, indicating its role in physical consistency.The effect is most pronounced for tasks sensitive to contact dynamics and pose accuracy.Source: §5.2 · p. 7, Table 3 · p. 7
  • Removing both progress and state guidance reduces average success to 72%, below either single-ablation setting, showing complementary roles.The joint ablation evaluates both imagination components under identical protocols.Source: §5.2 · p. 7, Table 3 · p. 7
  • Sparse imagination rewards reduce PegInsertion exploration steps from 800 to 650, while fixed reward weights cause late-stage degradation on precision tasks.Dynamic weighting balances early guidance with later autonomous fine-tuning, and the real-world evaluation reports 70% average success, exceeding DP and GR00T N1.5 by 43% and 14%.Source: §5.2 · p. 8, §5.3 · p. 8, Table 4 · p. 8

6. Conclusion

SC-VLA couples prediction and control through sparse world imagination to address static priors and limited physical modeling in VLA. Its intrinsic reinforcement-learning design improves task throughput and supports physically consistent actions without explicit world models.

  • SC-VLA couples prediction and control through sparse world imagination, addressing static priors and insufficient physical modeling in existing VLA.The framework uses structured predictive priors to guide physically consistent actions without explicit world models.Source: §6 · p. 8
  • SC-VLA unifies world-aware modeling with intrinsic reinforcement learning and eliminates the need for manual reward engineering.Source: §6 · p. 9

Appendix A: Task Setup and Evaluation Details

The simulation evaluation uses ManiSkill3 tasks built on the SAPIEN physics engine to test robotic manipulation under high-precision control and complex contact conditions.

  • ManiSkill3 simulation tasks use the SAPIEN physics engine to evaluate high-precision robotic manipulation under complex contact conditions.Source: Appendix A · p. 11

A.1 Task Descriptions

The evaluation covers four manipulation tasks spanning stacking, placement, upright peg lifting, and peg insertion, each emphasizing distinct precision or contact-control challenges.

StackCube

  • StackCube evaluates grasping, placement, pose accuracy, and execution stability, with sensitivity to positional errors and end-effector jitter.Source: §A.1 · p. 11

PlaceSphere

  • PlaceSphere requires grasping and placing a rolling sphere, imposing stringent end-effector velocity and dynamic-stability requirements.Source: §A.1 · p. 11

LiftPegUpright

  • LiftPegUpright tests contact adjustment, relative sliding, angle control, and pose refinement while converting a horizontal peg to an upright pose.Source: §A.1 · p. 11

PegInsertion

  • PegInsertion evaluates precise peg-to-hole alignment and fine-grained contact adjustments under small insertion tolerances.Source: §A.1 · p. 11

A.2 Training and Evaluation Settings

All methods share the same demonstration budget and evaluation protocols, while the appendix specifies the task criteria and baseline implementations used for comparison.

A.2 Training and Evaluation Settings

  • All simulation methods use the same demonstration-data amount and are evaluated independently for 50 episodes per task using task success rate as the primary metric.Reward definitions follow official ManiSkill3 specifications, with task success criteria summarized in Table 5.Source: §A.2 · p. 11
  • Table 5 defines the detailed success criteria for the four manipulation tasks used in evaluation.Source: Table 5 · p. 11

Appendix B: Baseline Details and Settings

  • The baseline appendix states that all baselines share the same demonstration-data budget and evaluation protocols for fair comparison.Source: Appendix B · p. 11

B.1 DP (Diffusion Policy)

  • Diffusion Policy models continuous action distributions through progressive denoising and use a two-frame history, 16-step horizon, and eight-step execution chunk.Source: §B.1 · p. 12

B.2 ACT (Action Chunking with Transformers)

  • ACT combines CVAEs with Transformers to generate coherent, smooth chunked action trajectories over long horizons.Source: §B.2 · p. 12
  • π0 is a robot foundation model trained across large-scale multi-task and cross-embodiment data, using a pretrained vision-language model with flow-based action generation.Source: §B.3 · p. 12
  • π0 uses cosine learning-rate decay with 3,000 warmup steps, a 2 × 10^-5 peak rate, and a 1.5 × 10^-6 final rate decaying over 100,000 steps.Source: §B.3 · p. 12

B.4 GR00T N1.5

  • GR00T N1.5 uses flow matching and DiT vector-field learning to model continuous actions with efficient and stable generation.Source: §B.4 · p. 12

Appendix C: Experiment Details

SC-VLA uses a three-stage protocol that transitions from frozen-policy data collection to residual reinforcement learning. Initialization phases stabilize training, while reported curves isolate main-phase convergence and asymptotic performance.

Multi-Stage Training Protocol

  • The protocol has three phases: Buffer Warm-up, Residual Injection, and Main Training, with residual weight λ transitioning from imitation to residual learning.The schedule is designed to stabilize a hybrid frozen base policy and randomly initialized SAC residual module.Source: Appendix C · p. 12, Figure 5 · p. 13

Initial Data Collection

  • During Buffer Warm-up, λ = 0 and the frozen base policy collects demonstration trajectories to populate the SAC replay buffer before policy updates.This gives the critic a meaningful state distribution before residual learning begins.Source: Appendix C · p. 12

Residual Injection

  • Residual Injection linearly increases λ from 0 to its target scaling factor, allowing the residual policy to adapt without immediately disrupting control.The gradual schedule addresses instability from the initially random residual policy and is associated with an adaptation dip during exploration.Source: Appendix C · pp. 12–13

Main Training Phase

  • After λ reaches its target, Main Training begins with λ fixed subject to dynamic scheduling, and formal training and evaluation proceed.The main phase follows the two transient initialization stages.Source: Appendix C · p. 13

Visualization Rationale

  • Performance curves exclude data-collection and injection artifacts, recording only the Main Training Phase to emphasize convergence behavior and asymptotic performance.Preliminary metrics fluctuate because the base policy is followed first and the residual module explores during injection.Source: Appendix C · p. 13

Appendix D: Implementation Details

The implementation separates base-policy training from residual-policy reinforcement learning and adapts residual strength to task difficulty. Appendix materials also provide configuration tables and visualization references for simulated and real-world tasks.

Stage I: SC-VLA Base Policy Training

  • SC-VLA base-policy training uses a GR00T N1.5 DiT architecture, jointly optimizing flow matching and multimodal prediction heads with AdamW for 50,000 iterations.Training is performed on a single NVIDIA L40 GPU, with configurations summarized in Table 6.Source: Appendix D · p. 13, Table 6 · p. 13

Stage II: SC-VLA Residual Policy Training

  • Residual-policy training uses Soft Actor-Critic while fully freezing the base model and optimizing only the residual network π_res.Residual-model settings are summarized in Table 7.Source: Appendix D · p. 13, Table 7 · p. 14

Task-Adaptive Residual Scaling

  • Task-adaptive residual scaling assigns smaller coefficients to tasks with higher base-policy success rates and larger coefficients to tasks where the base policy performs worse.The inverse-correlation strategy preserves strong priors while enabling larger corrections when refinement is needed.Source: Appendix D · pp. 13–14

Reward Scheduling

  • Predictive-guidance reward uses a consistent dynamic scheduling strategy across experiments, despite task-specific residual scaling factors.Reinforcement-learning hyperparameters are summarized in Table 7.Source: Appendix D · p. 14, Table 7 · p. 14

Visualization Results

  • Figures 6 and 7 provide visualization results for Maniskill simulated tasks and real-world tasks, respectively.The captions identify the evaluation settings but do not state a specific visual comparison.Source: Figure 6 · p. 15, Figure 7 · p. 15
Loading 2602.21633v1…