PaperlayerView exact revision on arXiv

Source-linked AI summary

PAVE: Predictive Alignment and Value-Guided Evolution for World-Action Policies

Botong Zhao, Fang Yu, Tim Yu, Senhua Zhu, Xinyuan Chen, Yue Lu

arXiv:2608.30378v2cs.ROcs.AI

TL;DR

Direct VLA policies do not explicitly represent multi-timescale scene transitions or distinguish useful dynamics from undesirable deployment behavior. PAVE combines multi-horizon predictive alignment with advantage-conditioned policy improvement, achieving higher average success across three simulation benchmarks while preserving direct online execution. Its scope includes limitations from temporal anchors, chunk-level critic labels, critic calibration, and repeated retraining from one base checkpoint.

  • Problem

    Direct VLA behavior cloning does not explicitly require multi-timescale transition representations and can reuse deployment trajectories of unequal quality without separating desirable behavior.

  • Method

    PAVE combines local and trajectory-relative multi-horizon JEPA alignment with an independent distributional critic that produces action-chunk-aligned advantage conditions for a flow-matching actor.

  • Results

    Across LIBERO, LIBERO-Plus, and RoboTwin 2.0, PAVE improves average success over JEPA-WAM by 2.6, 6.0, and 6.6 points, respectively, while matching online model size and memory footprint.

  • Takeaways & Limitations

    PAVE preserves a direct positive-conditioned flow actor online because its predictive and value-guidance mechanisms are restricted to training or offline annotation.

  • Takeaways & Limitations

    Trajectory-relative anchors are not semantic subgoals, the critic labels chunks rather than exact failing actions, and top-30% labels may be sensitive to overfitting, calibration, and data coverage.

Abstract

from arXiv · show

Direct vision-language-action policies generate continuous robot actions efficiently, but standard behavior cloning leaves two complementary gaps: their representations are not explicitly required to describe how the scene evolves over multiple time scales, and deployment trajectories of unequal quality are often reused without separating useful dynamics from undesirable behavior. We introduce \method, a direct world-action policy that combines outcome-agnostic predictive learning with outcome-aware policy improvement. \method first retains a local fixed-offset JEPA objective and adds trajectory-relative multi-horizon transition alignment at 25%, 50%, 75%, and 100% of the remaining episode. These training-only targets require the current policy representation to preserve both local physical changes and longer-range task progress, without supplying explicit future tokens to the action head. \method then trains an independent distributional value critic on cumulative deployment trajectories, computes action-chunk-aligned $N$-step advantages, and converts them into positive, negative, or null text conditions for a flow-matching actor. Thus, every valid trajectory can teach what physically happened, while the actor is deployed only under the condition associated with relatively better actions. The multi-horizon predictor and critic are removed from online execution, preserving direct action generation from the current observation, language instruction, and proprioception. \redclaim{Across the three simulation benchmarks, \method achieves the strongest overall performance while preserving the direct actor's online execution path.}

1 INTRODUCTION

PAVE addresses two complementary gaps in direct VLA policies: limited multi-timescale transition representation and indiscriminate reuse of mixed-quality deployment behavior. It combines trajectory-relative predictive alignment with value-guided action conditioning while keeping auxiliary modules out of online execution.

  • Direct VLA imitation reproduces demonstrated actions without explicitly requiring representations to encode associated environmental transitions over longer time scales.
  • Value-guided relabeling separates relatively better and worse behavior, but does not explicitly improve multi-timescale transition representation.
  • PAVE adds trajectory-relative transition anchors at 25%, 50%, 75%, and 100% of the remaining episode to a local JEPA target.
  • A distributional critic computes N-step advantages and converts them into positive, negative, or null text conditions for the actor.
  • PAVE lets valid failed trajectories supervise physical transitions while preventing low-quality actions from being indiscriminately imitated.
  • The multi-horizon predictor and value critic remain restricted to training or offline annotation rather than online execution.

2 RELATED WORK

Related work spans direct VLA policies, predictive world-action representations, and experience-driven policy improvement. PAVE follows this trajectory by combining predictive representation learning with critic-driven conditional policy extraction.

  • Direct VLA methods use pretrained visual-language representations and continuous action models, but standard action losses do not organize representations around multiple future time scales or mixed-quality experience.
  • Predictive representation methods learn temporally predictive visual features, while robot-policy variants incorporate future or current–future representations into direct control.
  • Experience-driven approaches use value functions, preferences, or failed-action redirection to improve policies from deployment experience.
  • PAVE uses a critic-driven conditional-policy route to annotate unpaired successful and failed rollouts at action-chunk resolution.

3 PROPOSED METHOD

PAVE combines direct flow-matching action generation with training-only multi-horizon predictive supervision and offline value-guided relabeling. The deployed policy uses only current observations while predictive and critic branches shape training and data selection.

  • Overview: PAVE jointly trains a direct actor with local transition alignment, multi-horizon future targets, and flow-matching action prediction.The actor maps current multimodal observations, task language, proprioception, and behavior-quality text to continuous action chunks.
  • Multi-horizon predictive learning: The multi-horizon predictor supplies training supervision but is never passed to the action expert or executed during inference.Future targets shape the shared representation during training, while the deployed path retains the current-observation actor.
  • Distributional value-guided annotation: An independent distributional critic maps multimodal state features to 201 value-bin logits and estimates state value without receiving a candidate action.The critic uses cumulative deployment trajectories and produces a scalar value from the expectation over its categorical supports.
  • Distributional value-guided annotation: Action-chunk-aligned N-step advantages rank eligible chunks, assigning positive labels to the top 30% and negative labels to the remainder.Labels are relative to cumulative data, so recovery segments can be positive and inefficient segments can be negative regardless of whole-episode outcome.
  • Advantage-conditioned policy evolution: PAVE retrains each actor from a fixed base initialization using demonstrations and critic-labeled deployment data, then deploys only the positive condition.Online execution samples a 20-step chunk directly; the paired-future encoder, predictors, critic, and advantage computation are omitted.

4 EXPERIMENTS

PAVE is evaluated on three simulation benchmarks, robustness perturbations, qualitative robot trajectories, online efficiency, and component ablations. It improves benchmark success while retaining a direct online execution path and restricting auxiliary predictor and critic computation to training or offline annotation.

  • Experimental setup: PAVE is evaluated on LIBERO, LIBERO-Plus, and RoboTwin 2.0, with success reported over three independent seeds using fixed task and initial-state manifests.LIBERO-Plus includes seven controlled perturbation dimensions, while RoboTwin 2.0 covers 50 bimanual tasks with structured randomization.
  • Main benchmark results: 2.6 points higher average success on LIBERO and 3.4 points higher long-horizon success than JEPA-WAM.PAVE’s LIBERO average is also 2.0 points above π∗.
  • Main benchmark results: 6.0 points higher average success on LIBERO-Plus than JEPA-WAM, with gains of 7.9 points under background changes and 7.4 points under sensor noise.The average improvement over π∗ is 4.7 points.
  • Main benchmark results: 6.6 points higher average success on RoboTwin 2.0 than JEPA-WAM, with margins of 7.0 points under clutter and 7.5 points under background changes.The average improvement over π∗ is 4.8 points.
  • Qualitative trajectory visualization: The real-robot sequence shows approach, interaction, and completion qualitatively, but does not provide a quantitative success or robustness evaluation.Figure 4 contains seven frames per row for simulation and real-robot trajectories.
  • Online efficiency: PAVE matches JEPA-WAM’s 1.29B online parameters and 10.4GB memory footprint, while adding only 0.2ms to p50 and p95 latency.The 4.63M additional parameters belong to the training-time predictor and offline critic.
  • Core ablations: With ACP disabled, the local fixed-offset target improves success by 8.1 points on LIBERO-Plus and 7.5 points on RoboTwin 2.0 over flow matching alone.Adding the shared four-horizon objective yields further improvements of 3.5 and 3.4 points, respectively.
  • Core ablations: Across three evolution rounds, success rises from 67.2 to 73.2 on LIBERO-Plus and from 53.4 to 60.0 on RoboTwin 2.0.The positive test condition reaches 66.6 average success, compared with 64.2 for null, 47.6 for negative, and 61.7 for unlabeled behavior cloning.

5 LIMITATIONS AND DISCUSSION

PAVE’s limitations concern representation supervision, critic granularity and calibration, offline computation, and the gap between simulation results and safe, robust deployment.

  • Trajectory-relative anchors mark temporal progress rather than semantic subgoals, may coincide near termination, and can encode unsuccessful endpoints.They supervise representations but do not guarantee plans.
  • The state-value critic assigns chunk-level relative labels and cannot identify the exact failing action within a chunk.Action-level correction methods such as RedFlow may provide denser supervision.
  • Top-30% labels use in-sample critic predictions, making them sensitive to critic overfitting, calibration, and data coverage.Trajectory-level cross-fitting remains an important extension.
  • Retraining the actor from one base checkpoint each round incurs substantial offline computation despite efficient online inference.
  • Simulation success and one qualitative real-robot sequence do not establish safety or robustness in unconstrained environments.Broader deployment requires conservative termination, intervention logging, and failure auditing.

6 CONCLUSION

PAVE separates learning what transitions occur from learning which behaviors should be preferred, while keeping auxiliary prediction and value mechanisms offline. It improves average success over JEPA-WAM across three benchmarks without increasing the online actor’s model size or memory footprint.

  • PAVE separates transition learning from behavior preference learning using multi-horizon JEPA alignment and a distributional critic.
  • Both auxiliary mechanisms operate only during training or offline annotation, leaving one direct positive-conditioned flow actor online.
  • 2.6, 6.0, and 6.6 points: PAVE improves average success over JEPA-WAM across LIBERO, LIBERO-Plus, and RoboTwin 2.0, respectively.The comparison preserves JEPA-WAM’s online model size and memory footprint.
  • Reported success rates use fixed task and initial-state manifests and mean ± standard deviation over three independent seeds.
  • The real-robot sequence is a qualitative demonstration rather than a safety or robustness evaluation.Broader deployment requires bounded workspaces, speed and force limits, emergency stops, operator supervision, and explicit handling of censored safety terminations.

A DETAILED EXPERIMENTAL PROTOCOL

The experimental protocol isolates data across training, evolution, diagnostics, and evaluation while enforcing valid action-chunk provenance and consistent statistical comparisons.

  • Training, evolution, development, and held-out evaluation use non-overlapping episode and task-variant manifests.Raw rollouts remain immutable, while derived quantities carry configuration hashes and actor lineage.
  • An anchor enters training or advantage ranking only when all 20 actions are valid, finite, and generated under consistent provenance.Endpoint replication is excluded as a real action, and future JEPA pairs remain within one provenance segment.
  • Each benchmark reports mean and standard deviation over three independent seeds using the same fixed task and initial-state manifests within comparisons.
  • Comparative baselines were reproduced under the evaluation protocol, with unavailable original weights or data treated as controlled reproductions.

B ADDITIONAL ABLATIONS

The additional ablations report evolution-round performance and the effect of advantage conditioning, with values summarized as mean ± standard deviation over three seeds.

  • Table 7 reports policy improvement across evolution rounds using mean ± standard deviation over three seeds.
  • Table 8 evaluates advantage conditioning using macro-average success over LIBERO-Plus and RoboTwin 2.0.Values are reported as mean ± standard deviation over three seeds.

B.2 VISUAL ENCODER CHOICE

The frozen visual encoder ablation compares DINOv2, V-JEPA 2.1, and LingBot-Vision on matched images, showing a performance–throughput trade-off.

  • V-JEPA 2.1 exceeds DINOv2 by 3.8 points and LingBot-Vision by 2.6 points on both LIBERO-Plus and RoboTwin 2.0.
  • DINOv2 has the highest training throughput at 1.31 steps/s, compared with 1.00 steps/s for V-JEPA 2.1.
  • Figure 5 shows source images and response maps from DINOv2, V-JEPA 2.1, and LingBot-Vision on matched inputs.

B.3 NUMBER OF FUTURE ANCHORS

The future-anchor ablation favors four trajectory-relative anchors, while adding more anchors reduces benchmark performance; the full configuration also benefits from advantage-conditioned policy evolution.

  • 73.2 on LIBERO-Plus and 60.0 on RoboTwin 2.0 are achieved with four anchors, versus 72.7 and 59.3 with six anchors and 71.9 and 58.5 with eight.
  • The full configuration reaches 73.2 on LIBERO-Plus and 60.0 on RoboTwin 2.0.
  • Relative to local plus multi-horizon supervision without ACP, the full configuration gains 2.5 and 3.2 points on LIBERO-Plus and RoboTwin 2.0.
  • Relative to local supervision with ACP, the full configuration gains 4.2 and 4.8 points on LIBERO-Plus and RoboTwin 2.0.

B.5 VALUE CRITIC ARCHITECTURE

The value-critic ablation compares a compact distributional token-fusion critic with scalar-regression and reproduced π∗ 0.6-style critics.

  • The distributional token-fusion critic uses 0.69M trainable parameters and reaches 73.2 final LIBERO-Plus success.
  • The scalar-regression token critic reaches 71.1 final LIBERO-Plus success with 0.59M trainable parameters.
  • The reproduced π∗ 0.6-style critic uses 682.4M trainable parameters and reaches 70.4 final LIBERO-Plus success.
Loading 2608.30378v2…