PaperlayerView exact revision on arXiv

Source-linked AI summary

World-Calibrated Proposal-to-Action Flow for Vision-Language-Action Models

Jie He, Wei Li, Junwen Tong, Rui Shao, Wei-Shi Zheng, Liqiang Nie

arXiv:2610.02323v1cs.ROcs.CV

TL;DR

Flow-based VLA policies typically generate actions from generic sources that do not use recent motion or explicitly calibrate proposals against predicted futures. PROACT forms a scene-aware motion proposal and uses prospective world reasoning to calibrate a proposal-centered source for action generation. It improves performance across simulation and real-world tasks while reducing denoising steps by up to 50%.

  • Problem

    Flow-based VLA decoders typically initialize action chunks from generic sources that ignore available recent motion, while prospective world reasoning is not explicitly used to calibrate an action hypothesis against anticipated task evolution.

    Source: §1 · p. 1

  • Method

    PROACT forms a scene-aware motion hypothesis from recent actions and uses prospective world reasoning to calibrate a proposal-centered source for flow-based action generation.

    Source: §1 · pp. 2–3

  • Results

    PROACT consistently outperforms strong baselines across simulation and real-world manipulation while reducing denoising steps by up to 50%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PROACT provides an interface between local motion continuity and future-aware action generation by treating recent motion as a soft hypothesis calibrated with prospective reasoning.

    Source: §1 · p. 3

    The World Expert targets the demonstrated future rather than the counterfactual outcome of executing the proposal, which remains a soft motion prior.

    Source: §2.3 · p. 5

Abstract

from arXiv · show

Flow-based Vision-Language-Action (VLA) policies generate action chunks by transporting samples from a task-agnostic isotropic Gaussian source. As this source is conditioned on neither recent execution nor predicted future evolution, (i) it discards the local continuity established by recently executed motion. (ii) Even when predictive world representations are introduced, they often only condition the transport dynamics rather than determine where generation starts, how far it may deviate, or along which action directions it may expand. Building on this observation, we introduce ProAct, a world-calibrated proposal-to-action framework that makes the generative source itself predictable. (i) To preserve motion continuity, a lightweight Proposal Expert converts recent actions into a scene-aware hypothesis via one motion-anchored endpoint flow-matching step, initializing generation near the demonstrated action manifold. (ii) To jointly capture intended scene evolution and proposal-future compatibility, a prospective World Expert treats the hypothesis as a soft motion prior while predicting the task-consistent latent future. (iii) From this compatibility, the model calibrates a proposal-centered anisotropic source, where a bounded per-step extent controls the allowed deviation and a trace-normalized low-rank geometry under a condition-number budget allocates refinement over coupled translation, rotation, and gripper directions. Compared with $\pi_{0.5}$, ProAct improves performance across simulation and real-world tasks while reducing denoising steps by 50%, inference latency by up to 25.8%, and increasing throughput by up to 34.8%.

1 INTRODUCTION

Flow-based VLA policies commonly start from generic sources that ignore recent motion and do not explicitly calibrate action hypotheses against predicted futures. PROACT proposes, calibrates, and refines actions using recent motion and prospective world reasoning to shape its source distribution.

1 INTRODUCTION

  • Generic action sources ignore recent execution, while recent motion alone can be unreliable at contact, reorientation, gripper actuation, and task-phase transitions.Recent motion is therefore a proposal to evaluate against the intended future, not the action itself.Source: §1 · pp. 1–2
  • PROACT uses a lightweight Proposal Expert to form a scene-aware motion hypothesis, then calibrates it against the task-consistent future before refining it into actions.The proposal is initialized through a single motion-anchored endpoint flow step and remains a soft motion prior.Source: §1 · p. 2, Figure 2 · p. 3
  • The prospective World Expert uses motion to assist future prediction while anchoring the desired future independently of proposal correctness, and predicts per-step refinement extent and geometry.These outputs encode proposal–future compatibility for subsequent refinement.Source: §1 · pp. 2–3
  • PROACT replaces the isotropic Gaussian source with a world-calibrated, proposal-centered source while preserving the base VLA’s transport parameterization.Its bounded extent controls deviation, while trace-normalized low-rank geometry allocates correction across coupled action directions.Source: §1 · pp. 2–3
  • PROACT consistently outperforms strong baselines across simulation and real-world manipulation while reducing denoising steps by up to 50%.The reported evaluation supports both effectiveness and inference-efficiency gains.Source: §1 · p. 3

2 PROACT

PROACT follows a propose–calibrate–refine pipeline: recent motion becomes a scene-aware hypothesis, prospective world reasoning evaluates it, and calibration defines the source for action generation.

2.1 PROBLEM FORMULATION

  • The policy predicts action chunks from visual-language context and executed action history, factorizing generation into proposal, future modeling, source calibration, and action generation.The source remains anchored to the proposal while calibration determines deviations around it.Source: §2.1 · p. 3

2.2 MOTION-INFORMED ACTION PROPOSAL

  • A motion-smoothed extrapolation initializes a shallow Proposal Expert, which grounds the prior in visual-language context with one endpoint flow step rather than producing an executable action.The motion prior captures local continuity but is not scene-aware, leaving contact and target changes to visual grounding.Source: §2.2 · p. 4

2.3 ASYMMETRIC DESIRED-FUTURE MODELING AND PROPOSAL CALIBRATION

  • The World Expert jointly reasons over proposal and scene semantics, while its future target remains the demonstrated task-consistent future rather than the proposal’s counterfactual outcome.The proposal therefore remains a soft motion prior instead of redefining the desired future to justify itself.Source: §2.3 · pp. 4–5

2.4 WORLD-CALIBRATED ACTION REFINEMENT

  • Proposal–future compatibility sets per-step refinement extent, while low-rank geometry allocates correction across coupled action dimensions such as translation, rotation, and gripper.Trace normalization preserves average variance as geometry redistributes it; a condition-number budget limits shape degeneracy.Source: §2.4 · p. 5, Figure 3 · p. 5

Appendix E.2 details sampling and covariance construction.

  • The calibrated covariance sets preferred correction axes, while the Action Expert supplies signed transport direction and converts the source into executable actions.Starting closer to the action manifold allows a coarser integration grid, reducing sampling cost and accumulated integration error.Source: Appendix E.2 · p. 6

3 EXPERIMENTS

The experiments compare PROACT with strong baselines across simulation and physical manipulation, then examine component contributions, qualitative refinement, and inference efficiency.

3.2 COMPARISON ON REAL-WORLD TASKS

  • On GALAXEA, overall TSR/CSR rises by 8.7/6.4 points; on AgileX, by 10.0/8.3 points over π0.5.Proposals preserve stable phases while calibration handles contact and target transitions during execution.Source: Table 3 · p. 7, §3.2 · pp. 7–8

3.1 COMPARISON ON SIMULATION BENCHMARKS

  • LIBERO reaches 98.4% average success, 1.5 points above its backbone; LIBERO-Plus reaches 86.8%, 13.2 points above π0.5.On LIBERO Long, the margin grows to 3.6 points; LIBERO-Plus gains most on Noise, Camera, and Robot shifts.Source: §3.1 · p. 7
  • On RoboTwin 2.0, PROACT averages 60.3%, 31.7 points above π0.5 and 25.7 above the strongest competing average.The policy is evaluated zero-shot on randomized hard rollouts across 50 tasks.Source: §3.1 · p. 7

3.4 QUALITATIVE ANALYSIS

  • On LIBERO/real robots, calibrated transport reduces latency by 25.8/16.3%, raises throughput by 34.8/19.5%, and improves success by 1.5/8.7 points.The reported efficiency comparison attributes shorter transport to starting near the demonstration and focusing corrections along task-relevant directions.Source: Table 7 · p. 8, §3.4 · p. 9

3.3 ABLATION STUDY

  • Motion extrapolation gains 0.6/3.3 points on LIBERO/real-world tasks; Proposal Expert grounding adds 0.6/3.4 points, while last-chunk reuse trails Gaussian noise.The ablation supports recent motion as a soft prior that needs scene grounding when continuity is unreliable.Source: §3.3 · p. 8
  • Removing the World Expert causes the largest ablation drop: 1.9 points on LIBERO and 8.0 points in the real world.Removing proposal conditioning, future supervision, or source calibration costs 1.3/5.4, 0.9/4.0, and 0.6/2.7 points, respectively.Source: §3.3 · p. 8
  • Adaptive extent and low-rank geometry improve success by 0.9/3.3 and 0.4/2.0 points on LIBERO/real-world tasks; together, gains reach 1.5/6.0 points.Extent controls correction magnitude, while geometry allocates correction across coupled action dimensions.Source: §3.3 · p. 8

3.4 QUALITATIVE ANALYSIS

  • Proposal Expert attention focuses on manipulated objects; calibrated samples encode action-dimension correlations, and refinement bends incompatible dimensions toward the demonstration.Stable dimensions remain close to the proposal while refinement preserves its coarse direction.Source: §3.4 · pp. 8–9

A.2.2 TRAINING CONFIGURATION

PROACT uses a shared optimizer and peak learning rate across settings, while adapting training schedules and batch sizes. Most calibration and architecture settings are shared, with action-history and world-query settings adjusted by benchmark.

A.2.2 TRAINING CONFIGURATION

  • The optimizer and peak learning rate are shared across LIBERO, RoboTwin 2.0, and real-world experiments, while training schedules and batch sizes vary by setting.The supplied passage names Tab. 8 as the configuration summary.Source: §A.2.2 · p. 15, Table 8 · p. 16
  • Action-history length matches the action horizon, with H = 10 for LIBERO and H = 50 for RoboTwin 2.0 and real-world experiments.The real-world configuration otherwise follows RoboTwin 2.0.Source: §A.2.2 · p. 16
  • Most architecture and calibration hyperparameters are shared, while the number of world queries is Nz = 16 for LIBERO and Nz = 4 for RoboTwin 2.0 and real-world experiments.The shared settings include Proposal Expert depth, geometry rank and budget, extent threshold, and momentum coefficients.Source: §A.2.2 · p. 16

B SUPPLEMENTARY ROBOTWIN 2.0 RESULTS

Under the Randomized (Hard) setting, PROACT leads the 50-task RoboTwin 2.0 benchmark in average success rate. Its gains vary by task, with larger improvements on several challenging tasks.

B SUPPLEMENTARY ROBOTWIN 2.0 RESULTS

  • 47.06% average success rate across 50 tasks exceeds π0.5's 43.84% by 3.22 percentage points.PROACT also exceeds Abot-M0 (30.4%), Spatial Forcing (26.7%), X-WAM (25.8%), and X-VLA (20.9%).Source: B SUPPLEMENTARY ROBOTWIN 2.0 RESULTS · p. 16, Table 9 · p. 17
  • PROACT improves over π0.5 on 22 tasks, ties on four, and is best or tied-best among compared methods on 19 tasks.The benchmark uses 100 rollouts per task under the Randomized (Hard) setting.Source: B SUPPLEMENTARY ROBOTWIN 2.0 RESULTS · p. 16
  • Figure 8 shows representative successful rollouts across six tasks, with GALAXEA R1 Lite above AgileX Cobot Magic in each panel.Snapshots are ordered left to right and selected at semantic milestones rather than uniform time intervals.Source: Figure 8 · p. 18
  • The listed gaps over π0.5 include 60 percentage points on Put Bottles Dustbin and 41 on Open Laptop.The reported rates are 69% vs. 9% for Put Bottles Dustbin and 76% vs. 35% for Open Laptop.Source: B SUPPLEMENTARY ROBOTWIN 2.0 RESULTS · p. 18

C.1 EVALUATION PROTOCOL AND METRICS

The evaluation uses fixed, shared trial conditions and prohibits human correction after a trial begins. Checkpoint metrics enforce task order and distinguish full completion from partial progress.

C.1 EVALUATION PROTOCOL AND METRICS

  • All methods share task instructions, sensing, camera placement, control interface, and randomized initialization, with 25 independent trials per task unless stated otherwise.Initial states are sampled from task-specific admissible ranges fixed before evaluation and shared across methods; no human correction is allowed after a trial starts.Source: §C.1 · p. 18
  • A checkpoint counts progressively only when all preceding stages are completed.The protocol defines trial-level checkpoint attainment and evaluates checkpoint j through a progressive indicator.Source: §C.1 · p. 19
  • Task Success Rate (TSR) measures full-task completion, while Cumulative Success Rate (CSR) measures average checkpoint progress.CSR captures how far unsuccessful trials progress; aggregate suite- and platform-level values average task-level metrics.Source: §C.1 · p. 19

C.2 TASK DESCRIPTIONS

The real-world evaluation covers six sequential manipulation tasks on GALAXEA R1 Lite and AgileX Cobot Magic. The task set tests varied object, container, drawer, sorting, and contact-rich interactions, and the rollout figures provide qualitative examples across the platforms.

C.2 TASK DESCRIPTIONS

  • The six tasks span repeated sorting, pot-lid retrieval, drawer manipulation and retrieval, and table wiping across two robot platforms.The suite uses identical semantic completion criteria across embodiments and tests visual grounding, grasping and placement, phase transitions, and robustness to accumulated execution error.Source: §C.2 · p. 19
  • Block Sorting, Pot Lid Retrieval, Drawer Manipulation, and Drawer Toy Retrieval require ordered multi-stage completion, including re-grounding or handling changing access to objects.The tasks include sequential placements, retrieval after opening a covered container, and drawer opening, object transfer, and closure.Source: §C.2 · p. 19
  • Table 10 reports successful trials out of 25 at each progressive checkpoint for six AgileX Cobot Magic tasks.CSR is the mean success rate across a task's checkpoints, and its final checkpoint equals task TSR.Source: Table 10 · p. 20
  • Table Wiping tests cloth grasping and sustained surface contact, while Color Sorting tests repeated language-conditioned placement into matching bins.Table Wiping has two progressive stages; Color Sorting has three.Source: §C.2 · p. 20

C.3 QUALITATIVE REAL-WORLD ROLLOUTS

  • Figure 8 presents representative successful rollouts on both platforms, with snapshots ordered left to right and selected at semantic milestones.The sequences show re-grounding after scene changes, sustained wiping contact, and repeated object localization and placement.Source: §C.3 · p. 20

C.4 DETAILED REAL-WORLD RESULTS AND ABLATIONS

Across six real-world tasks, PROACT’s advantage is concentrated at later checkpoints, where re-grounding is needed after scene changes. The checkpoint and ablation results show that the full method retains more trials through transitions, with the World Expert contributing across the sequence.

C.4 DETAILED REAL-WORLD RESULTS AND ABLATIONS

  • On Drawer Toy Retrieval, PROACT loses two trials at Open →Retrieve (21 →19), versus four for π0.5 (20 →16) and six for OpenVLA-OFT (14→8).The same pattern produces the largest suite-wide CSR gaps, +10 points on both drawer tasks.Source: §C.4 · pp. 20–21
  • PROACT attains the highest per-checkpoint success in all six GALAXEA R1 Lite tasks, with its margin widening at later transition and terminal checkpoints.More trials reach the terminal checkpoint on all six tasks, narrowing the CSR–TSR gap.Source: Figure 9 · p. 21, §C.4 · p. 21
  • Across four-checkpoint tasks, PROACT loses at most three trials from first to last checkpoint, versus five for π0.5 and six or more for weaker baselines.Terminal contact-rich stages improve while preceding stages are near-saturated.Source: §C.4 · pp. 21–22
  • Removing any single component leaves initial checkpoints essentially unchanged but erodes later ones; removing the World Expert also lowers early performance and yields the largest aggregate loss, −8.0 points.The World Expert ablation decays fastest across the sequence.Source: §C.4 · p. 22
  • Extent Only reaches 76.0 real-world SR versus 74.7 for Geometry Only, while only their combination recovers PROACT across the full checkpoint sequence.Both partial variants match PROACT on initial grasps but trail at later checkpoints.Source: §C.4 · p. 22

D EXTENDED QUALITATIVE ANALYSIS

Across additional examples and execution phases, calibration changes the proposal’s support according to phase and action dimension. Refinement remains on reliable proposal segments and bends toward the demonstration only where support widens.

D EXTENDED QUALITATIVE ANALYSIS

  • Across additional cases, calibrated support widens along steps and dimensions where the proposal must leave the momentum prior, then tightens when proposal and demonstration agree.The refined trajectory stays on reliable proposal segments and bends toward the demonstration only inside widened support.Source: D EXTENDED QUALITATIVE ANALYSIS · pp. 22–23

E DERIVATIONS

The derivation states parameter and noise assumptions for Equation 1, including independent Gaussian noise across sources and horizon steps unless otherwise noted.

E DERIVATIONS

  • Equation 1 assumes 0 ≤ β, λ < 1, τu > 0, 0 < σmin ≤ σmax < ∞, da > 1, 1 ≤ r < da, and 0 ≤ Smax < ∞.H is even when temporal tubelets are used.Source: E DERIVATIONS · p. 23
  • Gaussian noises are mutually independent across sources and horizon steps unless stated otherwise.Deterministic modules correspond to Dirac conditional kernels.Source: E DERIVATIONS · p. 23
  • The displayed Equation 1 suppresses perception–language context, and its stop-gradient operator is identity in the forward pass with zero backward derivative.Source: E DERIVATIONS · p. 23

E.1 THREE-LAYER KV FUSION: RISK-OPTIMAL WINDOW AND WEIGHTS

The derivations characterize a risk-optimal three-layer fusion window and weights under stated local assumptions, then constrain the calibrated source to avoid degeneracy. The resulting window and weights remain conditional on the model assumptions and implementation choices.

E.1 THREE-LAYER KV FUSION: RISK-OPTIMAL WINDOW AND WEIGHTS

  • Under the stated local quadratic-mean and noise assumptions, the derivation evaluates fusion risk over windows and derives the window at uniform weights and weights at fixed m = 1.The estimand is the smooth context, not the observed value; omitted deterministic higher-order terms contribute bias rather than noise.Source: §E.1 · p. 23
  • The risk values R(0) = σ2, R(1) = σ2/3 + κ2/9, and R(2) = σ2/5 + κ2 imply a unique m⋆ = 1 within the stated open interval, with ties at endpoints.The interval characterization is conditional, and one forward pass does not identify κ2/σ2.Source: §E.1 · pp. 23–24
  • The implemented weights use local stationarity on the three-layer window because the optimum requires unknown κ and symmetric neighbors.The resulting estimate is approximately unbiased only when the stated bias terms are small relative to layer variances.Source: §E.1 · p. 24

E.2 CALIBRATED SOURCE: WELL-POSEDNESS

  • The unconstrained pointwise calibrated-source objective is unbounded below; the following projection removes this degeneracy without assuming enough empirical-head capacity to align with each residual.Source: §E.2 · p. 24
  • For da = 7 and Smax = 5, the bound is −0.991; negative losses remain possible, but divergence to −∞ is excluded.Equality requires zero residual, S = Smax, and one nonzero singular direction; a nonzero residual has a strictly higher sample-wise minimum.Source: §E.2 · p. 25

F LIMITATIONS AND FUTURE WORK

ProAct has two stated limitations: calibration lacks explicit counterfactual supervision, and prospective reasoning uses a fixed horizon aligned with the action chunk.

Counterfactual supervision

  • Training pairs expert actions with the futures that occurred, so calibration does not directly compare proposal-induced consequences with the desired future.The authors suggest proposal perturbations, simulator rollouts, or action-conditioned world models as sources of explicit counterfactual supervision.Source: F LIMITATIONS AND FUTURE WORK · p. 25

Fixed-horizon prospective reasoning

  • The fixed prospective horizon may not capture both short-term interaction cues and consequences that emerge only after several action chunks.The authors propose multi-scale or adaptive-horizon world reasoning as a possible extension.Source: F LIMITATIONS AND FUTURE WORK · p. 25

Related papers

Loading 2610.02323v1…