PaperlayerView exact revision on arXiv

Source-linked AI summary

ForeTime-VLA: Causal Future-Token Distillation from a World Action Model for Conveyor-Belt Manipulation

Siyuan Ma, Yutian Zhang, Boshi Zhang, Qinglian Wu, Jiaqi Zhai, Dong Wei, Xiaojin Huang

arXiv:2608.20735v1cs.AIcs.RO

TL;DR

Dynamic manipulation needs predictive contact timing, but deploying video-scale world-model inference is costly. ForeTime-VLA distills future-aware structure into a causal π0.5 policy, which improves offline and real-robot performance while keeping the teacher offline.

  • Problem

    Dynamic conveyor manipulation requires anticipating reachability, closure timing, and contact orientation, while current-observation VLA policies lack explicit future-structure supervision and deployed world-model imagination is costly.

  • Method

    ForeTime-VLA trains an eight-frame causal encoder to predict a future-aware action-equivalent teacher code, phase, and transition horizon that condition both π0.5 policy paths while preserving recorded action targets.

  • Results

    ForeTime-VLA achieves 81.1% stationary and 58.9% slow-moving grasp success, exceeding the next-best references by 12.2 and 22.2 percentage points.

  • Takeaways & Limitations

    The results support causal future-token distillation as a way to retain useful world-model foresight for dynamic manipulation without deploying the video-scale teacher.

Abstract

from arXiv · show

Manipulating moving objects requires a policy to anticipate contact events, yet vision-language-action (VLA) policies are commonly fine-tuned from the current observation alone. World action models (WAMs) learn predictive dynamics, but running a video-scale teacher or explicitly imagining future frames at deployment is costly. We introduce ForeTime-VLA, a dense pi0.5 policy that distills a future-aware, action-equivalent representation from a frozen Fast-WAM-derived teacher while remaining causal at inference. Offline, current and future video latents are compressed into a whitened 64-D target. Online, an eight-frame history encoder predicts this target together with manipulation phase and normalized time-to-transition. Four future tokens and one phase token condition the VLM prefix, while the predicted future and transition horizon condition the action expert. Training retains the original flow-matching action target and adds cosine, relational geometry, phase, time-to-transition, and action-equivalence objectives. On a deduplicated conveyor-belt dataset, we compare 40k-step checkpoints on 768 matched windows per split. Test MAE decreases from 0.134119 to 0.130593 (2.63%; paired-bootstrap 95% CI: 0.82-4.48% improvement), and test L2 decreases by 3.02%, at a 2.46-2.93% latency cost. In quantitative real-robot evaluation, ForeTime-VLA achieves 81.1% stationary and 58.9% slow-moving grasp success, exceeding the next-best reference by 12.2 and 22.2 percentage points, respectively. Across three belt speeds, it completes 44/90 grasps versus 23/90 for pi0.5, including 11/30 versus 2/30 at fast speed. The agreement between offline orientation gains and reduced real-robot contact-pose failures supports causal future-token distillation as an effective way to improve dynamic manipulation without deploying the world-model teacher.

I. INTRODUCTION

ForeTime-VLA addresses dynamic conveyor manipulation by distilling future-conditioned structure from a world action model into a causal VLA without future frames or teacher inference at deployment. The method augments π0.5 with future, phase, and transition-horizon predictions while retaining recorded action targets.

  • Dynamic conveyor manipulation requires anticipating object reachability, gripper closure timing, and contact orientation beyond the current visual observation.
  • ForeTime-VLA compresses current and future video latents into an action-equivalent teacher code, then trains an eight-observation causal encoder to predict it.
  • Future, phase, and time-to-transition predictions condition both the slow π0.5 VLM prefix and fast action-expert path.
  • The evaluation pairs offline policy metrics with quantitative real-robot studies across stationary, moving, and speed-stress conditions.
  • The training design combines action flow with future alignment, relational geometry, transition, phase, and auxiliary action-reconstruction objectives.

II. RELATED WORK

Related work spans generalist VLA control, predictive world models, and knowledge distillation. ForeTime-VLA combines these directions by transferring a compact action-equivalent future code into a pretrained causal VLA rather than deploying a video-scale WAM.

  • VLA systems transfer vision-language priors to robot control across tasks, scenes, embodiments, and action-decoding regimes.
  • Latent world models and video-conditioned policies use predictive representations or imagined futures to support planning and control.
  • ForeTime-VLA distills a 64-D future-aware code from privileged WAM information into a pretrained VLA using compact causal histories.
  • Its distillation combines instance-level future cosine alignment with batch-level Gram-matrix geometry, alongside variance and covariance constraints.

A. Problem Formulation

The formulation preserves π0.5’s conditional flow-matching action target while adding an offline action-equivalent teacher and cached future-code supervision. The teacher uses privileged future features, whereas deployment uses only causal inputs.

  • A. Problem Formulation: At each time step, π0.5 receives RGB observations, language, robot state, and an eight-step causal history, and predicts a 20 × 16 action chunk.
  • A. Problem Formulation: The noisy-action and target-flow construction remains unchanged, so future-sensitive conditioning is added without replacing the recorded action target or using future observations at inference.
  • B. Offline Action-Equivalent Future Teacher: The offline teacher samples the present and eight future offsets, encodes them with a frozen Wan2.2 video VAE, and produces current and future 96-D features.
  • B. Offline Action-Equivalent Future Teacher: A non-collapsed adapter maps current feature, future feature, and 26-D state to a 64-D teacher code while a causal auxiliary encoder uses current feature and state.
  • B. Offline Action-Equivalent Future Teacher: Shared action decoding preserves action relevance, while future reconstruction, teacher–causal alignment, variance, covariance, and pairwise geometry regularize the representation.
  • B. Offline Action-Equivalent Future Teacher: Only the per-dimension whitened teacher output is cached for causal-student supervision; the privileged video and teacher are excluded from deployment.

C. Causal Temporal Student

The causal temporal student summarizes eight past observations into predictions of future state, manipulation phase, and time to transition. These outputs provide deployable temporal structure without requiring future observations.

  • The deployment history contains eight normalized robot states and visual statistics, while full-resolution images continue through the standard π0.5 vision backbone.
  • The temporal encoder combines state and visual inputs with learned temporal embeddings before residual-MLP processing.
  • Linear heads predict a 64-D compressed future state, four manipulation-phase logits, and normalized time-to-transition.
  • Phase labels derive from gripper-state transitions, while time-to-transition is the clipped, horizon-normalized distance to the next transition.

D. Dual-Path Conditioning

ForeTime-VLA injects predicted future information into both the slow VLM pathway and the fast action-expert pathway while preserving the pretrained policy interface.

  • Dual-Path Conditioning: The slow path maps the predicted future code to four 2048-D future tokens plus a phase-derived fifth token appended to the π0.5 prefix.The expected embedding under the predicted phase distribution supplies the fifth token.
  • Dual-Path Conditioning: The fast path maps predicted future state and transition horizon to a 1024-D residual added to every action-expert suffix token and its adaptive normalization condition.This conditions the action expert with both future representation and timing information.
  • Dual-Path Conditioning: Learned scalar gates initialize both injections at 0.05, limiting early disruption to the pretrained policy.
  • Dual-Path Conditioning: The added modules contain 1.251M parameters, while the Fast-WAM teacher, video VAE, and offline adapter are absent at deployment.

E. Training Objective

Training combines the original action-flow objective with future-state alignment, phase, transition-time, and action-equivalence supervision, while jointly optimizing π0.5 and the added modules.

  • Training Objective: Five functional objectives supervise action flow, transition timing, future-state alignment, auxiliary action reconstruction, and manipulation phase.The future-state term includes both pointwise and relational components.
  • Training Objective: Pointwise cosine and relational geometry losses align predicted and teacher future codes at both instance and batch-structure levels.
  • Training Objective: Cross-entropy supervises phase, Huber loss with δ = 0.1 supervises transition time, and a linear decoder reconstructs the normalized action chunk.
  • Training Objective: The total objective retains Lflow and adds 0.04Ltransition and 0.05Lae alongside cosine, geometry, and phase terms.
  • Training Objective: All π0.5 parameters and ForeTime-VLA modules are optimized jointly, and the recorded action target is never replaced by a teacher action.

A. Dataset

The study uses a deduplicated conveyor-belt corpus and matched evaluation protocol to compare 40k-step π0.5 checkpoints with a cross-family GR00T reference across offline and real-robot settings.

  • Dataset: The corpus contains 458 valid episodes and 96,512 frames after removing one unreadable file and 28 exact duplicates.Episodes cover boxes, corn, and watermelons under varied belt speeds, viewpoints, and scene layouts.
  • Dataset: The deterministic duplicate-free split assigns 402/31/25 episodes to train, validation, and test, yielding 77,125/6,075/4,610 valid action windows.
  • Models and Protocol: The baseline and ForeTime-VLA are fully dense fine-tunes from the same π0.5 initialization and are evaluated at step 40k.The baseline uses the original current-observation interface; ForeTime-VLA adds temporal modules and cached auxiliary supervision.
  • Models and Protocol: A public GR00T N1.6-3B checkpoint is additionally fine-tuned for exactly 10,000 optimizer steps using the same 402 training episodes.Its protocol freezes the vision encoder while training the action head, projectors, and top four language layers.
  • Models and Protocol: Each split is evaluated on 768 matched windows using identical episode/start indices and raw target action chunks, reporting MSE, MAE, and unnormalized 16-D action-vector L2.Confidence intervals use 20,000 paired bootstrap resamples, and latency is measured on the same A100.
  • Evaluation: Real-robot evaluation includes 90 trials per method for stationary and slow-moving tasks, plus 30 trials per method at each of three belt speeds.Moving-task outcomes are partitioned into success, early failure, late failure, and contact-pose error.

V. RESULTS

ForeTime-VLA improves offline action reconstruction over π0.5 and achieves the lowest MAE and L2 across the evaluated VLA policies, including cross-family references.

  • Aggregate Reconstruction: 2.63%: test MAE improves for ForeTime-VLA versus the π0.5 baseline, while test L2 improves by 3.02%.Validation MAE improves by 2.96%, with 95% intervals above zero for validation and test MAE.
  • Cross-Family References: 13.46%: GR00T’s test MAE is higher than the π0.5 baseline, with a paired-window bootstrap interval of [9.97, 17.14]%.GR00T also has higher error than both 40k-step π0.5 models on all six point estimates.
  • Cross-Family References: ForeTime-VLA achieves the lowest MAE and L2 on both splits across all five evaluated policies.The comparison includes the π0.5 baseline, the ForeTime-VLA checkpoint, and cross-family references reported in the evaluation.

C. Action-Group Analysis

ForeTime-VLA’s gains concentrate in end-effector orientation, while inference-time ablations show that future, phase/time, and dual-path conditioning each contribute to transition accuracy. The added conditioning improves action accuracy with a small latency increase.

  • Action-group results: Test end-effector RPY MAE improves by 3.78%, while end-effector position improves by 1.03%.Validation RPY MAE improves by 4.12%; arm error also improves on both splits.
  • Inference-time ablation: 10.1% higher transition MAE follows removal of the future condition.
  • Inference-time ablation: 10.6% higher transition MAE follows removal of phase/time conditioning.
  • Inference-time ablation: Restricting conditioning to either path degrades transition MAE, supporting complementary slow- and fast-path use.The intervention study uses shared checkpoint weights and masks components only at inference.
  • Efficiency: Test latency rises from 66.97 to 68.62 ms, a 2.46% increase.Validation latency rises from 66.69 to 68.65 ms (+2.93%).

F. Quantitative Real-Robot Evaluation

ForeTime-VLA achieves the strongest reported real-robot grasp success in stationary and slow-moving tasks, with larger advantages as motion increases. Its speed-stress results and failure profile indicate particular benefits for difficult interception and contact-pose conditions.

  • Stationary and slow-moving tasks: 81.1% stationary grasp success exceeds the next-best StarVLA result by 12.2 percentage points.ForeTime-VLA succeeds in 73/90 trials.
  • Stationary and slow-moving tasks: 58.9% slow-speed moving grasp success exceeds the next-best π0.5 baseline by 22.2 percentage points.ForeTime-VLA succeeds in 53/90 trials.
  • Failure analysis: Early, late, and contact-pose failures decrease by 15.8%, 37.5%, and 57.1%, respectively, relative to π0.5.ForeTime-VLA has the best rate in all three moving-task failure categories.
  • Speed stress test: 44/90 three-speed grasps succeed for ForeTime-VLA versus 23/90 for π0.5.Success counts are 19 versus 13 at slow speed, 14 versus 8 at medium speed, and 11 versus 2 at fast speed.
  • Speed stress test: The absolute margin widens from six to nine successful trials as speed increases from slow to fast.ForeTime-VLA drops from 19 to 11 successes, while π0.5 drops from 13 to 2; deployment uses no future frames or WAM forward pass.

H. Temporal-Task Contribution

ForeTime-VLA makes temporal structure explicit through causal predictions of future state, manipulation phase, and transition timing, while conditioning both policy paths without deploying privileged video or the WAM. Its advantage grows with motion difficulty and aligns with orientation and event-level evidence.

  • Temporal-task contribution: An action-equivalent future code, phase, and time-to-transition are predicted from eight past observations.The code conditions the slow VLM path as future tokens and the fast action path as a residual condition.
  • Temporal-task contribution: The stationary-to-slow-motion advantage over π0.5 grows from 15.6 to 22.2 percentage points.The advantage widens further at fast speed.
  • Evidence for the mechanism: The unchanged recorded flow-matching target rules out teacher-action replacement as an explanation for the improvement.Event-level ablations, orientation gains, and speed-dependent robot outcomes support predictive temporal conditioning as the source.
  • Qualitative rollout: At 9.8 s, the phase head assigns 87.3% probability to grasp before transport probability rises.The rollout shows an approach–grasp–transport sequence rather than a static scene cue.
  • Overall contribution: The method reduces test MAE/L2 by 2.63%/3.02% with under 3% latency overhead.The conclusion reports 81.1% stationary and 58.9% slow-moving success, plus 44/90 versus 23/90 three-speed trials.
Loading 2608.20735v1…