PaperlayerView exact revision on arXiv

Source-linked AI summary

PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Models

Davood Soleymanzadeh, Kaidi Zhang, Zhiyuan Zhang, Bihao Zhang, Xiao Liang, Yu She, Minghui Zheng

arXiv:2608.27609v1cs.ROcs.AI

TL;DR

VLAs often lack explicit reasoning over future task dynamics, despite its importance for fine-grained manipulation. PHR-VLA aligns VLA representations with privileged future latent dynamics during training, and reports improved success across benchmark and real-world settings. The framework preserves the original action-generation pipeline at inference, while its scope remains limited to representation shaping rather than test-time planning or correction.

  • Problem

    Most VLAs condition action prediction mainly on current observations rather than explicitly modeling future task dynamics needed for anticipatory manipulation.

  • Method

    PHR-VLA adds a lightweight auxiliary future head that aligns current VLA embeddings with latent dynamics extracted from future observations during training.

  • Results

    Patch-level wrist-camera supervision raises LIBERO average success from 84.1% to 88.4% and real-world disassembly success from 63.3% to 82.5%, while third-person supervision raises Meta-World performance from 56.7% to 57.8%.

  • Takeaways & Limitations

    Privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies.

  • Takeaways & Limitations

    PHR-VLA shapes representations during finetuning but is not an inference-time planner or world model and does not perform test-time correction.

Abstract

from arXiv · show

Vision-language-action models (VLAs) have shown strong promise for general-purpose robotic manipulation by mapping language instructions and vision observations directly to actions. However, most VLAs primarily condition action prediction on current observations and lack an explicit mechanism for reasoning over future task dynamics, which is particularly important for fine-grained, contact-rich manipulation. We present PHR-VLA, a framework that enables planning-horizon reasoning in VLAs through privileged latent representations of future dynamics. PHR-VLA introduces a lightweight auxiliary future head that, during training, aligns the VLA's internal representations with latent dynamics extracted from future observations. Evaluation results demonstrate that local, contact-centric, patch-level latent dynamics supervision from the wrist camera improves success rate on LIBERO from 84.1% to 88.4% and on real-world disassembly tasks from 63.3% to 82.5%. Patch-level supervision from a third-person camera also improves performance on Meta-World from 56.70% to 57.8%. These results demonstrate that privileged latent dynamics alignment provides an effective training signal for improving anticipatory reasoning in VLA policies. Project website: \href{https://davoodsz.github.io/PHR-VLA.github.io/}{https://davoodsz.github.io/PHR-VLA.github.io/}

I. INTRODUCTION

PHR-VLA addresses the limited future-dynamics reasoning of largely reactive VLAs through privileged latent supervision during training. Its local, patch-level supervision improves performance across benchmark and real-world manipulation settings without requiring explicit future rollouts at deployment.

  • Most VLAs predict actions from current observations without explicitly modeling future task evolution, limiting anticipation in temporally coordinated manipulation.
  • Dense observation histories add computational cost and inference latency, motivating more compact approaches to temporal context.
  • PHR-VLA aligns current VLA embeddings with latent representations of future task dynamics using a lightweight auxiliary future head during training.
  • PHR-VLA encodes future task dynamics without requiring explicit future rollouts or world-model inference during deployment.
  • 88.4% average LIBERO success versus 84.1%, 82.5% versus 63.3% on real-world disassembly, and 57.8% versus 56.7% on Meta-World are reported with patch-level supervision.

II. RELATED WORK

Related work improves temporal reasoning through memory, forecasting, and world models, but these approaches can increase inference cost or require explicit future generation. PHR-VLA instead transfers future-dynamics information through training-time latent alignment while preserving deployment efficiency.

  • Memory-based VLA methods add temporal context through observation histories or compact representations such as keyframes and point tracks.
  • Forecasting methods generate future-oriented language, feature, image, or visual-foresight representations before or during action generation.
  • World-model and video-prediction methods support planning by modeling environment evolution through predictive latents or future visual rollouts.
  • PHR-VLA uses privileged future frames and a training-time auxiliary objective, while its standard action head remains unchanged at inference.
  • Explicit forecasting and world-model approaches often require autoregressive reasoning, rollout generation, or additional inference modules that add computational cost and latency.

III. PHR-VLA

PHR-VLA is a training-time auxiliary-supervision framework that uses privileged future latent dynamics to improve planning-horizon reasoning in VLAs. It extends the standard observation-to-action setup with future-aware representation learning.

  • PHR-VLA uses privileged future latent dynamics to improve planning-horizon reasoning in VLA policies.
  • A standard VLA receives images, language, and proprioceptive state, then predicts an action chunk over planning horizon H.
  • The VLM backbone combines modality-specific visual and proprioceptive encodings with tokenized language to produce multimodal representations.
  • The action head processes backbone representations to parameterize an action distribution under the standard demonstration-training objective.

B. VLA Baseline

The baseline uses SmolVLA as a compact, computationally efficient VLA and supplies future observations as privileged training information. PHR-VLA supervises latent dynamics rather than static future visual content.

  • VLA Baseline: SmolVLA is a 0.45B VLA baseline combining a compact SmolVLM-2 backbone with a flow-matching action expert for continuous action chunks.
  • Planning Horizon Latent Dynamics: Future planning-horizon observations are encoded with a frozen visual encoder as privileged training information.
  • Planning Horizon Latent Dynamics: The main formulation supervises changes in planning-horizon visual latents rather than matching absolute future latents.
  • Planning Horizon Latent Dynamics: This dynamics target emphasizes scene evolution while reducing the need to predict static visual content already present at the current timestep.
  • Results on LIBERO: LIBERO success rates are reported across four task suites with 300 trials per suite, while some marked entries use different model scales and training setups.

D. Planning Horizon Reasoning

PHR-VLA adds a lightweight future head that predicts planning-horizon latent dynamics from action-token representations and aligns them with privileged targets during training.

  • PHR-VLA predicts planning-horizon latent dynamics from the VLA’s internal action-token representation using a lightweight future head.
  • The future head encourages planned-action representations to encode expected visual-scene evolution across the action horizon.
  • The alignment loss uses mean-squared error between predicted and privileged planning-horizon latent dynamics, weighted by λ.
  • Meta-World reports success rates across all difficulty levels, with some starred entries using different model scales, training setups, and computational budgets.

E. Inference

At inference, PHR-VLA discards the future head and executes the original SmolVLA action using only current-timestep observations.

  • At inference time, PHR-VLA discards the future head and executes the original SmolVLA action from current-timestep observations.

IV. RESULTS AND DISCUSSION

The evaluation examines whether privileged future-latent dynamics supervision improves VLA fine-tuning and how camera view, target granularity, latent formulation, and loss weight affect performance.

  • The study asks whether privileged future-latent dynamics supervision improves standard VLA fine-tuning.
  • Camera View: Camera-view ablations compare wrist-mounted, fixed-third-person, and multi-camera auxiliary supervision.
  • Baselines & Evaluation Protocol: The evaluation compares PHR-VLA with ACT, Diffusion Policy, Octo, DiT Policy, OpenVLA, TinyVLA, and SmolVLA.
  • Target Granularity: Target-granularity ablations evaluate varied supervision granularity across wrist-mounted, fixed-third-person, and multi-camera settings.
  • Baselines & Evaluation Protocol: LIBERO and Meta-World provide multi-task simulation evaluations involving varied task families, spatial layouts, objects, and goal conditions.
  • Latent Target Formulation: Absolute-latent and dynamics-latent supervision are compared as alternative planning-horizon targets.
  • Loss Weight: Auxiliary loss-weight ablations evaluate SmolVLA and PHR-VLA across multiple λ values.

B. Results

Across simulation and real-world evaluations, PHR-VLA improves baseline performance, with the strongest reported gains from fine-grained wrist-camera dynamics supervision.

  • 84.1% to 88.4%: PHR-VLA increases average LIBERO success rate over standard SmolVLA fine-tuning.
  • +8 points: PHR-VLA achieves its most pronounced LIBERO gain on LIBERO-Long.
  • PHR-VLA improves the overall average Meta-World success rate relative to the considered baseline.
  • Meta-World’s third-view camera limits fine-grained planning-horizon supervision during training.

C. Ablation Studies

Ablations show that PHR-VLA’s gains depend on the structure of privileged future supervision, with local patch-level, wrist-camera latent dynamics generally performing best.

  • Camera View: 88.4% success rate is achieved with patch-level wrist-camera latent-dynamics supervision, exceeding the 84.1% SmolVLA baseline on LIBERO.The wrist-camera configuration outperforms multi-camera supervision at 88.1% and third-person supervision at 86.8%.
  • Target Granularity: Patch vs. Mean: 88.4% versus 86.3% on the wrist camera shows that patch-level targets outperform mean-pooled targets.Patch-level supervision also exceeds mean pooling for multi-camera and third-person views.
  • Latent Dynamics vs. Absolute Latent Supervision: 88.4% versus 86.7% shows that latent-dynamics supervision outperforms absolute future-latent supervision with the wrist camera.The third-person comparison is narrower at 86.8% versus 86.7%.
  • Loss Weight λ: 85.8% to 88.4% is the average success-rate improvement when the auxiliary loss weight λ increases from 0.005 to 0.02.The study adopts λ = 0.02 as the default auxiliary loss.
  • Encoder Choice: SigLIP vs. JEPA: SigLIP reaches 88.4% versus 86.3% for V-JEPA 2 on LIBERO, while V-JEPA 2 reaches 59.8% versus 57.8% for SigLIP on Meta-World.On real-world disassembly, SigLIP reaches 82.5% average success versus 77.5% for V-JEPA 2.

D. Real-world Deployment

PHR-VLA is evaluated on contact-rich real-world disassembly using a Franka Panda, third-person and wrist-mounted RGB cameras, and synchronized teleoperation demonstrations.

  • Experimental Setup: A 7-DoF Franka Emika Panda with parallel grippers performs the real-world disassembly experiments.Visual perception uses a fixed third-person camera for global views and a wrist-mounted camera for close-range observations.
  • Disassembly Tasks: Contact-rich disassembly tasks require fine-grained adjustments for successful completion.Task-specific instructions are listed in Table V.
  • Dataset: 100 demonstrations per task are collected by human teleoperation, with synchronized RGB and language streams sampled at 10 Hz.Images come from fixed third-person and wrist-mounted cameras, while language provides task prompts.
  • Results and Analysis: 82.5% average success is achieved by PHR-VLA, exceeding SmolVLA at 63.3%, Diffusion Policy at 51.7%, and ACT at 35.8%.PHR-VLA (JEPA) achieves 77.5% average success.

V. CONCLUSIONS

The paper concludes that training-time privileged latent-dynamics supervision improves VLA success across simulated and real-world manipulation tasks, while remaining distinct from inference-time planning.

  • Framework: PHR-VLA aligns action-token representations with privileged latent dynamics during training, then removes the future head and auxiliary encoder at inference.This makes the framework a training-time planning-horizon supervision method rather than a deployment-time module.
  • Ablation Findings: Patch-level targets, wrist-camera supervision, and latent-dynamics targets outperform their respective alternatives across tested configurations.The conclusion attributes the gain to target structure rather than merely adding a future-prediction objective.
  • Limitations: PHR-VLA’s limitation is that it shapes representations during finetuning but does not provide inference-time planning, world-model inference, or test-time correction.The paper identifies tactile or force-aware targets and object-centric patch supervision as future directions.
Loading 2608.27609v1…