PaperlayerView exact revision on arXiv

Source-linked AI summary

World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation

Yuhao Pan, Haosong Peng, Zhengshen Zhang, Zhengyang Yan, Yalun Dai, Fushuo Huo, Chujie Wang, Tianyu Qi, Xiucheng Wang, Nan Cheng, Wenchao Xu

arXiv:2608.05369v2cs.ROcs.CV

TL;DR

VLA models often treat main-view and wrist-view observations as parallel inputs, despite their distinct roles in global task context and local interaction dynamics. W2-VLA predicts task-conditioned future wrist latents to inform action prediction, and reports strong results across benchmark and real-world tasks, including 84.0% zero-shot success on LIBERO-Plus.

  • Problem

    VLA models often fuse main- and wrist-view observations as parallel inputs without explicitly modeling the wrist view’s action-proximal role in fine-grained manipulation.

    Source: §1 · pp. 1–2

  • Method

    W2-VLA uses task-conditioned latent tokens and structured W2-CoT supervision to guide prediction of future wrist latents from wrist history for action generation.

    Source: §1 · pp. 2–3

  • Results

    W2-VLA reports strong performance across single-arm, bimanual, benchmark, and real-world evaluations, including 84.0% zero-shot success on LIBERO-Plus.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The policy requires no CoT generation at inference and enables real-time action generation above 80 Hz.

    Source: §1 · p. 3, §5 · p. 11

Abstract

from arXiv · show

Vision-language-action (VLA) models often treat main-view and wrist-view observations as parallel visual inputs, overlooking their distinct roles in robot manipulation. Fine-grained manipulation, however, benefits from anticipating how wrist-local interactions may evolve under the global task context. To address this limitation, we present World-to-Wrist VLA (W2-VLA), a VLA model for fine-grained robot manipulation with task-conditioned future wrist modeling. Given current multi-view observations and a task instruction, W2-VLA contextualizes a set of latent modeling tokens as a compact interface between the vision-language model and the wrist predictor. Conditioned on this interface and the observed wrist history, the predictor forecasts future wrist latents, which are transformed into future-aware context for action prediction. In addition, we introduce W2-CoT, a synthesis pipeline that produces structured annotations describing manipulation progress, physical transition cues, and wrist-local evidence. These annotations provide auxiliary supervision that shapes the task-conditioned latent interface. Experiments on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world manipulation tasks demonstrate improved fine-grained and contact-sensitive manipulation across single-arm and bimanual settings, while maintaining real-time action generation above $80$~Hz.

1 Introduction

W2-VLA uses global task context to guide prediction of future wrist-local dynamics, addressing the distinct roles of main and wrist views in fine-grained manipulation. The paper reports strong results across simulated and real-world settings.

Introduction

  • Main views provide global task context, while wrist observations reveal rapidly changing gripper–object interactions, making their coordination critical for fine-grained manipulation.Future wrist latents offer an action-proximal target for local interaction changes without reconstructing pixel-level appearance.Source: §1 · pp. 1–2
  • Because similar wrist histories can imply different futures, W2-VLA uses a compact task-conditioned interface, shaped by W2-CoT annotations of progress, physical transitions, and wrist-local evidence.The interface links VLM-encoded global context to future wrist prediction.Source: §1 · pp. 2–3
  • W2-VLA contextualizes fixed-length modeling tokens from current views and instruction, predicts future wrist latents from that interface and wrist history, then uses future-aware context for action prediction.Inference needs neither future wrist observations nor autoregressive W2-CoT decoding, and action generation exceeds 80 Hz.Source: §1 · pp. 2–3
  • 98.5% average success on LIBERO, 84.0% zero-shot success on LIBERO-Plus, and 60.71%/18.21% on RoboTwin 2.0 Easy/Hard; it outperforms all baselines on three real-world tasks under standard and OOD conditions.The evaluations span single-arm and bimanual manipulation, including long-horizon and fine-grained tasks.Source: §1 · pp. 2–3

2 Related Work

Related work advances generalist robot policies, cross-view and future-aware prediction, and intermediate supervision. W2-VLA combines these directions by predicting future wrist latents through a task-conditioned interface shaped by structured W2-CoT annotations.

Generalist VLA models

  • RT-1 (Brohan et al., 2023) and RT-2 established scalable transformer policies and web-to-robot transfer; W2-VLA connects VLM context to future wrist prediction.OpenVLA (Kim et al., 2025b) and Octo (Octo Model Team et al., 2024) improve accessibility and adaptability, while recent work advances continuous action generation, diffusion experts, cross-embodiment control, and modular VLM-action interfaces.Source: §2 · p. 3

Cross-view modeling and future-aware prediction

  • Cross-view fusion and future-aware prediction are established directions, but W2-VLA explicitly predicts wrist-local temporal dynamics conditioned jointly on task context and wrist history.Prior work includes cross-view attention (Jangir et al., 2022), wrist-view generation (Qian et al., 2025), and future latent prediction (Sun et al., 2026); W2-VLA uses a fixed-length interface between the VLM and wrist predictor.Source: §2 · p. 3

Intermediate supervision and structured grounding

  • Prior methods use linguistic, visual, spatial, and latent intermediate structures; W2-VLA uses structured W2-CoT annotations to shape its task-conditioned interface without decoding annotations at inference.Examples include future visual goals in CoT-VLA (Zhao et al., 2025), structured spatial reasoning in GraphCoT-VLA (Huang et al., 2026b), and dense frame-level annotations in RoboInter (Li et al., 2026).Source: §2 · pp. 3–4

3 Methodology

W2-VLA uses task-conditioned latent modeling tokens to connect current multi-view observations and instructions with future wrist prediction and action generation. Structured W2-CoT supervision shapes this interface to encode manipulation progress, physical transitions, and wrist-local evidence.

3.1 Task-Conditioned Latent Modeling Interface

  • Task- and observation-conditioned latent modeling states form a fixed-length interface that conditions wrist prediction and receives signals from annotation, wrist-prediction, and action objectives.The interface consists of K dedicated tokens contextualized by the current multi-view observations and instruction.Source: §3.1 · p. 4

3.2 Task-Conditioned Future Wrist Prediction

  • The wrist predictor combines interface states with encoded wrist history to forecast future wrist latents, modeling state transitions rather than reconstructing RGB observations.A future-wrist context adapter compresses the predicted latents into trainable, future-aware context for action prediction; the action objective does not update the predictor through this adapter path.Source: §3.2 · p. 5

3.3 Structured W2-CoT Annotation Synthesis and Auxiliary Supervision

  • W2-CoT annotations specify manipulation stage and progress, robot-centric physical transitions, and wrist-local evidence, with left/right evidence separated for bimanual data.An offline VLM proposes segment annotations from instructions, synchronized views, and state-action evidence; verification checks gripper consistency, temporal ordering, release preconditions, and wrist locality before normalized fields supervise next-token prediction.Source: §3.3 · pp. 5–6

3.4 Training Objective and Inference

  • A fused VLM and future-wrist context conditions a DiT flow-matching action head, while inference forecasts wrist latents and generates action chunks without future wrist observations or autoregressive W2-CoT decoding.The joint objective combines action generation, CoT prediction, and future wrist latent prediction; inference uses the same action-conditioning state selection as training.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate W2-VLA on simulated benchmarks and real-world manipulation, then examine component contributions, inference efficiency, prediction targets, and attention behavior.

4.1 Experimental Setup

  • Evaluation spans LIBERO and LIBERO-Plus, RoboTwin 2.0, and three CoBoT Magic tasks, using 2,000 LIBERO trials, 10,030 perturbation instances, and 100 RoboTwin trials per task per setting.Real-world evaluation covers Table Cleaning, Occluded Placement, and Bimanual Plug Insertion; the section assesses simulation performance, robustness, component contributions, inference efficiency, and OOD generalization.Source: §4 · p. 6, §4.1 · p. 7

4.2 Simulation Results

  • W2-VLA records overall success rates of 98.5% on LIBERO, 84.0% on LIBERO-Plus, 60.71% on RoboTwin Easy, and 18.21% on RoboTwin Hard.It achieves the highest overall rates on LIBERO and LIBERO-Plus and surpasses the strongest reported baselines in both RoboTwin settings.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 8

4.3 Real-World Evaluation

  • W2-VLA leads all three real-world tasks in both settings, averaging 70.00% standard and 52.22% OOD success, 15.56 and 14.44 points above VLA-JEPA, respectively.Its largest reported OOD margin over VLA-JEPA is on Bimanual Plug Insertion: 33.33% versus 10.00%.Source: §4.3 · p. 8, Figure 3 · p. 9
  • W2-VLA achieves the highest progress score on every task in both settings; on Bimanual Plug Insertion, it scores 2.60 versus 1.86 standard and 2.27 versus 1.53 OOD against VLA-JEPA.Policies often complete initial grasping but fail during final alignment or insertion; W2-VLA reaches these contact-sensitive stages more frequently.Source: Figure 3 · p. 9, §4.3 · p. 9

4.4 Ablation Studies

  • Removing the Wrist Predictor lowers average success from 98.5% to 97.5% and Long-suite success from 95.2% to 93.6%; removing W2-CoT supervision lowers average success to 98.0%.The predictor contributes a localized predictive objective, while structured annotations shape the task-conditioned interface.Source: Table 5 · p. 10, §4.4 · p. 10
  • The 16-token fixed latent interface achieves 98.5% average success at 110.58 ms latency, while explicit CoT decoding takes more than 1.5 seconds per action chunk.Fixed latent modeling tokens reduce latency by more than an order of magnitude; increasing token count from 4 to 32 brings no consistent improvement and gradually increases latency.Source: Table 5 · p. 10, §4.4 · p. 10
  • Wrist-only future prediction performs best: wrist observations capture evolving gripper–object geometry, contact, alignment, and release, whereas joint-view prediction adds cost and may introduce interference.The fixed main camera contains substantial static scene content, making its future latents less focused on action-induced local changes.Source: Table 5 · p. 10, §4.4 · p. 10

4.5 Qualitative Analysis

  • Latent-token attention shifts among target objects, grippers, and contact areas as manipulation progresses, drawing on global and wrist-local evidence.In LIBERO-10 it moves from graspable regions during approach and grasp to the held object and target during transport and alignment; in Table Cleaning it follows the object and gripper before cloth–stain contact.Source: Figure 5 · p. 11, §4.5 · p. 11

5 Conclusion

W2-VLA connects global task context to wrist-local prediction through a compact latent interface shaped by structured W2-CoT supervision. It forecasts future wrist latents to inform action generation and demonstrates performance across single-arm and bimanual settings while generating actions above 80 Hz.

5 Conclusion

  • W2-VLA uses a compact latent interface shaped by structured W2-CoT supervision to connect global task context with wrist-local prediction.Source: §5 · p. 11
  • Conditioned on the interface and wrist history, a predictor forecasts future wrist latents as future-aware context for action generation.Source: §5 · p. 11
  • Over 80 Hz action generation is achieved without requiring CoT generation at inference, with strong performance across single-arm and bimanual settings.The experiments span LIBERO, RoboTwin 2.0, and real-world tasks.Source: §5 · p. 11

Supplementary Material · A Details of the W2-CoT Annotation Pipeline · A.1 Representative Generation and Verification Templates

W2-CoT turns robot trajectories into structured supervision by grounding VLM-generated annotations in physical states and visual evidence, then verifying and normalizing them. The templates adapt this process across LIBERO, RoboTwin, and real-world settings while preserving manipulation progress, physical transitions, and wrist-local evidence.

Supplementary Material

  • The pipeline converts trajectories into candidate segments, grounds each in state and visual evidence, generates structured VLM proposals, verifies them, and expands them into frame-level supervision.Available state fields and camera views vary by dataset, but the construction principle remains unchanged.Source: Figure 6 · p. 17, A Details of the W2-CoT Annotation… · p. 17

A Details of the W2-CoT Annotation Pipeline

  • Final annotations use Subtask for manipulation progress, Reasoning for physical transition cues, and Wrist for local evidence such as contact, object motion, placement stability, and gripper-object separation.For bimanual trajectories, Wrist uses the ordered format left=...; right=....Source: A Details of the W2-CoT Annotation… · p. 17
  • Physical checks reject open-gripper grasp or carry claims, require prior holding for release and stable-grasp evidence for carrying, and enforce wrist locality and plausible temporal order.Short jitter and retry motions do not create task stages; post-release withdrawal is retained only when sustained and physically meaningful.Source: A Details of the W2-CoT Annotation… · p. 17, Figure 6 · p. 17
  • Language normalization gives each stage consistent physical descriptions, including open-gripper approach, fingertip contact and closure for grasp, and carried-object destination for transport.Release descriptions capture gripper opening, object stability, and gripper-object separation; withdrawal is described as an open gripper moving away from the placed object.Source: A Details of the W2-CoT Annotation… · pp. 17–18

A.1 Representative Generation and Verification Templates

  • Figure 7 visualizes W2-CoT vocabulary frequency by dataset, with rows for datasets and columns for Subtask, Reasoning, Wrist, and Full CoT.Word size is proportional to token frequency.Source: Figure 7 · p. 18, §A.1 · p. 18
  • The representative interfaces differ by setting: LIBERO uses episode-level planning with segment-wise visual grounding, while RoboTwin uses contact-sheet-based, state-anchored bimanual adjudication.Real-world annotations follow the RoboTwin-style interface with task-specific physical constraints; deterministic post-generation projection is listed separately from VLM prompts.Source: §A.1 · p. 18
  • W2-CoT covers actionable behaviors from reach and grasp through carry, placement, release, and handover, with fields encoding task progress, state transitions, contact dynamics, and wrist-local visual evidence.The supervision supports task decomposition and subtask-level reasoning throughout trajectories.Source: §A.1 · p. 18

B Additional Implementation Details · B.1 Model Architecture. · B.2 Information-Flow Masks for W2-VLA Training and Inference

W2-VLA pairs a Qwen3-VL vision-language backbone with a DiT flow-matching action head and a wrist predictor that supplies future-aware context. Structured annotations supervise training, while the action head uses the same deployable context during training and inference.

B.1 Model Architecture.

  • W2-VLA uses Qwen3-VL-4B-Instruct with a DiT-B flow-matching action head.The Qwen3-VL backbone has approximately 4.44B parameters.Source: §B.1 · p. 18
  • The Qwen3-VL language model has 36 layers, 2560 hidden dimensions, 32 attention heads, and 8 key-value heads; its vision encoder has 24 layers and uses bfloat16 forward passes.The vision encoder uses patch size 16 and temporal patch size 2, projecting features into the 2560-dimensional language space.Source: §B.1 · p. 18
  • The 16-block action transformer uses 768-dimensional tokens, 12 attention heads, adaptive timestep normalization, dropout 0.2, and interleaved self- and cross-attention.It uses 32 learned future/action query tokens; LIBERO actions are 7-dimensional delta end-effector actions, while RoboTwin 2.0 and real-world actions are 14-dimensional absolute joint positions.Source: §B.1 · pp. 18–19
  • A frozen V-JEPA 2.1 encoder and four-layer wrist predictor feed predicted future wrist latents to an adapter that produces 32 future-aware context tokens.These tokens are appended to the action context.Source: §B.1 · p. 19, Figure 8 · p. 20

B.2 Information-Flow Masks for W2-VLA Training and Inference

  • During training, the VLM predicts structured annotation targets from visual context, the CoT prompt, task instruction, and latent modeling tokens, while the DiT excludes prompt and annotation states.The DiT context contains visual, instruction, and latent modeling tokens, together with the adapter-produced future-aware wrist context.Source: §B.2 · p. 20, Figure 8 · p. 20
  • At inference, no annotation sequence is decoded; the VLM processes visual tokens, the prompt template, instruction, and latent tokens, with prompt states excluded from the DiT context.The deployed action context is visual, instruction, and latent tokens plus adapted future-wrist context, matching the training action context; structured annotations remain training-only supervision.Source: §B.2 · p. 20

B.3 Training Details and Simulation Setup · C Real-world Experimental Details

The appendix specifies optimization, preprocessing, and wrist-frame sampling settings for LIBERO and RoboTwin 2.0, and identifies a figure showing the real-world experiment setup.

B.3 Training Details and Simulation Setup

  • The trainable components are Qwen, the JEPA predictor, wrist-context adapter, and action head; V-JEPA 2.1 is frozen.Training uses AdamW, gradient clipping at 1.0, cosine learning-rate scheduling, and 5000 warmup steps; listed learning rates are 1.0 × 10^-5 for Qwen3-VL and 5.0 × 10^-5 for the predictor and adapter.Source: §B.3 · p. 20
  • The objective uses λcot = 0.1 and λwrist = 0.2.Source: §B.3 · p. 20

C Real-world Experimental Details

B.3 Training Details and Simulation Setup

  • Both main and wrist views are resized to 224 × 224.Source: §B.3 · p. 21
  • When wrist-history or future-view sequences are shorter than the prediction horizon at episode boundaries, available boundary images pad them to the required length.A discount coefficient is also applied to the JEPA prediction loss to control learning strength.Source: §B.3 · p. 21
  • LIBERO training uses 60K optimization steps on 4 A100 GPUs, batch size 16 per device, and h = 8 for 8 input and 8 predicted wrist frames.Source: §B.3 · p. 21
  • RoboTwin 2.0 training uses 100K steps on 8 B200 GPUs, batch size 16 per device, and h = 16 with 8 input and 8 predicted frames sampled every other frame.Source: §B.3 · p. 21

C.1 Experimental Setup · D Further Analysis and Visualizations

The real-world setup tests complementary main-view global context and wrist-view interaction evidence across long-horizon cleaning, occluded placement, and contact-rich bimanual insertion. It also evaluates robustness to clutter, lighting, and background changes.

C.1 Experimental Setup

  • Figure 9 shows the experimental setup for the three real-world tasks.Source: §C.1 · p. 21
  • The three tasks test distinct manipulation demands: long-horizon bimanual sequencing, global-to-local visual grounding, and precise contact-rich bimanual control.The main view provides global layout and task-stage information, while wrist views supply local interaction cues for grasping, alignment, and contact transitions.Source: §C.1 · p. 21
  • Table Cleaning requires ordered object handling across both arms, followed by wiping a spill until it is removed or sufficiently covered.Success requires all three objects in the basket and the spill removed or sufficiently covered; wrist views verify grasp stability, placement, and wiping contact and coverage.Source: §C.1 · p. 21
  • Occluded Placement tests global-to-local grounding because wrist cameras initially cannot see the mango or plate, while the main view reveals the scene layout and route context.The task succeeds when the mango is placed on the plate without knocking over the obstacle, dropping the mango, or letting it roll off.Source: §C.1 · pp. 21–22
  • Bimanual Plug Insertion requires the left arm to stabilize the power strip while the right arm aligns and inserts the plug into the target socket.Wrist views support local alignment, pre-contact adjustment, and insertion-depth control; small translational or angular errors can cause misalignment, slippage, or jamming.Source: §C.1 · pp. 21–22
  • The setup includes three out-of-distribution conditions: irrelevant table objects, rotating colored-light perturbations, and changes to tablecloth color.Source: §C.1 · p. 22

D.1 How effective is the wrist predictor? · D.2 More visualizations of latent-modeling-to-image attention.

The JEPA wrist predictor more closely matches future wrist latents than copying the current representation, while latent-token attention visualizations show focus on manipulation-stage-relevant regions across LIBERO-10 and RoboTwin 2.0 tasks.

D.1 How effective is the wrist predictor?

  • On LIBERO-10, the evaluation compares predicted future wrist latent tokens with encoded future-frame tokens and a baseline that copies current wrist latents forward.It reports latent-token MSE and cosine similarity; lower MSE and higher cosine similarity indicate stronger consistency with the actual rollout.Source: §D.1 · p. 22
  • JEPA achieves latent-token MSE 0.749 versus 2.183 for the copy-current baseline, and cosine similarity 0.888 versus 0.699, a gain of 0.189.The evaluation compares predicted future wrist latents with latents encoded from observed future wrist frames across 500 episode records.Source: §D.1 · p. 22, Figure 10 · p. 22

D.2 More visualizations of latent-modeling-to-image attention.

  • The attention visualizations cover one LIBERO-10 task and two RoboTwin 2.0 tasks, pairing main- and wrist-view maps with Subtask/Reasoning/Wrist descriptions for semantic reference.The descriptions come from the full VLM context and are paired with the maps to interpret the current manipulation stage.Source: §D.2 · p. 23
  • Maps use final-layer Qwen3-VL post-softmax self-attention, averaged uniformly across 32 heads and 16 learned latent modeling tokens, rather than gradients or attention rollout.The weights come from a no-gradient forward pass, not the vision encoder or DiT action head.Source: §D.2 · p. 23
  • Attention maps are independently min-max normalized per panel, so brightness indicates relative spatial attention within a panel and cannot compare absolute magnitude across views or rollout steps.Current views are resized to 224 × 224 and tokenized into an 8 × 8 grid; each panel represents one policy call without temporal averaging.Source: §D.2 · p. 23
  • Across the three tasks, attention concentrates on stage-relevant targets, active grippers, and contact areas, shifting as LIBERO can manipulation and RoboTwin stapler pressing and microphone handover progress.For the LIBERO task, highlighted regions move from the can to the gripper and basket; for press_stapler, they move from the approaching gripper and stapler to their contact region; for handover_mic, they follow the microphone and both grippers through approach, grasp, transfer, and release.Source: §D.2 · p. 23

D.3 Efficiency Analysis · D.4 Visualization of rollouts in real-world OOD scenarios.

W2-VLA achieves the highest mean standard-condition success rate among the compared methods while generating actions at a high frequency. Real-world rollout visualizations show its generalization scenarios under table clutter, lighting perturbations, and background variations.

D.3 Efficiency Analysis

  • Action-generation throughput is computed by dividing the generated action-chunk length by the time to generate the complete chunk.The latency is amortized over all actions in the chunk.Source: §D.3 · p. 23
  • 87.43 Hz (W2-VLA), 102.12 Hz (VLA-JEPA), and 119.75 Hz (π0) are the methods’ action-generation throughputs.These values come from generating chunks of 16 actions in 183 ms, 7 actions in 68.55 ms, and 50 actions in 417.55 ms, respectively.Source: §D.3 · p. 23
  • All three methods operate in a comparable high-frequency regime of 87.43–119.75 Hz, and W2-VLA retains sufficient capacity for real-time control despite lower peak throughput.The reported throughput measures action-generation capacity, not policy forward passes per second.Source: §D.3 · p. 23
  • 70.00% mean standard-condition success for W2-VLA exceeds VLA-JEPA (54.44%) by 15.56 percentage points and π0 (42.22%) by 27.78 percentage points.Figure 12 compares success rate, generated actions per second, and model size; higher baseline throughput does not translate into higher task completion.Source: §D.3 · p. 23, Figure 12 · p. 24

D.4 Visualization of rollouts in real-world OOD scenarios.

  • W2-VLA maintains strong generalization performance under table clutter, lighting perturbations, and background variations.The paper illustrates these conditions with rollout image sequences from three real-world tasks.Source: §D.4 · p. 25, Figure 13 · p. 25
  • Figure 13 shows W2-VLA rollout sequences for three real-world tasks in generalization settings.The visualized OOD settings are table clutter, random lighting perturbations, and background variations.Source: Figure 13 · p. 25, §D.4 · p. 25

Related papers

Loading 2608.05369v2…