PaperlayerView exact revision on arXiv

Source-linked AI summary

World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation

Yuhao Pan, Haosong Peng, Zhengshen Zhang, Zhengyang Yan, Yalun Dai, Fushuo Huo, Chujie Wang, Tianyu Qi, Xiucheng Wang, Nan Cheng, Wenchao Xu

arXiv:2608.05369v1cs.ROcs.CV

TL;DR

Existing VLA models often treat main and wrist views as parallel inputs, despite their distinct global and action-proximal roles in fine-grained manipulation. W2-VLA uses task-conditioned future wrist modeling with structured supervision, achieving strong results across simulated and real-world single-arm and bimanual tasks while generating actions above 80 Hz.

  • Problem

    VLA models often treat main and wrist observations as parallel inputs, despite wrist views directly revealing rapidly changing end-effector interactions needed for fine-grained manipulation.

  • Method

    W2-VLA uses a fixed-length task-conditioned latent interface, structured W2-CoT supervision, and wrist-history-conditioned prediction of future wrist latents for action generation.

  • Results

    98.5% average success on LIBERO, with strong performance across RoboTwin 2.0 and real-world single-arm and bimanual tasks, including standard and OOD conditions.

  • Takeaways & Limitations

    W2-VLA connects global task context with wrist-local prediction while enabling real-time action generation above 80 Hz without CoT generation at inference.

Abstract

from arXiv · show

Vision-language-action (VLA) models often treat main-view and wrist-view observations as parallel visual inputs, overlooking their distinct roles in robot manipulation. Fine-grained manipulation, however, benefits from anticipating how wrist-local interactions may evolve under the global task context. To address this limitation, we present World-to-Wrist VLA (W2-VLA), a VLA model for fine-grained robot manipulation with task-conditioned future wrist modeling. Given current multi-view observations and a task instruction, W2-VLA contextualizes a set of latent modeling tokens as a compact interface between the vision-language model and the wrist predictor. Conditioned on this interface and the observed wrist history, the predictor forecasts future wrist latents, which are transformed into future-aware context for action prediction. In addition, we introduce W2-CoT, a synthesis pipeline that produces structured annotations describing manipulation progress, physical transition cues, and wrist-local evidence. These annotations provide auxiliary supervision that shapes the task-conditioned latent interface. Experiments on LIBERO, RoboTwin 2.0, and real-world manipulation tasks demonstrate improved fine-grained and contact-sensitive manipulation across both single-arm and bimanual settings, while maintaining action-generation rates above 80 Hz.

1 Introduction

W2-VLA models future wrist dynamics as task-conditioned latent prediction, linking global task context to action-proximal wrist interactions for fine-grained manipulation. It uses W2-CoT supervision to shape this interface and achieves strong benchmark and real-world results while generating actions above 80 Hz.

  • Motivation: Main-view observations provide global task context, whereas wrist views expose rapidly changing gripper–object interactions that are critical for fine-grained manipulation.The distinction includes scene layout, object identity, goal relations, task progress, and end-effector interactions.
  • Motivation: Wrist-centered latent prediction captures action-proximal evolution of local interactions without reconstructing pixel-level appearance.It focuses on end-effector-relevant changes beyond the current state.
  • Method: W2-VLA uses dedicated latent modeling tokens to form a fixed-length, task-conditioned interface between the VLM and wrist predictor.The interface is contextualized from current multi-view observations and the instruction, then supports future wrist latent prediction from wrist history.
  • Results: 98.5% average success on LIBERO and 60.71% and 18.21% on RoboTwin 2.0 Easy and Hard show strong benchmark performance, alongside baseline-leading results across three real-world tasks.The real-world evaluations include standard and out-of-distribution conditions and span single-arm and bimanual manipulation.
  • Method: W2-CoT structured annotations shape the task-conditioned interface and guide future wrist latent prediction without requiring CoT generation during inference.The annotations provide evidence-grounded supervision, while the policy maintains real-time action generation above 80 Hz.

2 Related Work

Related work frames VLA models as unified policies mapping visual observations and language instructions to robot actions. It highlights RT-1 and RT-2 as foundational scalable transformer policies, followed by OpenVLA and Octo as efforts improving accessibility and adaptability.

  • Generalist VLA models: VLA models learn unified mappings from visual observations and language instructions to robot actions.This establishes the general formulation of vision-language-action policies for robot control.
  • Generalist VLA models: RT-1 and RT-2 established scalable transformer policies for real-world manipulation.The passage identifies these models as foundational systems in generalist VLA research.
  • Generalist VLA models: RT-1 and RT-2 demonstrated transfer of web-scale knowledge to robot control, while OpenVLA and Octo improved policy accessibility and adaptability.These works represent successive advances in knowledge transfer and the practical usability of generalist robot policies.

3 Methodology

W2-VLA uses task-conditioned latent modeling tokens to connect multi-view observations and instructions with future wrist-state prediction, then fuses predicted wrist context with VLM features for action generation. W2-CoT provides auxiliary structured supervision covering manipulation progress, physical transitions, and wrist-local evidence.

  • Task-conditioned interface: W2-VLA contextualizes K latent modeling tokens from current multi-view observations and the instruction, forming a fixed-length interface for wrist prediction and action generation.The interface states receive supervision from annotations, future wrist prediction, and action generation.
  • Future wrist modeling: A frozen V-JEPA 2.1 encoder maps wrist-history and training-only future clips to latent tokens, while a bidirectional Transformer predicts future wrist latents conditioned on the interface.The future target is detached with stop-gradient, so the objective supervises the predictor and conditions the interface without RGB reconstruction.
  • Future-aware wrist context: A Q-Former-style adapter with M learnable queries compresses predicted future wrist latents into fixed-size context tokens projected to the VLM hidden dimension.Stop-gradient blocks action-objective updates through the adapter path while keeping the adapter trainable.
  • Structured auxiliary supervision: W2-CoT annotations contain Subtask, Reasoning, and Wrist fields describing manipulation progress, physical transitions, and wrist-local evidence for auxiliary next-token supervision.An offline VLM proposes annotations from trajectory segments, task instructions, synchronized actions, and visual keyframes.
  • Action generation and inference: The fused VLM and future-aware wrist context conditions a DiT-based flow-matching action head, and inference needs neither future wrist observations nor autoregressive W2-CoT decoding.The joint objective combines action generation, CoT prediction, and future wrist latent prediction.

4 Experiments

W2-VLA is evaluated across simulation and real-world manipulation settings, including robustness, component contributions, inference efficiency, and OOD generalization. It achieves strong task success, improves progress under standard and OOD conditions, and benefits from future-wrist prediction, structured supervision, and a fixed latent interface.

  • Evaluation setup: W2-VLA is evaluated on LIBERO, RoboTwin 2.0, and three CoBoT Magic real-world tasks under simulation, robustness, efficiency, and OOD-generalization protocols.The real-world tasks are Table Cleaning, Occluded Placement, and Bimanual Plug Insertion.
  • LIBERO: 98.5% average success on LIBERO makes W2-VLA the best-performing method, exceeding the strongest baseline by 1.3 percentage points.It reaches 99.6% on Spatial, 99.8% on Object, and 99.2% on Goal, while achieving 95.2% on Long.
  • Real-world evaluation: W2-VLA achieves the highest real-world progress score on every task under both standard and OOD conditions.Evaluation compares W2-VLA with π0 and VLA-JEPA using 30 standard and 30 OOD trials per task, with OOD trials covering clutter, lighting, and background variations.
  • Real-world evaluation: 2.60 versus 1.86 progress on Bimanual Plug Insertion under standard conditions shows W2-VLA’s advantage over VLA-JEPA in partial task completion.Under OOD conditions, W2-VLA scores 2.27 versus 1.53 for VLA-JEPA.
  • Ablations and efficiency: Removing the Wrist Predictor lowers average LIBERO success from 98.5% to 97.5%, while removing W2-CoT supervision reduces it to 98.0%.The Wrist Predictor reduction is largest on Long, where performance falls from 95.2% to 93.6%; the fixed latent interface reduces latency by more than an order of magnitude versus explicit decoding.
  • Attention analysis: Latent modeling tokens attend to stage-relevant target objects, grippers, and contact areas, shifting focus as manipulation progresses.Attention visualization uses main- and wrist-view observations and decodes Subtask/Reasoning/Wrist descriptions only for semantic interpretation.

5 Conclusion

W2-VLA presents task-conditioned future wrist modeling for fine-grained robot manipulation, using structured W2-CoT supervision and a compact latent interface to connect global task context with wrist-local prediction. Experiments demonstrate strong performance across LIBERO, RoboTwin 2.0, and real-world tasks in both single-arm and bimanual settings.

  • Conclusion: W2-VLA connects global task context with wrist-local prediction through a compact latent interface shaped by structured W2-CoT supervision.This replaces treating main- and wrist-view observations as parallel inputs.
  • Conclusion: Conditioned on the latent interface and wrist history, a predictor forecasts future wrist latents that provide future-aware context for action generation.
  • Conclusion: Experiments on LIBERO, RoboTwin 2.0, and real-world tasks demonstrate strong performance across single-arm and bimanual settings.

Supplementary Material … C Real-world Experimental Details

The supplementary material covers the W2-CoT annotation pipeline, additional implementation details for W2-VLA, and real-world experimental setup. It includes representative generation and verification templates, model and training specifications, information-flow masks, simulation setup, and experimental setup.

  • A Details of the W2-CoT Annotation Pipeline: The supplementary material includes representative generation and verification templates for the W2-CoT annotation pipeline.
  • Supplementary Material: The listed supplementary sections collectively span annotation, implementation, simulation, and real-world experiment documentation.
  • Supplementary Material: The supplementary material is organized into sections A, B, and C covering W2-CoT annotations, W2-VLA implementation, and real-world experimental details.
  • B Additional Implementation Details: The W2-VLA implementation details cover model architecture and information-flow masks for training and inference.
  • B Additional Implementation Details: Additional implementation details also specify training details and the simulation setup.
  • C Real-world Experimental Details: The real-world experimental details include an experimental setup subsection.

D Further Analysis and Visualizations … B Additional Implementation Details

The paper supplements W2-VLA with a physically grounded W2-CoT annotation pipeline that converts trajectories into structured manipulation labels. It also adapts annotation interfaces and vocabularies across LIBERO, RoboTwin, and real-world settings.

  • D Further Analysis and Visualizations: W2-CoT converts trajectory segments into verified, language-normalized annotations through evidence grounding, structured proposal generation, consistency checking, and frame-level expansion.The pipeline uses synchronized state-action transitions and visual context to define candidate segments before verification and normalization.
  • A Details of the W2-CoT Annotation Pipeline: The pipeline extracts gripper openness, end-effector motion, and action changes to distinguish manipulation stages such as approach, transport, and post-release withdrawal.Available state fields and camera views vary across datasets, but the construction principle remains unchanged.
  • A Details of the W2-CoT Annotation Pipeline: Final supervision contains Subtask, Reasoning, and Wrist fields describing manipulation progress, physical transition cues, and wrist-local evidence.The Wrist field includes local contact, object motion, and related wrist evidence.
  • A Details of the W2-CoT Annotation Pipeline: Physical verification rejects inconsistent descriptions, enforces release-after-holding and stable-grasp requirements, and keeps bimanual wrist descriptions local to the corresponding gripper and object.Temporal consistency is also checked across the generated annotation sequence.
  • A Details of the W2-CoT Annotation Pipeline: Language normalization standardizes approach, grasp, transport, release, and post-release withdrawal descriptions while preserving their physical meaning.Descriptions explicitly encode gripper state, contact, object destination or stability, and gripper-object separation where applicable.
  • A.1 Representative Generation and Verification Templates: Annotation interfaces differ by setting: LIBERO uses episode-level planning with segment-wise visual grounding, while RoboTwin uses contact-sheet-based, state-anchored bimanual adjudication.Real-world annotations follow the RoboTwin-style interface with task-specific physical constraints.
  • A.1 Representative Generation and Verification Templates: The annotation vocabulary spans actionable arm and gripper behaviors, including reach, grasp, carry, place, release, retract, contact, secure, hold, and handover.Subtask captures high-level progress, while Reasoning and Wrist provide physically grounded explanatory and local evidence.

B.1 Model Architecture. · B.2 Information-Flow Masks for W2-VLA Training and Inference

W2-VLA combines a Qwen3-VL-4B-Instruct backbone with a DiT-B flow-matching action head and a wrist branch that supplies future-aware context. Its information-flow masks use structured annotations for training supervision while keeping the deployable action context consistent at inference.

  • B.1 Model Architecture.: W2-VLA uses Qwen3-VL-4B-Instruct as its vision-language backbone, with approximately 4.44B parameters and bfloat16 forward passes.The language model has 36 transformer layers, hidden dimension 2560, 32 attention heads, and 8 key-value heads; the vision encoder has 24 layers and projects into the 2560-dimensional language space.
  • B.1 Model Architecture.: The DiT-B flow-matching action head contains 16 transformer blocks with 768-dimensional tokens, 12 attention heads, adaptive timestep normalization, dropout 0.2, and interleaved self- and cross-attention.For LIBERO, it predicts 7-dimensional delta end-effector actions comprising translation, axis-angle rotation, and gripper dimensions.
  • B.1 Model Architecture.: The wrist branch uses a frozen V-JEPA 2.1 encoder, a four-layer predictor, and an adapter that converts predicted wrist latents into 32 future-aware context tokens.These future-aware tokens are provided to the action-prediction pathway.
  • B.2 Information-Flow Masks for W2-VLA Training and Inference: During training, structured annotation targets receive auxiliary next-token prediction supervision conditioned on visual context, the prompt template, the task instruction, and latent modeling tokens.The training masks exclude prompt-formatting and annotation-target states from the DiT action context.
  • B.2 Information-Flow Masks for W2-VLA Training and Inference: The training action context includes visual, instruction, and latent modeling states together with future-aware wrist context produced by the wrist-context adapter.This composition separates annotation supervision from the deployable action-conditioning states.
  • B.2 Information-Flow Masks for W2-VLA Training and Inference: At inference, W2-VLA decodes no annotation sequence and processes visual tokens, the prompt template, the task instruction, and latent modeling tokens.The prompt template structures the latent modeling interface, while its hidden states are excluded from the DiT context.
  • B.2 Information-Flow Masks for W2-VLA Training and Inference: The inference wrist predictor produces a future wrist latent that the adapter converts into W before appending it to the action context.The resulting deployable context is V, I, Q, and W, matching the action-context composition used in training.

B.3 Training Details and Simulation Setup · C Real-world Experimental Details

The paper specifies trainable components, optimization settings, image preprocessing, sequence padding, and dataset-specific simulation training configurations. It also includes a real-world experimental setup figure.

  • B.3 Training Details and Simulation Setup: λcot = 0.1 and λwrist = 0.2 weight the auxiliary objectives; Qwen, JEPA predictor, wrist-context adapter, and action head are trainable, while V-JEPA 2.1 is frozen.The objective passage introduces the complete objective, while the implementation details identify trainable and frozen modules.
  • B.3 Training Details and Simulation Setup: AdamW uses β1 = 0.9, β2 = 0.95, ϵ = 10−8, weight decay 10−8, gradient clipping 1.0, cosine decay, 10−6 minimum learning rate, and 5000 warmup steps.Learning rates are 1.0 × 10−5 for Qwen3-VL, 5.0 × 10−5 for the JEPA predictor, and 5.0 × 10−5 for the wrist-context adapter.
  • C Real-world Experimental Details: The real-world experiments are documented with a dedicated setup figure.Figure 9 is captioned “Real-world Experiments Setup.”
  • B.3 Training Details and Simulation Setup: Both main-view and wrist-view images are resized to 224 × 224.The preprocessing applies the same spatial resolution to both visual inputs.
  • B.3 Training Details and Simulation Setup: Historical or future wrist-view sequences shorter than the prediction horizon are padded with available boundary images, and a discount coefficient scales the JEPA prediction loss.This handles episode beginnings and endings where insufficient context is available.
  • B.3 Training Details and Simulation Setup: LIBERO training uses 60K optimization steps on 4 A100 GPUs with per-device batch size 16 and h = 8, using 8 historical frames to predict 8 future frames.The prediction horizon and input/output frame counts are both eight.
  • B.3 Training Details and Simulation Setup: RoboTwin 2.0 training uses 100K optimization steps on 8 B200 GPUs with per-device batch size 16 and h = 16, while every-other-frame sampling yields 8 input and 8 predicted frames.Although the nominal horizon is sixteen, temporal subsampling keeps the actual frame counts at eight.

C.1 Experimental Setup · D Further Analysis and Visualizations

The real-world evaluation uses three manipulation tasks that probe long-horizon decomposition, complementary global-local grounding, and contact-rich bimanual insertion. It also tests robustness under clutter, lighting, and background shifts.

  • C.1 Experimental Setup: The study evaluates Table Cleaning, Occluded Placement, and Bimanual Plug Insertion in real-world experimental setups.Figure 9 presents the deployment setup for all three tasks.
  • C.1 Experimental Setup: Table Cleaning requires long-horizon stage decomposition, sequential object handling, and switching between two arms.The instruction combines collecting crumpled paper and blocks, placing them in a tray, and wiping a table stain.
  • C.1 Experimental Setup: Occluded Placement tests complementary global and local visual grounding when wrist cameras initially lack object visibility.The main view supplies localization and route context, while wrist views support grasping, alignment, and release.
  • C.1 Experimental Setup: Bimanual Plug Insertion requires the left arm to stabilize a power strip while the right arm grasps, aligns, and inserts a corded plug.The task uses an unpowered power strip and target socket, emphasizing fine-grained, contact-rich manipulation.
  • C.1 Experimental Setup: In Occluded Placement, the robot must place a foam mango on a plastic plate without disturbing the low foam obstacle.The scene also includes table-positioning tape, and success excludes knocking over the obstacle, dropping the mango, or letting it roll away.
  • C.1 Experimental Setup: Additional out-of-distribution tests vary table clutter, colored lighting, and tablecloth background.These settings randomly add irrelevant objects, rotate a colored light, or change the tablecloth color.

D.1 How effective is the wrist predictor? · D.2 More visualizations of latent-modeling-to-image attention.

The JEPA wrist predictor more accurately forecasts future wrist latents than copying the current tokens, while latent-modeling attention concentrates on manipulation-relevant visual regions across tasks. Attention visualizations use averaged final-layer weights and show stage-dependent shifts among objects, grippers, and contact areas.

  • D.1 How effective is the wrist predictor?: The evaluation compares predicted future wrist latent tokens with ground-truth tokens encoded from observed future wrist frames using the same visual encoder.Latent-token MSE and cosine similarity are the reported measures.
  • D.1 How effective is the wrist predictor?: JEPA achieves latent-token MSE 0.749 across 500 LIBERO-10 episode records, versus 2.183 for the copy-current baseline.Lower MSE indicates stronger consistency with ground-truth future wrist tokens.
  • D.1 How effective is the wrist predictor?: Cosine similarity rises from 0.699 for copy-current to 0.888 for JEPA, a gain of 0.189.Higher cosine similarity indicates stronger consistency with future wrist observations.
  • D.2 More visualizations of latent-modeling-to-image attention.: Attention visualizations cover one LIBERO-10 task and two RoboTwin 2.0 tasks, pairing main- and wrist-view maps with Subtask/Reasoning/Wrist descriptions.The descriptions provide semantic references for the current manipulation stage.
  • D.2 More visualizations of latent-modeling-to-image attention.: The maps average post-softmax self-attention from Qwen3-VL’s final 36th language-transformer layer across all 32 heads and 16 learned latent modeling tokens.Weights are extracted with output_attentions=True during a no-gradient eager forward pass.
  • D.2 More visualizations of latent-modeling-to-image attention.: Current RGB views are resized to 224 × 224, converted to an 8 × 8 image-token grid, independently min-max normalized, bicubically upsampled, and overlaid at opacity 0.5.LIBERO uses main and wrist views; RoboTwin uses main, left-wrist, and right-wrist views.
  • D.2 More visualizations of latent-modeling-to-image attention.: Across three tasks, latent-modeling attention concentrates on target objects, active grippers, and local contact areas relevant to the current manipulation stage.In LIBERO-10, highlighted regions shift among the active can, gripper, and basket as the policy progresses from approach and grasp to transport and release.

D.3 Efficiency Analysis · D.4 Visualization of rollouts in real-world OOD scenarios.

D.3 defines generated-actions-per-second throughput and compares W2-VLA with π0 and VLA-JEPA, while D.4 visualizes W2-VLA rollouts that generalize across several real-world OOD conditions.

  • D.3 Efficiency Analysis: 87.43 Hz is W2-VLA’s throughput from generating a 16-action chunk in 183 ms.The calculation is 16/0.183 = 87.43 Hz.
  • D.3 Efficiency Analysis: 119.75 Hz is π0’s throughput from generating a 50-action chunk in 417.55 ms, compared with 102.12 Hz for VLA-JEPA’s 7-action chunk in 68.55 ms.W2-VLA has lower peak throughput, while all three methods operate in a comparable high-frequency range.
  • D.4 Visualization of rollouts in real-world OOD scenarios.: The rollout visualizations include LIBERO-10’s task of placing alphabet soup and tomato sauce in a basket.The accompanying examples describe reaching toward each can, carrying it toward the basket, and releasing the tomato sauce can into stable placement.
  • D.4 Visualization of rollouts in real-world OOD scenarios.: The RoboTwin2.0 press_stapler rollout shows the right gripper approaching while open, then making fingertip contact and closing to secure the stapler.The left gripper remains still during these described subtasks.
  • D.3 Efficiency Analysis: Generated-actions-per-second throughput is computed as action-chunk length divided by the latency for generating one complete chunk.Mean standard-condition success is plotted against generated actions per second, with bubble area representing total model parameters.
  • D.4 Visualization of rollouts in real-world OOD scenarios.: W2-VLA’s real-world rollout sequences cover table clutter, random lighting perturbations, and background variations.Figure 13 identifies these three OOD settings, including the instruction to pick up crumpled paper and wipe a brown stain on the table.
Loading 2608.05369v1…