World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation

1 Introduction

• Main views supply global task context, while wrist views reveal rapidly changing gripper–object interactions, so treating views as parallel inputs overlooks wrist observations’ action-proximal role.This distinction matters for fine-grained tasks such as plug insertion, which coordinate global understanding with local interaction dynamics.Source: §1 · p. 1
• Wrist-centered latent prediction captures end-effector-relevant interaction changes beyond the current state without reconstructing pixel-level appearance.This contrasts with predictive targets defined over the global scene or unified policy representations.Source: §1 · p. 2
• Similar wrist histories can imply different futures, so global task context and manipulation stage must disambiguate the relevant transition.W2-CoT annotations supervise manipulation progress, physical transition cues, and wrist-local evidence to shape the task-conditioned interface.Source: §1 · p. 2
• W2-VLA forecasts task-relevant future wrist latents from a fixed-length interface contextualized by current multi-view observations and instruction, together with encoded wrist history.A lightweight adapter turns predicted latents into future-aware wrist context for a flow-matching action head; inference needs neither future wrist observations nor autoregressive W2-CoT decoding.Source: §1 · p. 2, Figure 1 · p. 2
• W2-CoT supervision shapes the interface through autoregressive annotation prediction, future wrist prediction, and action generation, while inference exceeds 80 Hz without CoT generation.The annotations are produced by an evidence-grounded synthesis pipeline.Source: §1 · pp. 2–3
• 98.5% average success on LIBERO and 60.71% and 18.21% on RoboTwin 2.0 Easy and Hard accompany baseline-leading results across three real-world tasks under standard and OOD conditions.The evaluations span single-arm and bimanual settings.Source: §1 · pp. 2–3

Generalist VLA models

• RT-1 (Brohan et al., 2023) and RT-2 established scalable VLA policies, while later systems improved accessibility, adaptability, and continuous action generation.OpenVLA (Kim et al., 2025b), Octo (Octo Model Team et al., 2024), flow-matching policies (Black et al., 2026; Intelligence et al., 2025), and diffusion-based action experts (Wen et al., 2025) extend this landscape; W2-VLA adds a task-conditioned interface connecting VLM context to future wrist prediction.Source: §2 · p. 3

Cross-view modeling and future-aware prediction

• Prior work combines external and wrist views or predicts future-aware representations, but generally does not isolate wrist-local temporal dynamics conditioned jointly on task context and wrist history.W2-VLA addresses this gap with future wrist latent prediction through a fixed-length interface between the VLM and wrist predictor.Source: §2 · p. 3

Intermediate supervision and structured grounding

• Prior intermediate-supervision methods use linguistic, visual, spatial, or latent structures; W2-VLA uses structured W2-CoT annotations to shape its task-conditioned interface without annotation decoding at inference.Examples include future visual goals in CoT-VLA (Zhao et al., 2025), combined reasoning and foresight in HALO (Shou et al., 2026), continuous latent reasoning in LaRA-VLA (Bai et al., 2026), and dense annotations in RoboInter (Li et al., 2026).Source: §2 · p. 3

3.1 Task-Conditioned Latent Modeling Interface

• The model contextualizes dedicated latent tokens with current multi-view observations and the instruction, forming a fixed-length interface supervised through annotation, future-wrist prediction, and action-generation objectives.These objectives affect the interface through different computational paths.Source: §3.1 · p. 4

3.2 Task-Conditioned Future Wrist Prediction

• Conditioned on the interface and encoded wrist history, a predictor forecasts future wrist latents as state transitions, which a trainable adapter compresses into future-aware context for action generation.The action objective is prevented from updating the future-wrist predictor through the adapter path; the future latent target is detached, and the predictor does not reconstruct RGB observations.Source: §3.2 · pp. 4–5

3.3 Structured W2-CoT Annotation Synthesis and Auxiliary Supervision

• W2-CoT annotations encode manipulation stage and progress, robot-centric physical transitions, and wrist-local evidence, then supervise the interface to capture these cues.An offline VLM proposes annotations from task instructions, synchronized views, and state–action evidence; proposals are checked for consistency, temporal order, and wrist locality, then normalized and propagated to frames. For bimanual data, wrist evidence is ordered left=...; right=....Source: §3.3 · p. 5

3.4 Training Objective and Inference

• A DiT-based flow-matching action head uses fused VLM and future-wrist contexts, while training jointly optimizes action generation, CoT prediction, and future-wrist latent prediction.At inference, one forward pass forecasts wrist latents and generates an action chunk without future wrist observations or autoregressive W2-CoT decoding.Source: §3.4 · p. 6

4.1 Experimental Setup

• The evaluation spans LIBERO, RoboTwin 2.0, and three CoBoT Magic real-world tasks, using a 4.97B-parameter model with 16 latent modeling tokens and structured Subtask/Reasoning/Wrist supervision.The real-world tasks are Table Cleaning, Occluded Placement, and Bimanual Plug Insertion; evaluation includes standard and OOD trials, with OOD conditions split across table clutter, lighting perturbations, and background variations.Source: §4 · p. 6, §4.1 · pp. 6–7, §4.3 · p. 8

4.2 Simulation Results

• 98.5% average LIBERO success rate beats the strongest baseline by 1.3 percentage points; RoboTwin success reaches 60.71% Easy and 18.21% Hard, leading strongest baselines by 7.79 and 1.87 points.On LIBERO, W2-VLA leads on Spatial (99.6%), Object (99.8%), and Goal (99.2%) and reaches 95.2% on Long. On RoboTwin, it exceeds UP-VLA by 7.79 points on Easy and π0 by 1.87 points on Hard.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.3 Real-World Evaluation

• 70.00% standard real-world success and 52.22% OOD success are highest across all three tasks; deployment generates 16-step action chunks at 87.43 Hz.The standard-condition average exceeds VLA-JEPA and π0 by 15.56 and 28.89 percentage points; the OOD average exceeds VLA-JEPA by 14.44 points. On OOD Bimanual Plug Insertion, W2-VLA reaches 33.33% success versus 10.00% for VLA-JEPA.Source: Figure 3 · p. 8, §4.3 · p. 8
• W2-VLA achieves the highest progress score on every real-world task in both settings; on Bimanual Plug Insertion, it scores 2.60 versus VLA-JEPA’s 1.86 standard and 2.27 versus 1.53 OOD.Policies often complete initial grasping but fail at final alignment or insertion; W2-VLA reaches these contact-sensitive stages more frequently, even without full task completion.Source: §4.3 · p. 8

4.4 Ablation Studies

• Removing the Wrist Predictor reduces average LIBERO success from 98.5% to 97.5% and Long success from 95.2% to 93.6%; removing W2-CoT reduces average success to 98.0%.The findings indicate complementary contributions: structured annotation prediction shapes the task-conditioned interface, while future-wrist prediction provides a localized predictive objective.Source: Table 4 · p. 9, §4.4 · p. 9
• The 16-token latent interface achieves 98.5% average LIBERO success at 110.58 ms latency, while explicit CoT decoding takes over 1.5 seconds per action chunk; wrist-only prediction performs best.Increasing latent tokens from 4 to 32 yields no consistent improvement and gradually increases latency. The paper attributes wrist-only prediction’s advantage to wrist observations capturing evolving gripper–object geometry, contact, alignment, and release, whereas joint-view prediction can add cost and interference.Source: Table 4 · p. 9, §4.4 · p. 10

4.5 Qualitative Analysis

• Attention shifts among target objects, grippers, and contact areas as manipulation progresses, drawing on task-relevant evidence from main and wrist views.Visualizations show attention moving from graspable regions to the held object and target during LIBERO transport and alignment, and toward cloth–stain contact during Table Cleaning wiping.Source: Figure 5 · p. 10, §4.5 · p. 10

• W2-VLA demonstrates strong performance on LIBERO, RoboTwin 2.0, and real-world tasks across single-arm and bimanual settings.Source: §5 · p. 10
• W2-VLA uses a compact latent interface shaped by structured W2-CoT supervision to connect global task context with wrist-local prediction.Conditioned on the interface and wrist history, a predictor forecasts future wrist latents as future-aware context for action generation.Source: §5 · p. 10
• The policy generates actions at over 80 Hz without requiring CoT generation at inference time.Source: §5 · p. 10

Supplementary Material

• The four-stage W2-CoT pipeline selects candidate trajectory segments, grounds them in physical and visual evidence, verifies VLM proposals, then normalizes and expands annotations into frame-level supervision.The overview highlights checks for gripper-state consistency, release preconditions, wrist locality, and temporal consistency.Source: Figure 6 · p. 15, A Details of the W2-CoT Annotation… · p. 15

A Details of the W2-CoT Annotation Pipeline

• Robot states and actions identify gripper openness, end-effector motion, and action changes; keyframes near phase boundaries verify object identity, contact, placement, and local motion.The instruction, state evidence, and visual observations form the evidence package for VLM-based CoT generation; available fields and camera views vary across datasets.Source: A Details of the W2-CoT Annotation… · p. 15
• Final supervision has Subtask for manipulation progress, Reasoning for physical transition cues, and Wrist for local evidence; bimanual Wrist descriptions use ordered left=...; right=... fields.Wrist evidence includes contact, object motion, placement stability, and gripper-object separation.Source: A Details of the W2-CoT Annotation… · p. 15
• Physical checks reject grasping with an open gripper, require prior holding for release and stable-grasp evidence for carrying, and constrain each bimanual wrist description to its local interaction.Temporal checks prevent implausible phase regressions; jitter and retries do not create spurious stages, and withdrawal is retained only when sustained and meaningful.Source: A Details of the W2-CoT Annotation… · p. 15, Figure 6 · p. 15
• Language normalization consistently describes approach, grasp, transport, release, and post-release withdrawal while preserving each stage’s physical meaning.Descriptions specify supported state cues, including gripper opening or closure, fingertip contact, carried object and destination, object stability, and gripper-object separation.Source: A Details of the W2-CoT Annotation… · pp. 15–16

A.1 Representative Generation and Verification Templates

• LIBERO uses episode-level planning with segment-wise visual grounding, RoboTwin uses contact-sheet-based state-anchored bimanual adjudication, and real-world annotations adapt the RoboTwin-style interface to task-specific constraints.The deterministic post-generation projection is listed separately from the VLM prompts.Source: §A.1 · p. 16
• W2-CoT covers actionable arm and gripper behaviors, with Subtask expressing progress, Reasoning encoding transitions and contact dynamics, and Wrist grounding evidence in local visual cues.The fields include behaviors such as reach, grasp, carry, align, release, and handover, and support task decomposition and subtask-level reasoning across trajectories.Source: §A.1 · p. 16, Figure 7 · p. 16

B.1 Model Architecture.

• W2-VLA uses Qwen3-VL-4B-Instruct, whose language model has approximately 4.44B parameters, 36 layers, hidden dimension 2560, 32 attention heads, and 8 key-value heads.Source: §B.1 · p. 16
• Its vision encoder has 24 transformer layers, hidden dimension 1024, patch size 16, temporal patch size 2, and projects features into the 2560-dimensional language space; Qwen3-VL runs in bfloat16.Source: §B.1 · p. 16
• The DiT-B action head has 16 blocks, 768-dimensional tokens, 12 attention heads, adaptive timestep normalization, dropout 0.2, and interleaved self- and cross-attention.Source: §B.1 · p. 16
• For LIBERO it predicts 7-dimensional delta end-effector actions; for RoboTwin 2.0 and real-world tasks it predicts 14-dimensional absolute joint positions, 7 per arm including gripper.Source: §B.1 · p. 16
• The wrist branch uses a frozen V-JEPA 2.1 encoder, a four-layer predictor, and an adapter that converts predicted future wrist latents into 32 future-aware context tokens.Source: §B.1 · p. 17

B.2 Information-Flow Masks for W2-VLA Training and Inference

• During training, the VLM predicts structured annotation targets from visual context, prompt, instruction, and latent tokens, while the DiT receives only V, I, Q, and future-aware wrist context W.Source: §B.2 · p. 18, Figure 8 · p. 18
• At inference, no annotation sequence is decoded; the prompt still structures the latent interface, and the DiT uses the same deployable context as in training: V, I, Q, and W.Source: §B.2 · p. 18, Figure 8 · p. 18

B.3 Training Details and Simulation Setup

• Training uses AdamW with gradient clipping at 1.0 and a cosine learning-rate schedule with 5000 warmup steps and a 10^-6 minimum learning rate.AdamW settings are β1 = 0.9, β2 = 0.95, ϵ = 10−8, and weight decay 10−8.Source: §B.3 · p. 18
• Qwen, the JEPA predictor, wrist-context adapter, and action head are trainable, while V-JEPA 2.1 is frozen.Learning rates are 1.0 × 10−5 for Qwen3-VL, 5.0 × 10−5 for the JEPA predictor, 5.0 × 10−5 for the wrist-context adapter, and 1.0 × 10−4 for the action head.Source: §B.3 · p. 18

C Real-world Experimental Details

• Figure 9 presents the real-world experiments setup.Source: Figure 9 · p. 19

B.3 Training Details and Simulation Setup

• Both main and wrist views are resized to 224 × 224, and short boundary sequences are padded with available boundary images; a discount coefficient controls JEPA prediction-loss strength.Historical wrist views or future views may be shorter than the prediction horizon at the beginning or end of an episode.Source: §B.3 · p. 19
• LIBERO training runs for 60K optimization steps on 4 A100 GPUs with per-device batch size 16 and horizon h = 8, using 8 historical and predicting 8 future wrist frames.Source: §B.3 · p. 19
• RoboTwin 2.0 training runs for 100K optimization steps on 8 B200 GPUs with per-device batch size 16 and horizon h = 16, sampling historical and future frames every other frame to use and predict 8 each.Source: §B.3 · p. 19

C.1 Experimental Setup

• The real-world experiments use three task setups, illustrated in Fig. 9.The supplied passage identifies Fig. 9 as showing the deployment setups.Source: §C.1 · p. 19
• Main views provide global localization and task context, while wrist views supply local evidence for grasping, alignment, contact, and release.The division is tested explicitly in Occluded Placement and also supports close-range verification and control in the other tasks.Source: §C.1 · pp. 19–20
• Table Cleaning tests long-horizon stage decomposition, ordered object handling, bimanual switching, and wiping until the spill is removed or substantially covered.Success requires placing all three objects in the basket and removing or sufficiently covering the spill.Source: §C.1 · p. 19
• Occluded Placement tests global-to-local grounding: neither wrist initially sees the mango or plate, and success requires placing the mango without disturbing the obstacle or losing it.The main view supports localization and route choice around or over the obstacle; wrist views support grasping, clearance, alignment, and release.Source: §C.1 · pp. 19–20
• Bimanual Plug Insertion tests contact-rich coordination: the left arm stabilizes the power strip while the right arm aligns and inserts the plug into its target socket.The task requires precise 6-DoF control, with wrist views supporting pre-contact adjustment and insertion-depth control.Source: §C.1 · pp. 19–20
• Three OOD settings vary table clutter, camera lighting, and tablecloth color.Irrelevant objects are randomly placed on the table, lighting is perturbed with a rotating colored light, and background color is changed.Source: §C.1 · p. 20

D.1 How effective is the wrist predictor?

• On LIBERO-10, the evaluation compares predicted future wrist latent tokens with future-frame latents encoded by the same visual encoder, using copied current latents as a simple baseline.The reported metrics are latent-token mean squared error and cosine similarity.Source: §D.1 · p. 20
• Across 500 episode records, JEPA achieves latent-token MSE 0.749 versus 2.183 for copy-current, and cosine similarity 0.888 versus 0.699, a gain of 0.189.Lower MSE and higher cosine similarity indicate stronger consistency with ground-truth future wrist tokens.Source: Figure 10 · p. 20, §D.1 · p. 20

D.2 More visualizations of latent-modeling-to-image attention.

• Visualizations pair attention maps with Subtask/Reasoning/Wrist descriptions for one LIBERO-10 task and two RoboTwin 2.0 tasks, using main and wrist views or main and both wrist views, respectively.The descriptions provide semantic references for the current manipulation stage.Source: §D.2 · p. 21
• Attention maps use post-softmax weights from Qwen3-VL’s final language-transformer layer, averaged across 32 heads and all 16 learned latent modeling tokens.The weights come from a no-gradient eager-attention forward pass with output_attentions=True; they are not gradients or attention rollout.Source: §D.2 · p. 21
• Views are tokenized into an 8 × 8 grid, then maps are independently min-max normalized and overlaid; brightness is relative within each panel, not comparable across views or rollout steps.Each panel shows one policy call without temporal averaging, and the CoT description is decoded separately rather than used to compute attention.Source: §D.2 · p. 21
• Across the three tasks, attention concentrates on stage-relevant target objects, active grippers, and contact areas, shifting with approach, grasp, transport or transfer, and release.For press_stapler it focuses on the approaching gripper and stapler, then their contact region; for handover_mic it follows the microphone and both grippers.Source: §D.2 · p. 21

D.3 Efficiency Analysis

• Action-generation throughput is generated action-chunk length divided by complete chunk-generation latency, not policy forward passes per second.The comparison amortizes chunk-generation latency over the actions in that chunk.Source: §D.3 · p. 21, Figure 12 · p. 22
• 70.00% mean standard-condition success rate exceeds VLA-JEPA (54.44%) by 15.56 percentage points and π0 (42.22%) by 27.78 percentage points despite lower peak throughput.Figure 12 compares success against generated actions per second, with bubble area representing model parameters.Source: §D.3 · p. 21, Figure 12 · p. 22
• 87.43 Hz for W2-VLA, 119.75 Hz for π0, and 102.12 Hz for VLA-JEPA place all three methods in a comparable 87.43–119.75 Hz regime.W2-VLA generates a 16-action chunk in 183 ms; the paper says this capacity supports real-time control in its deployment.Source: §D.3 · p. 21
• Figure 11 visualizes latent modeling tokens’ attention to main- and wrist-view image tokens across three task rollouts, paired with decoded Subtask/Reasoning/Wrist descriptions.Attention maps use final-layer post-softmax weights averaged over 32 heads and 16 latent modeling tokens, with each panel independently normalized.Source: Figure 11 · p. 22
• Figure 12 compares mean standard-condition success with generated actions per second and represents total model parameters by bubble area.The actions-per-second metric uses chunk length divided by latency to generate one complete chunk.Source: Figure 12 · p. 22

D.4 Visualization of rollouts in real-world OOD scenarios.

• W2-VLA rollout sequences show strong generalization performance across table clutter, lighting perturbations, and background variations.Figure 13 presents these three real-world OOD settings.Source: §D.4 · p. 23, Figure 13 · p. 23