PaperlayerView exact revision on arXiv

Source-linked AI summary

CW-WAM: A Structured Contact-Wrench World Action Model for Force-Sensitive Manipulation

Houzhu Ding, Xinda Qi, Luqi Wang, Ming Yang

arXiv:2610.30117v1cs.RO

TL;DR

Visual observations do not explicitly provide the contact and load information needed for force-sensitive manipulation. CW-WAM jointly predicts video, motion, and contact wrench, using predicted wrenches as admittance references; compared with WAM without wrench input, it raises insertion success from 25% to 70% for pegs and from 40% to 95% for adapters.

  • Problem

    WAMs predict future visual observations and actions, but visual observations do not explicitly provide the contact and load information needed for force-sensitive manipulation.

    Source: §I · p. 1

  • Method

    CW-WAM predicts time-aligned motion and wrench chunks from paired interaction history and uses predicted wrenches as references for admittance feedback.

    Source: §I · p. 1

  • Results

    25% to 70% for peg insertion and 40% to 95% for adapter insertion, compared with WAM without wrench input; across four real-robot tasks, CW-WAM also improves release success and adapts lifting height to different loads.

    Source: §VI · p. 7

  • Takeaways & Limitations

    During successful execution, predicted-wrench admittance feedback reduces force-reference RMSE and contact-force variation.

    Source: §VI · p. 7

Abstract

from arXiv · show

Force-sensitive manipulation requires contact and load information that is not provided explicitly by visual observations. We present CW-WAM, a world action model that jointly predicts video, motion, and contact wrench. Inspired by hybrid motion-force control, it contains a unified interaction module that treats both motion and contact wrench as modeling targets and predicts time-aligned motion-wrench chunks as control references. A pair-causal attention mechanism structures the motion-wrench dependencies, while a long fixed-length wrench memory retains past interaction information. The predicted wrench chunk acts as a force reference for a 20-Hz outer-loop admittance controller within a slow-fast control system that maintains force feedback and smooth motion between model queries. CW-WAM improves contact prediction and increases the insertion success rate from 25% to 70% for square pegs with 1-mm clearance per side and from 40% to 95% for power adapters, compared with WAM without wrench input. It also improves lifting-height accuracy across different loads. Our framework with an admittance controller reduces force-reference RMSE from 4.8 to 2.7 N for peg insertion and from 2.0 to 1.3 N for wiping, respectively, with smaller force fluctuations during successful execution. Project website: https://cw-wam-project.github.io/

I. INTRODUCTION · II. RELATED WORK

CW-WAM extends video–action world action models with paired motion–wrench prediction and uses predicted wrench as an admittance reference during feedback between model queries. The related work situates it among visuomotor and world-action models, contact-aware methods, and slow–fast control.

I. INTRODUCTION

  • Force sensing addresses visually ambiguous contact and load: a jammed peg can appear seated, and visually similar objects can differ in weight.Force feedback complements visual observations and enables rapid motion correction [2], [3], [4].Source: §I · p. 1
  • CW-WAM pairs past motion commands with measured wrenches, retains longer wrench histories, and predicts future motion and wrench through separate tokens, heads, and losses.Pair-causal attention links each future wrench to its paired action and earlier pairs.Source: §I · p. 1, §II · p. 2, Figure 2 · p. 2
  • Across four real-robot tasks, CW-WAM achieves the strongest overall performance among evaluated methods, improving insertion and release success, force tracking in peg insertion and wiping, and load-adaptive motion.Source: §I · p. 2
  • Predicted wrench trajectories guide admittance corrections without a separately trained corrective policy, reducing predicted-to-measured force error and contact-force variation during successful trials.Source: §I · p. 2, §II · p. 2
  • The slow–fast system combines low-rate model inference with 20-Hz admittance pose corrections and interpolates waypoints into smooth trajectories for a 1-kHz robot controller.It maintains force feedback and continuous motion between model queries during contact.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior work spans visuomotor policies,, [9],, imagined-trajectory world models, video-to-action methods,, [14], and video–action WAMs [1].Cosmos Policy plans with predicted states and values [15], while Fast-WAM omits future-video generation at inference [16]; CW-WAM adds interaction history and wrench prediction.Source: §II · p. 2
  • Contact-aware methods use tactile or force feedback [2], [3], [4], [17], [18], [19], [20], [21], predict action–wrench pairs [5], [6], or combine planning with fast control [22], [23],,, [26].CW-WAM retains longer wrench histories and uses predicted wrench trajectories as references for admittance feedback, without a learned corrective policy.Source: §II · pp. 2–3

III. STRUCTURED CONTACT-WRENCH WORLD ACTION MODELING · A. Problem formulation and shared backbone

CW-WAM jointly predicts future video, robot motion, and contact wrench for motion–force control, using shared modeling of action and wrench information. Its formulation pairs motion targets with compensated wrench targets at matching timestamps.

A. Problem formulation and shared backbone

  • CW-WAM jointly predicts future video, motion, and contact wrench to support motion–force control.The model takes RGB observations, robot state, and a language instruction as context, alongside recent action–wrench history and longer wrench memory.Source: A. Problem formulation and shared… · p. 3
  • Motion targets are paired during training with the compensated wrench measured at the same timestamp.Source: A. Problem formulation and shared… · p. 3
  • Each 10-dimensional action contains position, continuous 6-D rotation, and gripper position.Position and rotation specify a delta from the TCP pose at observation time, then convert to an absolute base-frame pose.Source: A. Problem formulation and shared… · p. 3
  • Each predicted 6-D wrench represents bias-corrected, tool-gravity-compensated force and torque in the paired predicted pose’s TCP frame.Torque is referenced to the TCP origin.Source: A. Problem formulation and shared… · p. 3
  • The shared backbone extends DreamZero [1] with a Wan2.2-TI2V-5B DiT [28] and a VAE encoding of three RGB views as a 160 × 320 composite.Source: A. Problem formulation and shared… · p. 3
  • Video, robot-state, history, and future action–wrench tokens share self-attention, with OpenCLIP image and UMT5 language features supplied through cross-attention.Source: A. Problem formulation and shared… · p. 3

B. Interaction history and paired prediction

The interaction module pairs motion commands with measured wrenches over aligned history intervals, extending wrench history with long-term memory. It predicts future motion and wrench tokens jointly using pair-causal attention.

B. Interaction history and paired prediction

  • Recent history pairs each motion-target token with measured wrench samples from the same interval.Each interval contains up to Kw high-rate wrench samples.Source: B. Interaction history and paired… · p. 3
  • A temporal wrench encoder processes each sample’s 6-D value, relative sampling time, and inter-sample interval, with validity masks excluding padding from attention and pooling.The action encoder produces a separate token, and both tokens receive the same history-interval encoding.Source: B. Interaction history and paired… · p. 3
  • Long-term wrench memory extends history with wrench-only intervals to retain preceding payload tension or sustained resistance.It uses the same encoder with additional window-time and history embeddings; both histories overlap and end at the current observation.Source: B. Interaction history and paired… · p. 3
  • Future motion and wrench tokens use separate encoders, prediction heads, and losses, while pair-causal attention lets both attend to earlier pairs and update jointly at each denoising step.The tokens share a Transformer.Source: B. Interaction history and paired… · p. 3

C. Training and inference

CW-WAM is trained with flow matching on normalized motion and wrench targets, and inference jointly generates time-aligned action–wrench chunks for control.

Training and inference

  • Flow matching mixes normalized video, action, and wrench targets with independent Gaussian noise at a shared block noise level, and predicts flow velocity with noise-level weighting and padded-target masking.Motion and wrench losses are averaged over their 10 and 6 valid channels, respectively, with λm = 1 for all modalities.Source: C. Training and inference · p. 3
  • Teacher forcing supplies clean preceding video context to noisy current blocks [1].Source: C. Training and inference · p. 3
  • The DiT and modality projectors train for 50k steps with task-balanced sampling while the VAE, OpenCLIP, and UMT5 remain frozen.Settings are H = Hs = 24, Hℓ = 150, fh = 15 Hz, and Kw = 16, corresponding to 1.6 s of recent history and 10 s of wrench memory.Source: C. Training and inference · p. 3
  • At inference, four Euler steps generate two video latent frames and 24 action–wrench pairs spanning 1.6 s, with observed latents fixed.Control uses paired predictions without decoding video to RGB.Source: C. Training and inference · p. 3

IV. SLOW–FAST MOTION–FORCE CONTROL · A. Admittance control using predicted wrenches

The controller applies task-weighted admittance corrections at 20 Hz while joint-impedance tracking runs at 1 kHz. It uses predicted wrenches as feedback references to adjust nominal motion according to each task’s contact constraints.

IV. SLOW–FAST MOTION–FORCE CONTROL

A. Admittance control using predicted wrenches

B. Continuous action-chunk execution

Chunks execute sequentially, with an adaptive C2 bridge and contact admittance maintaining execution between model queries. At 20 Hz, motion and wrench references are interpolated and aligned to measured TCP progress, while a 1-kHz controller tracks replanned joint trajectories.

B. Continuous action-chunk execution

  • Four-step queries typically take 0.5–0.6 s on one NVIDIA RTX 5090, and chunks execute sequentially while the robot holds the final nominal pose awaiting the next prediction.The query-time figure excludes network communication.Source: B. Continuous action-chunk execution · p. 4
  • An adaptive-duration C2 bridge joins each new chunk to the current commanded pose by matching velocity and acceleration, with admittance active during contact.Source: B. Continuous action-chunk execution · p. 4
  • At 20 Hz, nominal poses are interpolated with C2 continuity and predicted wrenches linearly, with wrench references selected by matching measured TCP pose to planned motion.Matched progress ϕw cannot move backward or exceed scheduled progress ϕ, accommodating execution lag while preserving reference order.Source: B. Continuous action-chunk execution · p. 4
  • Every 50 ms, the system replans a 0.5-s piecewise-quintic joint trajectory with C2 continuity, tracked by a 1-kHz joint-impedance controller with gravity and friction compensation.Source: B. Continuous action-chunk execution · p. 4

V. EXPERIMENTS · A. Tasks and data collection

The experiments assess open-loop prediction, real-robot force-sensitive manipulation, and execution quality with and without predicted-wrench admittance. Four tasks test contact-guided alignment, precise seating, sustained force regulation, and adaptation to payload.

V. EXPERIMENTS

  • The evaluation combines open-loop prediction ablations, real-robot manipulation tasks, and execution-quality tests with and without predicted-wrench admittance.The task set targets contact-guided alignment, precise seating, force regulation, and adaptation to payload.Source: §V · p. 4, A. Tasks and data collection · p. 4

A. Tasks and data collection

  • The 38-mm square peg must seat about 50 mm into a 40-mm hole with 1-mm clearance per side, testing alignment under jamming risk.Demonstrations include first-attempt insertion, recovery after failures, and alignment after light jamming; release success also requires the peg to remain seated after gripper opening.Source: A. Tasks and data collection · p. 4
  • Power-adapter insertion tests precise positioning and contact while seating a loosely placed adapter in a socket.Because the adapter is less prone to jamming than the peg, demonstrations emphasize successful first-attempt insertion with minimal lateral adjustment; success criteria match peg insertion.Source: A. Tasks and data collection · p. 4
  • Whiteboard wiping requires removing marker traces of varying orientation and curvature through spatial coverage and sustained surface-force regulation.Success requires complete marker removal after automatic eraser pickup; patterned-wiping recordings also quantify ink removal, with contact-force accuracy evaluated separately.Source: A. Tasks and data collection · p. 4
  • Spring-scale transport tests whether contact and load information guide lifting when visually identical boxes weigh 30 g or 830 g.The heavier box stretches the spring farther and requires a higher lift to clear the rim; success requires rim clearance, transport, and placement in the destination bin.Source: A. Tasks and data collection · p. 4
  • Training uses task-specific language instructions with 450 demonstrations: 213 peg insertions, 97 adapter insertions, 64 whiteboard wipes, and 76 spring-scale transports.Source: A. Tasks and data collection · p. 4
  • A wrist-mounted six-axis force-torque sensor records wrench at 150 Hz, while motion, robot state, and three RGB views are recorded at 15 Hz and timestamp-aligned.Wrench is bias- and gravity-corrected, transformed to the TCP frame, and causally low-pass filtered at 10 Hz; it captures external contact and grasped-payload loads.Source: A. Tasks and data collection · p. 5
  • Training uses eight NVIDIA H20 GPUs with AdamW, global batch size 8, peak learning rate 10^-5, and weight decay 10^-5.The schedule uses 5% linear warmup followed by cosine decay.Source: A. Tasks and data collection · p. 5

B. Model variants · C. Contact-phase prediction and ablations

The ablations show that structured interaction modeling improves contact prediction beyond loss normalization, while pair-causal attention improves contact-position and orientation accuracy and the full model leads on dynamic force metrics. The sections also define the model variants and evaluation protocol used to assess these results.

B. Model variants

  • WAM omits wrench input and output, WAM + Wrench encodes pooled wrench history, and CW-WAM w/o LTM adds paired histories, separate future streams, and pair-causal attention.The recent history pairs 24 motion commands with wrench intervals over 1.6 s; full CW-WAM adds long-term memory, while the no-LTM PC ablation replaces pair-causal with bidirectional attention.Source: B. Model variants · p. 5
  • All variants share training data, backbone initialization, and a 50k-step schedule; the WAM + Wrench normalization comparison isolates valid-channel normalization from padded-width normalization.PC and LTM ablations retain the CW-WAM objective; PC is tested without LTM, and LTM with PC retained.Source: B. Model variants · p. 5

C. Contact-phase prediction and ablations

  • Evaluation uses 373 24-step windows from 22 test episodes, with four denoising steps and seed 42; contact is defined as force magnitude at least 5 N.Windows use stride 24 plus a final tail window; dynamic windows contain contact and require a maximum pairwise force-vector difference of at least 2 N. Scores are averaged within episodes and then tasks, with equal task weighting.Source: C. Contact-phase prediction and… · p. 5
  • Contact-position and orientation errors measure mean Euclidean distance and mean SO(3) geodesic angle at contact samples, while dynamic force-vector RMSE is measured at recorded timestamps.Force-magnitude integral error is the absolute difference between trapezoidal integrals of predicted and measured force magnitudes over all 24 steps; evaluation retains approach, loading, and release samples in the 20-step window interior.Source: C. Contact-phase prediction and… · p. 5
  • 19.92 mm and 2.84 N for CW-WAM without LTM improve on WAM + Wrench with valid-channel normalization by 5.58 mm and 0.19 N, respectively.The paired reductions have 95% CIs of [3.65, 7.84] mm and [0.04, 0.33] N. Orientation error falls from 2.80° to 2.02°, and dynamic torque-vector RMSE from 0.0908 to 0.0837 N m.Source: C. Contact-phase prediction and… · p. 5
  • Full CW-WAM achieves the lowest dynamic force-vector RMSE (2.79 N) and force-magnitude integral error (2.01 N s), while the no-LTM variant retains slightly lower position and orientation errors.PC reduces no-LTM contact-position error by 2.96 mm (95% CI [2.05, 3.89]) and orientation error by 0.23° (95% CI [0.12, 0.35]).Source: C. Contact-phase prediction and… · p. 5

D. Real-robot evaluation and metrics

The real-robot evaluation compares CW-WAM variants and admittance feedback using repeated contact-task and load trials. It reports task outcomes and execution quality alongside force-reference, tracking, variation, and lifting-height measures.

D. Real-robot evaluation and metrics

  • Real-robot tests focus on the interaction module, long-term memory, and admittance feedback, after evaluating WAM, WAM + Wrench (padded), CW-WAM w/o LTM, and CW-WAM.Loss normalization and pair-causal attention are compared in open-loop tests.Source: D. Real-robot evaluation and metrics · p. 5
  • Trials use 50k-step checkpoints and begin after resetting the arm and gripper, with objects randomly placed near demonstration locations within the demonstration distribution.Source: D. Real-robot evaluation and metrics · p. 5
  • Peg-insertion and wiping force accuracy and variation are assessed against demonstration-median and CW-WAM-predicted force references.The demonstration reference is the componentwise median of phase-aligned training force profiles.Source: D. Real-robot evaluation and metrics · p. 5, Figure 4 · p. 6
  • Position-tracking RMSE compares commanded and measured TCP positions, while Fig. 4 defines force and torque variation over contact periods using centered 5-s windows.Real-robot continuous-valued metrics are summarized by medians and IQRs, and aggregate curves show pointwise medians with shaded IQR bands.Source: D. Real-robot evaluation and metrics · p. 5, Figure 4 · p. 6
  • Fig. 4 reports success rates and quality metrics for 20 trials per contact task and successful subsets, which coincide for wiping.An operator verifies outcomes from video using task criteria; each load uses 10 trials.Source: D. Real-robot evaluation and metrics · p. 5, Figure 4 · p. 6
  • Fig. 5 compares measured and demonstration force magnitudes over contact phases, Fig. 6 shows phase-aligned lifting heights, and Fig. 7 analyzes feedback over detected contact intervals.For Fig. 7, gripper state and robot motion exclude pickup and free motion from force-detected contact intervals.Source: D. Real-robot evaluation and metrics · p. 6

E. Task completion and contact behavior · F. Load-dependent lifting

CW-WAM improves task completion and contact behavior, with longer wrench memory benefiting insertion and release. It also reproduces load-dependent lifting heights more accurately than the other models.

E. Task completion and contact behavior

  • CW-WAM without LTM raises adapter insertion from 9/20 to 16/20 and release from 0/20 to 13/20 versus WAM + Wrench; peg release rises from 1/20 to 6/20, with insertion at 7/20 versus 8/20.The interaction module improves insertion and release overall, though peg insertion is slightly lower in this comparison.Source: E. Task completion and contact behavior · p. 6
  • Longer wrench memory raises peg insertion from 7/20 to 14/20 and release from 6/20 to 11/20, and adapter insertion from 16/20 to 19/20 and release from 13/20 to 16/20.It also lowers force-reference error in adapter insertion and wiping.Source: E. Task completion and contact behavior · p. 6
  • In successful peg trials, position-tracking RMSE falls from 12.5 to 2.8 mm and force-reference RMSE from 15.2 to 4.8 N versus WAM + Wrench.Force profiles also show less variation during search and seating.Source: E. Task completion and contact behavior · p. 6, Figure 5 · p. 7
  • CW-WAM improves adapter insertion completion and lowers position-tracking RMSE from 25.3 to 8.8 mm, but has higher force-reference RMSE and contact-force variation than WAM + Wrench.The passage links this pattern to learned search and firm insertion, which improve success while producing higher contact forces.Source: E. Task completion and contact behavior · p. 6, Figure 5 · p. 7
  • All methods remove ink in 20/20 simple-pattern wiping trials; complicated patterns average 85.9% removal across six selected traces.From WAM + Wrench to CW-WAM with admittance, force-reference RMSE and contact-force variation both decrease, indicating more accurate force tracking and smoother contact.Source: E. Task completion and contact behavior · p. 6, Figure 5 · p. 7

F. Load-dependent lifting

  • CW-WAM’s median peak heights are 423.6 mm for light loads and 481.8 mm for heavy loads, close to demonstration medians of 420.9 and 492.7 mm.Absolute differences from demonstration are 2.7 and 10.9 mm for CW-WAM, versus 10.4 and 24.8 mm for WAM + Wrench; WAM peaks at 464.4 and 447.3 mm, showing no higher lift for heavier loads.Source: F. Load-dependent lifting · p. 7

G. Predicted-wrench feedback · VI. CONCLUSION AND DISCUSSION

CW-WAM jointly predicts video, motion, and contact wrench, improving insertion success over a WAM baseline without wrench input. Predicted-wrench admittance feedback reduces force-reference error and contact-force variation, while evaluation remains limited to four in-distribution tasks.

G. Predicted-wrench feedback

  • Force-reference RMSE fell from 4.8 to 2.7 N for peg insertion and from 2.0 to 1.3 N for wiping with admittance versus CW-WAM alone, while success rates remained unchanged.Measured-force variation also decreased.Source: G. Predicted-wrench feedback · p. 7
  • In representative trials, force-reference RMSEs changed from 3.88 and 2.41 N without feedback to 2.69 and 1.20 N with predicted-wrench tracking.The values correspond to the representative trials shown in Fig. 7.Source: G. Predicted-wrench feedback · p. 7
  • Median contact-force variation fell from 1.86 to 1.39 to 0.94 N for peg insertion and from 1.11 to 0.92 to 0.85 N for wiping across tracking variants.Predicted-wrench tracking had the lowest variation; the larger peg-insertion reduction occurred as contact changed during search and seating, while torque errors and variations were generally small.Source: G. Predicted-wrench feedback · p. 7

VI. CONCLUSION AND DISCUSSION

  • Peg insertion success rose from 25% to 70%, and adapter insertion success from 40% to 95%, versus the WAM baseline without wrench input.CW-WAM also adapts lifting height to different loads across four real-robot tasks.Source: §VI · p. 7
  • CW-WAM jointly predicts video, motion, and contact wrench using paired action–wrench history, separate prediction streams, and longer wrench memory.Across four real-robot tasks, it adapts lifting height to different loads.Source: §VI · p. 7
  • The slow–fast execution system uses predicted wrenches as references for admittance feedback between model queries, reducing contact-force variation during successful execution.It maintains feedback between model queries while the model provides updated references.Source: §VI · p. 7
  • Evaluation covers four tasks within the demonstration distribution, using manually tuned feedback gains and predefined compliant directions.Future work will study controller settings, learn contact parameters such as the feedback-weighting matrix and contact frame alongside motion and wrench prediction, and evaluate generalization to new objects and tasks.Source: §VI · p. 7

Related papers

Loading 2610.30117v1…