FAWAM: Force-Aware World Action Models for Closed-Loop Contact-Rich Manipulation

• Contact states such as weak contact, excessive pressure, or jamming can be difficult to infer from images alone, motivating complementary force and tactile feedback [1] [3] [5].Source: §1 · p. 1
• Existing force-aware methods mainly use force as an observation or for reactive control, without predicting future contact forces to guide execution-time corrections.Source: §1 · p. 1
• FAWAM encodes recent force/torque histories, jointly predicts future actions and force signals, and uses predicted-versus-sensed force discrepancies to refine actions online.Source: §1 · p. 2
• 36.25% and 21.25% gains over the best vision-only and force-aware baselines, respectively, are reported for FAWAM across contact-rich tasks.Source: §1 · p. 2

• Force-aware policies use force in feature fusion, compliance learning, curricula, reactive refinement, hybrid control, or auxiliary prediction, but typically lack future force modeling and prediction-guided correction [15] [18] [19] [20].Source: §2 · p. 2
• World action models predict visual or latent dynamics through two-stage, joint-generation, or unified approaches, while force prediction and correction remain underexplored [13] [24] [25] [26] [27] [28] [29] [30].Source: §2 · p. 3
• DAgger and robotic variants aggregate expert labels on learner-visited states [38] [39] [43] [45], while residual policies target local execution errors [47] [50] [51].CR-DAgger collects smooth delta corrections with force feedback through a compliant intervention interface [46].Source: §2 · p. 3

3.1 Overview

• The overview connects force-conditioned action generation and future force prediction with a separate online residual-correction stage.The action model encodes recent force/torque histories and uses them to condition action generation.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Force-Envisioned Action Model

• The Force-Envisioned Action Model jointly predicts future actions and wrench trajectories, using force-conditioned generation to model how robot motions relate to contact consequences.It is trained in two stages: first the video generation model, then an action decoder trained to predict action and wrench trajectories together.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 Force-Guided Residual Correction

• During execution, a 10 Hz Force-Guided Residual Corrector adjusts remaining actions within an action chunk using the predicted force trajectory and real-time feedback.This is intended to address unexpected contact variations that are difficult to handle with the base action chunk's open-loop execution.Source: §3.1 · p. 3, §3.3 · p. 5

3.3.1 Residual Corrector Design

• The residual corrector uses the base action chunk, predicted wrench reference, and sensed wrench feedback, with a gate controlling whether corrections are applied.The gate uses a conservative threshold of 0.8 to avoid perturbing nominal executions.Source: §3.3.1 · p. 5

3.3.2 Correction Dataset Collection and Training

• Correction training combines human-intervention and non-intervention segments, labeling residuals and gates to correct abnormal executions while preserving nominal behavior.The correction data is collected through FACTR force-feedback teleoperation; tasks include controlled changes to inclination, table height, or box position.Source: §3.3.2 · p. 5, Figure 3 · p. 6

4.1 Experimental Setup

• Experiments use an FR3 arm across four varied tasks, with 90 shared training demonstrations per policy, 20 correction-target episodes for FAWAM’s corrector, and 20 evaluation trials per method per task.Vision-only baselines are π0.5 and GE-Act; force-aware baselines are ForceVLA and TA-VLA.Source: §4.1 · p. 6

4.2 Main Experiment Results

• FAWAM reaches an 85% average success rate, outperforming vision-only and force-aware baselines across tasks; force-aware baselines average 63.75%, while vision-only policies average below 50%.Without residual correction, FAWAM achieves 73.75% without additional correction data; the residual corrector further improves execution robustness.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Ablation Study

• Force-conditioned observation alone adds 7.5%, joint action-force prediction adds a further 17.5%, and residual correction alone adds 18.75%, while remaining below full FAWAM.The full model completes tasks faster than all ablated variants, with force-guided correction adjusting actions using sensed and predicted forces.Source: §4.3 · p. 7, Table 2 · p. 7, Figure 4 · p. 7

4.4 Robustness under Execution-Time Perturbations

• Across four manually perturbed tasks, adding correction consistently improves robustness; discrepancy-driven updates recover stable contact and complete tasks where open-loop chunks may jam or topple.Perturbations include tilting the board, changing cucumber height, moving the box, and changing the vase-tail height.Source: Figure 5 · p. 8, §4.4 · p. 8, Table 3 · p. 8

5 Conclusion

• FAWAM uses force as both a predictive signal for contact-aware action generation and a corrective signal for execution-time feedback.The approach comprises a Force-Envisioned Action Model and a Force-Guided Residual Corrector.Source: §5 · p. 8
• FAWAM achieves an 85.0% average success rate, improving over the strongest vision-only and force-aware baselines by 36.25% and 21.25%, respectively.The reported results are from real-world experiments.Source: §5 · p. 8
• Ablations show complementary improvements from action-force prediction and residual correction, while perturbation experiments show recovery from unexpected contact changes.The paper concludes that force prediction and online correction are important for robust contact-rich manipulation.Source: §5 · p. 8

• The lack of large-scale video datasets with synchronized force/torque signals keeps force out of video generation, coupling actions and wrenches only in the action branch.The authors identify synchronized force/torque data as the constraint on integrating force into video generation.Source: §6 · p. 8
• Residual correction is trained on limited human-intervention episodes, which may miss diverse failure modes and limits recovery under large deviations.Source: §6 · p. 8
• End-effector wrench prediction supplies only indirect contact information; the authors propose local contact frames and dense tactile signals for more direct, generalized action guidance.Source: §6 · pp. 8–9

• RealSense RGB is recorded at 640 × 480 and resized to 256×192; wrist wrenches are sampled at 120 Hz, low-pass filtered at 10 Hz, and encoded from the latest 0.5 s (60 frames).The force sensor is an ATI wrist force/torque sensor, and filtering uses a second-order Butterworth filter.Source: §A.1 · p. 13
• Training runs on 8 NVIDIA A100 GPUs: the video world model and then the frozen-model force-aware action decoder are each trained for 30,000 steps.The respective learning rates are 3 × 10−5 and 1.5 × 10−4, with λF = 1 balancing action and future-wrench losses.Source: §A.1 · p. 13
• The residual corrector is separately trained for 500 epochs on balanced batches of 64 intervention and 64 non-intervention samples, using a dual-head MLP with residual-action and gate outputs.Its inputs include filtered wrench history, predicted future wrenches, upcoming base actions, past tracking error, and current proprioception.Source: §A.1 · p. 13
• The setup uses a Franka robot for execution, a FACTR leader for force-feedback teleoperation, and multiple RealSense cameras for multi-view observation.Source: Figure 6 · p. 13

• Without joint action-force prediction, the residual-only variant has no predicted wrench trajectory, future-wrench preview, or measured-predicted tracking error.Source: §A.2 · p. 13
• Its separate corrector takes the contact feature, upcoming base action chunk, and current proprioceptive state; the contact feature is encoded from recent measured wrench history with the frozen force encoder.Source: §A.2 · p. 14
• The residual-only gate uses the same threshold as the full corrector, and its correction target, gate labels, and intervention-based training protocol remain the same.Source: §A.2 · p. 14

• Across the four tasks, full FAWAM achieves the best combination of high success rate and low completion time, reducing inefficient recovery behaviors during contact-rich execution.Completion-time results are most informative alongside success, since shorter rollouts can also reflect brittle open-loop behavior or early termination.Source: §A.3 · p. 14
• On Erase Board, force histories and residual updates maintain consistent wiping pressure, avoiding repeated re-contact attempts caused by lost contact or inappropriate pressure.Source: §A.3 · p. 14
• On Peel Cucumber, some ablations are faster but less successful, consistent with stereotyped motions that adapt unreliably to local height and contact variation.FAWAM trades some local adjustment time for substantially better task completion.Source: §A.3 · p. 14
• On Pivot Box, joint action-force prediction supports physically consistent pushes, while residual correction prevents small force errors from accumulating into ineffective contacts.Source: §A.3 · p. 14
• On Wipe Vase, sensed force histories and predicted wrench references help adapt pressure and tangential motion over the curved surface, avoiding prolonged slipping or over-pressing.Source: §A.3 · p. 14

A.4 Additional Perturbation Examples

• Figures 7–10 provide additional qualitative visualizations for the perturbation and correction experiments.Source: §A.4 · p. 14, Figure 7 · p. 15, Figure 8 · p. 15

A.5 Additional Visualization Supplement

• Figures 11–14 show representative executions across four tasks, illustrating joint force-action prediction and online correction during contact and interaction errors.The examples depict stable contact, adaptation to local geometric changes, and recovery from interaction errors.Source: §A.5 · p. 16, Figure 11 · p. 17, Figure 12 · p. 17

A.6 Additional Analysis on Residual Activation Threshold

• τ = 0.8 achieves the best or near-best performance on Peel Cucumber and Wipe Vase; thresholds that are too low or high reduce success.Low thresholds can perturb nominal execution, while overly high thresholds may miss necessary corrections.Source: §A.6 · p. 19, Figure 15 · p. 19