PaperlayerView exact revision on arXiv

Source-linked AI summary

FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

Zhi Cao, Yining Hong, Jiaolong Yang, Yejin Choi, Li Fei-Fei, Jiajun Wu, Huang Huang

arXiv:2610.19155v1cs.RO

TL;DR

Human demonstrations are scalable for dexterous robot learning, but most lack force measurements needed for contact-rich manipulation. FACT combines EgoForce human tactile data with robot demonstrations, using visual planning followed by force-aware action refinement. Across ten tasks, it outperforms baselines, including on pipetting, and experiments suggest human demonstrations contribute nearly as much as robot demonstrations during fine-tuning.

  • Problem

    Human demonstrations offer scalable manipulation experience, but most human datasets omit interaction forces, while collecting force-supervised robot data at comparable scale is expensive and difficult.

    Source: §I · p. 1, §III · p. 2

  • Method

    FACT combines EgoForce human tactile data with robot demonstrations, jointly predicting future video, tracks, and coarse actions before refining actions alongside future force.

    Source: §I · p. 2

  • Results

    40% pipetting success for FACT versus 0% for T-Rex; FACT consistently outperforms baselines across the ten-task suite.

    Source: B. Main Results · p. 6

  • Takeaways & Limitations

    Across ten dexterous tasks, experiments suggest human demonstrations can substantially reduce task-specific robot data needed for fine-tuning.

    Source: §VI · p. 8

Abstract

from arXiv · show

Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements. Dexterous manipulation requires not only visual and proprioceptive understanding but also the ability to anticipate and regulate interaction forces. To scale force-aware human-robot co-training, we introduce EgoForce, a large-scale human tactile dataset capturing force-rich manipulation experience, and combine it with robot demonstrations for pretraining. We present FACT, a robot foundation model that cascades from prospective visual planning to force-aware refinement for dexterous manipulation. FACT comprises two jointly trained components. A Track-Guided Coarse Planner jointly predicts actions, future video, and visual tracks. Future video and visual tracks also provide a shared task representation that facilitates cross-embodiment knowledge transfer and coarse action generation. Building on this coarse plan, a Force-Action Joint Refiner jointly predicts future force and refines the coarse action, allowing anticipated force and action to shape each other during refinement. We evaluate the model on ten tasks requiring fine-grained coordination and force feedback, including tool use, musical instruments, and precision laboratory procedures. Using as few as 10 robot trajectories with 90 human trajectories for fine-tuning, FACT outperforms both baselines, with success-rate gains of up to 60 percentage points over T-Rex. Ablations show that force supervision in human pretraining data is crucial for downstream robot performance after fine-tuning, while visual track prediction and training with future-force prediction improve dexterous execution. Experiments varying the human-robot data ratio during fine-tuning further suggest that FACT learns from human demonstrations nearly as effectively as from robot demonstrations.

I. INTRODUCTION

Human demonstrations are scalable, but most lack force measurements needed for contact-rich dexterity. FACT combines force-rich human data with robot demonstrations, pairing visual planning with force-aware action refinement.

I. INTRODUCTION

  • FACT jointly predicts future video, visual tracks, and coarse actions, then refines actions while predicting future force so action and force shape one another.The shared visual representation supports knowledge transfer between human and robot embodiments.Source: §I · p. 2
  • EgoForce provides approximately 2k hours of human manipulation across 40k episodes, 1k tasks, and 1k demonstrators for force-aware pretraining.The collection platform captures synchronized vision, hand motion, and tactile force.Source: §I · p. 2
  • Across ten real-world tasks, force supervision in human pretraining benefits contact- and force-control tasks, while track and future-force prediction improve dexterous execution.Human demonstrations contribute nearly as much as robot demonstrations during fine-tuning, according to data-ratio experiments.Source: §I · p. 2

II. RELATED WORK

  • Prior human corpora and human-to-robot learning approaches are largely force-blind, while FACT jointly denoises continuous future force and action during refinement.T-Rex instead conditions coarse actions on compressed force history.Source: §II · p. 2
  • FACT uses future video and visual tracks as an embodiment-agnostic task representation to ground coarse action generation in anticipated motion.Related work has used visual point tracks as policy interfaces, transfer media, and cross-domain motion representations.Source: §II · p. 2

III. THE EGOFORCE DATASET

EgoForce collects synchronized visual, motion, and calibrated tactile data at scale, with a distribution intentionally weighted toward contact-rich fine manipulation. FACT uses shared visual predictions for coarse planning across embodiments and force-aware refinement for robot execution.

III. THE EGOFORCE DATASET

  • Automatic calibration maps raw A/D readings to physical force using synchronized reference pressure and glove measurements.Pressure is swept from 0 to 50 psi, and per-element force is computed as Fi = PiAi before fitting a calibration mapping.Source: §III · p. 3
  • EgoForce contains approximately 2,000 hours across 40k episodes, 1,000 tasks, 120 scenes, and 1,000 demonstrators, with synchronized egocentric RGB, hand motion, and calibrated tactile force.The platform uses three egocentric cameras and bimanual tactile gloves.Source: §III · pp. 2–3
  • Fine manipulation comprises over two thirds of EgoForce; deformable manipulation, assembly and construction, and storage and organization account for 21.0%, 17.3%, and 14.8%.Frequent skill annotations include fold, regrasp, thread, smooth, and crease; similar visual states can require different force profiles.Source: §III · p. 3, Figure 2 · p. 3

IV. THE FACT FRAMEWORK

  • FACT's coarse planner predicts future video, tracks, and actions for shared human–robot task representations; its refiner jointly refines action and predicts future fingertip force.Pretraining combines EgoForce with 100 hours of robot teleoperation demonstrations.Source: §IV · p. 3
  • At inference, FACT takes visual and robot-state observations, task instructions, track history, and tactile history, and outputs a 16-step action chunk with future-force, track, and video predictions.The action chunk runs at 20 Hz, with future force predicted at 5 Hz refresh slots.Source: §IV · p. 3

A. Track-Guided Coarse Planner

The Track-Guided Coarse Planner jointly predicts future video, tracks, and actions, using a shared temporal representation to ground coarse motion in anticipated visual change. Its per-token noise conditioning supports clean context and noisy targets in the same model pass before handoff for force-aware refinement.

A. Track-Guided Coarse Planner

  • The planner jointly predicts future video, visual tracks, and actions with a block-causal video DiT that models interaction autoregressively over time.The coarse plan is passed to the Force–Action Joint Refiner at an intermediate noise level.Source: A. Track-Guided Coarse Planner · pp. 3–4, Figure 3 · p. 4
  • Task instruction and anchor frame provide cross-attention conditioning, while robot state, past tracks, and conditioning video latents remain clean context for jointly denoised future video, actions, and tracks.Track tokens attend only to tracks, preserving a dedicated geometric pathway.Source: A. Track-Guided Coarse Planner · pp. 3–4
  • Per-token noise conditioning keeps context at τ=0 and lets future video, actions, and tracks use sampled noise levels, enabling action denoising to stop at an intermediate level.The refiner continues the action refinement after the planner handoff.Source: A. Track-Guided Coarse Planner · p. 4, Figure 3 · p. 4
  • The planner jointly optimizes visual dynamics, action, and track prediction, with visibility-masked track losses and motion-weighted emphasis on informative trajectories.Position and smoothness terms are computed from one-step reconstruction.Source: A. Track-Guided Coarse Planner · p. 4

B. Force–Action Joint Refiner

The Force–Action Joint Refiner builds on the coarse visual plan to jointly refine actions and predict future fingertip forces. Observed tactile signals provide current contact context while predicted forces and actions shape one another during refinement.

B. Force–Action Joint Refiner

  • At the τa=0.4 handoff, the refiner uses future video and tracks while refining the remaining action and predicting future fingertip force.Observed tactile signals supply contact context, and force and action reciprocally shape one another during refinement.Source: B. Force–Action Joint Refiner · p. 4
  • Unlike T-Rex, FACT predicts a continuous future fingertip-force trajectory jointly with the action suffix, rather than using force only as vector-quantized observation codes.Observed fingertip force enters as clean context; the future-force trajectory starts from noise and is jointly denoised with the action suffix.Source: B. Force–Action Joint Refiner · pp. 4–5
  • Cached cross-attention key–values let the refiner reuse the fixed coarse visual plan across all four Euler steps and four tactile refreshes.The coarse hidden states are projected once, avoiding repeated computation of the Track-Guided Coarse Planner.Source: B. Force–Action Joint Refiner · p. 5
  • Four joint Euler steps use action and force schedules of τa: 0.4→0 and τf: 1→0, updating only the remaining action suffix and unobserved future-force slots.Executed action prefixes and observed force slots stay fixed; newly observed interaction enters at the next tactile refresh.Source: B. Force–Action Joint Refiner · p. 5
  • The refiner jointly optimizes action refinement and future-force prediction, with λf=0.5 in all experiments.The objective covers the action suffix and future force.Source: B. Force–Action Joint Refiner · p. 5

C. Implementation Details

FACT uses a large Track-Guided Coarse Planner and a smaller Force–Action Joint Refiner, trained end to end with human and robot demonstrations. The training recipe applies force supervision during pretraining rather than adding it after vision-only pretraining.

C. Implementation Details

  • The Track-Guided Coarse Planner is a 5B video DiT, while the Force–Action Joint Refiner is a 6-block bidirectional transformer.Training uses four teacher-forced blocks spanning 3.2 seconds; inference generates one block at a time autoregressively.Source: C. Implementation Details · p. 5
  • The model is trained end to end: the video backbone is fine-tuned, while the track codec, joint refiner, tactile VQ, and modality embeddings are trained from scratch.Tactile dropout zeros all tactile tokens with probability 0.1 per sample.Source: C. Implementation Details · p. 5, D. Training Recipe · p. 5

D. Training Recipe

  • Pretraining mixes 2,000 hours of EgoForce demonstrations with 100 hours of T-Rex robot teleoperation data, sampled proportionally to hours.Calibrated glove forces supervise the force pathway from the first training step.Source: D. Training Recipe · p. 5
  • The pretraining-recipe ablation is evaluated across all ten tasks.The supplied caption identifies the scope but does not report its results.Source: Table II · p. 6

V. EXPERIMENTS

FACT is evaluated on ten contact-rich robot tasks using a 10-robot/90-human trajectory mixture per task. It outperforms the baselines on reported tasks, while ablations show benefits from force supervision, visual tracks, and future-force prediction.

A. Experimental Setup

  • Each task uses a 100-trajectory fine-tuning mixture of ten robot trajectories and 90 human demonstrations, evaluated across ten contact-rich tasks.Binary tasks score success over ten rollouts; music tasks count clean notes, and volume adjustment reports both dial engagement and mean error among engaged trials.Source: A. Experimental Setup · p. 6

B. Main Results

  • FACT consistently outperforms both baselines across the ten-task suite, including 40% pipetting success versus 0% for T-Rex and π0.5.It sounds 6.7 of 10 harp notes cleanly, versus 2.1 for T-Rex and 0.5 for π0.5; pipette-dial engagement is 5/10 with 6.4 µL mean error, versus 2 engagements and 14 µL for T-Rex.Source: B. Main Results · p. 6

C. Ablation Studies

  • Removing force prediction substantially degrades contact-sensitive tasks, while removing track prediction causes smaller but consistent drops on precise-motion tasks.Force-prediction removal reduces test-tube cap opening from 80% to 40%, harp notes from 6.7 to 0, and dial engagement from 5/10 to 2/10; track-prediction removal reduces pipetting from 40% to 20%, card shuffling from 30% to 20%, and xylophone notes from 5.4 to 3.9.Source: C. Ablation Studies · p. 6
  • Increasing the robot share of the fine-tuning mixture yields modest gains on flip-cap opening, multi-object grasping, and dial engagement, while pipette-volume error remains similar.Across 10:90, 30:70, and 50:50 robot-to-human ratios, flip-cap opening changes from 50% to 60%, multi-object grasping from 40% to 50%, dial engagement from 5/10 to 6/10, and volume error is 6.4, 5.2, and 6.1 µL.Source: Figure 6 · p. 7, C. Ablation Studies · pp. 7–8
  • Tactile supervision lowers validation loss by 10–14% at every tested human-data scale, with the benefit persisting as data grows.From 1k to 2k hours, validation loss decreases by 11% with tactile supervision versus 8% without it.Source: Figure 7 · p. 7, C. Ablation Studies · p. 7
  • Removing human force supervision causes large drops on contact-sensitive tasks, whereas removing human pretraining data also degrades performance.Without force supervision, test-tube cap opening falls from 80% to 0%, pipetting from 40% to 20%, dial engagement from 5/10 to 2/10, and xylophone performance from 5.4 to 2.2 clean notes; multi-object grasping remains at 40%. Robot-only pretraining yields 0% pipetting and flip-cap opening, 1.2 clean harp notes, and no successful dial engagement.Source: C. Ablation Studies · pp. 6–7
  • Adding 90 human fine-tuning demonstrations to ten robot trajectories improves shoelace tying, card shuffling, and pipette-dial performance.Shoelace tying rises from 10% to 20%, card shuffling from 10% to 30%, and dial engagement from 4/10 to 5/10 while mean error falls from 11.2 to 6.4 µL.Source: Table III · p. 6, C. Ablation Studies · p. 7

VI. CONCLUSION

FACT combines force-rich human experience with robot demonstrations in a model that links prospective visual planning and force-aware action refinement. Experiments across ten dexterous tasks show complementary roles for tracks and force prediction and reduced task-specific robot-data needs through human demonstrations.

VI. CONCLUSION

  • Across ten dexterous tasks, experiments show complementary roles for visual tracks and force prediction and that human demonstrations can reduce task-specific robot data required for fine-tuning.FACT couples future video and tracks for coarse motion with predicted future force for fine-grained action refinement.Source: §VI · p. 8

APPENDIX A MODEL AND TRAINING DETAILS

The appendix summarizes FACT’s model configuration, data pipeline, and inference schedule.

APPENDIX A MODEL AND TRAINING DETAILS

  • FACT’s configuration is listed in Table IV; the coarse planner is a fully fine-tuned Wan2.2-TI2V-5B diffusion transformer, while several projections and the entire refiner are trained from scratch.The VAE, umT5-XXL text encoder, and CLIP image encoder remain frozen.Source: Appendix A · p. 9, Table IV · p. 9
  • Visual tracks are extracted offline from hand and scene points, and demonstrations are resampled to modality-specific rates before encoding.Track extraction follows 250 points per block through episodes; actions and tracks are resampled to 20 Hz, video to 10 Hz, and tactile force to 5 Hz.Source: Appendix A · p. 9
  • At inference, the coarse planner runs six Euler steps to τ=0.4, then the refiner runs four joint Euler steps and executes a 16-step chunk at 20 Hz.A further planner pass at τ=0.4 produces hidden states cached as the refiner’s cross-attention memory.Source: Appendix A · p. 9

APPENDIX B FORCE–ACTION JOINT REFINER

The Force–Action Joint Refiner conditions on current and historical fingertip forces while jointly refining actions and predicting future forces. At inference, it can re-refine the unexecuted action suffix when new tactile measurements arrive.

APPENDIX B FORCE–ACTION JOINT REFINER

  • Each hand contributes five fingertip six-axis wrenches, producing a 60-dimensional force observation per time step.The refiner receives a 16-sample force-history window spanning 3.2 seconds at 5 Hz, with missing initial samples zeroed and masked.Source: Appendix B · p. 9
  • A shared convolutional encoder quantizes each fingertip’s force history into one 256-dimensional code, with 64 codebook entries.The VQ objective combines reconstruction MSE and a commitment term weighted by 0.25.Source: Appendix B · p. 9
  • A refiner instance processes 76 tokens spanning actions, current forces, force-history codes, and future-force slots, with action and force heads predicting flow-matching velocities.The layout includes 16 action tokens, 10 current-force tokens, 10 history tokens, and 40 future-force tokens.Source: Appendix B · pp. 9–10
  • Training combines action-suffix flow-matching error, half-weighted error on unobserved future-force slots, and VQ loss; tactile dropout prevents reliance on touch alone.Each sample instantiates the refiner for four blocks and four refresh offsets, and future-force targets are measured forces at four slot times.Source: Appendix B · p. 10
  • At inference, four joint Euler steps update only the unexecuted action suffix and unobserved force slots, while executed actions and observed forces remain fixed.The refiner reruns at subsequent 5 Hz tactile refreshes against cached coarse memory; predicted future force is recorded for analysis, not used as a control reference.Source: Appendix B · p. 10

APPENDIX C HARDWARE AND DATA COLLECTION

The collection systems capture synchronized egocentric video, hand motion, and tactile force for human and robot demonstrations. Human demonstrations are mapped into the robot’s action and force spaces for pretraining or fine-tuning.

EgoForce collection (pretraining data)

  • EgoForce pairs head and wrist cameras with tactile gloves that record calibrated per-taxel forces, synchronized with video for frame-level pairing.The gloves also record bend and orientation data; calibration uses a pressure–A/D curve and converts taxel pressure to force.Source: Appendix C · p. 10

Human action and force labels

  • Human hand motion is estimated from stereo video and glove sensors, while fingertip taxel forces are reduced to 6-axis wrenches for force supervision.The pressure array does not observe tangential components or twisting moment, which are set to zero.Source: Appendix C · p. 10

Robot platform

  • Robot rollouts use two bimanual arms with dexterous hands, three camera views, ten fingertip wrenches, and a 62-D action space.Actions combine delta-base end-effector poses with absolute hand-joint targets.Source: Appendix C · p. 10

Teleoperation

  • Robot teleoperation logs synchronized camera, state, tactile, and commanded-action data; ten successful trajectories are retained per task.Operators use Manus gloves and wrist trackers, with hand motion retargeted to the robot hands.Source: Appendix C · pp. 10–11

Human demonstrations for fine-tuning

  • The 90 human fine-tuning demonstrations per task use a lighter interface on the evaluation tasks, rather than EgoForce recordings.Wrist poses, finger angles, tactile readings, and camera views are mapped into the robot demonstration action and force spaces.Source: Appendix C · p. 11

Robot pretraining data

  • Pretraining also mixes in 100 hours of T-Rex robot teleoperation data from a different bimanual platform.Its end-effector-pose action representation permits the same 62-D action space across both arm embodiments.Source: Appendix C · p. 11

APPENDIX D IMPLEMENTATION DETAILS OF BASELINES

The appendix specifies baseline training and input differences, then defines FACT’s ablations and pretraining-recipe variants. π0.5 lacks tactile input, while T-Rex receives fingertip wrenches.

π0.5

  • π0.5 uses three camera views, instruction, and robot state without tactile input, encoding bimanual actions in its fixed 32-D action head.The 32-D representation uses arm pose deltas and PCA-projected hand joints; training uses full-parameter updates.Source: Appendix D · pp. 11–12

T-Rex

  • T-Rex uses the same 62-D action space as FACT and receives camera views, instruction, and ten fingers’ fingertip wrenches through its tactile encoder.Its deformation-image branch is disabled, while its FLARE visual-prediction objective remains on.Source: Appendix D · p. 12

FACT ablations

  • FACT’s ablations remove either future-force prediction from the refiner or visual-track prediction from the coarse planner.Pretraining variants keep the architecture fixed while disabling human force supervision and tactile inputs, or using robot data alone.Source: Appendix D · pp. 12–13

APPENDIX E EVALUATION TASKS AND ADDITIONAL RESULTS

The evaluation covers ten contact-rich tasks with binary, partial-credit, and dial-adjustment scoring. Failure analysis distinguishes missed contact, force precision, and maintaining dexterity under contact.

Evaluation protocol

  • Ten contact-rich tasks span binary grasp-and-adjust sequences, partial-credit harp and xylophone performance, and pipette dial adjustment scored by engagement and final-reading error.Each task has ten rollouts with randomized object placement, and all methods share scenes and objects.Source: Appendix E · p. 13

Failure modes

  • Harp and xylophone rollouts score clean notes individually, so contacts that are too light or too forceful reduce the note count.A pluck that is too forceful can snap or detune a string; one that is too light produces no note.Source: Appendix E · p. 13
  • Evaluation failures cluster around lost contact without feedback, force precision, and dexterity under contact rather than gross motion.Removing force prediction produces open-loop and force-precision failures, while removing track prediction mostly produces misplaced but force-appropriate motions.Source: Appendix E · p. 13

Evaluation tasks

  • The test-tube task requires unscrewing a cap while the other hand steadies the tube; screwdriver use requires axial pressure and torque resistance.Success requires the cap fully separated and tube returned, or the screw removed and both screwdriver-task objects returned to the table.Source: Appendix E · pp. 14–15

Task 10: Adjust Pipette Volume

  • Pipette-volume trials count as engaged only when the dial turns, then measure mean absolute error from the 400 µL target.The robot turns the dial with the right thumb and index fingertip while the left hand holds the pipette body.Source: Appendix E · p. 16

Related papers

Loading 2610.19155v1…