PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Learning with Visual Predicted Force

Haonan Chen, Feiyang Wu, Yuxiang Ma, Mustafa Mete, Pengfei Ye, Junxuan Shen, Cheng Zhu, Aurora Ruggeri, Kelvin Cheung, Jiayuan Mao, Edward Adelson, Jiajun Wu, Robert D. Howe, Yilun Du

arXiv:2610.04741v1cs.ROcs.AI

TL;DR

Force feedback from specialized sensors adds hardware and integration overhead, so this paper estimates force from compliant Fin Ray gripper deformation and trains an action–force proposal policy using force-annotated demonstrations. Across four force-sensitive tasks, inference-time force search improves success over single-sample policy inference without force or tactile sensors at deployment.

  • Problem

    Force-aware contact-rich manipulation often relies on force or tactile sensors, which add hardware and integration overhead.

    Source: §I · p. 1

  • Method

    A visual estimator labels task demonstrations with force estimates, and an action–force policy proposes candidate actions and forces for target-based selection.

    Source: §I · p. 1, B. Force Calibration and Visual… · p. 3

  • Results

    Force search consistently improves success over single-sample policy inference across four force-sensitive manipulation tasks.

    Source: B. Force-Aware Manipulation Results · p. 6, §VII · p. 7

  • Takeaways & Limitations

    The approach enables force-aware action selection without dedicated force or tactile sensors, policy updates, or changes to the robot control interface at deployment.

    Source: §VII · p. 7

    The method requires a task-specific target force obtained from successful demonstrations, rather than automatically inferring and adapting the desired force.

    Source: §VI · p. 7

Abstract

from arXiv · show

Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action--force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.

I. INTRODUCTION

Force-aware manipulation can avoid dedicated force or tactile sensing at deployment by using visual deformation of a compliant Fin Ray gripper. The paper trains an estimator and action–force policy, then evaluates force-guided selection on four contact-rich tasks.

I. INTRODUCTION

  • Controlling interaction force matters in contact-rich tasks because excessive or insufficient force can damage objects or cause task failure.Existing force and tactile sensing options add hardware, integration, fabrication, wiring, or calibration burdens.Source: §I · p. 1
  • Visible Fin Ray gripper deformation provides cues for force estimation, and the gripper can be inexpensively fabricated by 3D printing with TPU.The estimator uses wrist-camera images of the gripper during contact.Source: §I · p. 1
  • The method labels task demonstrations with estimated forces, then trains a policy to jointly propose candidate actions and associated forces for target-based selection.At test time, it executes the candidate action whose force is closest to a task-specific target derived from demonstrations.Source: §I · p. 1
  • The approach is evaluated on berry picking, empty-can grasping, in-hand reorientation, and plug insertion without force or tactile sensors at deployment.Source: §I · p. 1

II. RELATED WORKS

The related work covers force and tactile sensing hardware, manipulation methods using contact feedback, and test-time search over policy-generated actions. The paper distinguishes its approach by using visual force estimates to label demonstrations and policy-generated forces to guide action selection.

A. Hardware-Based Force and Tactile Sensing

B. Force and Tactile Feedback for Robotic Manipulation

  • Contact feedback supports grasp adjustment, sliding, pivoting, rolling, learned manipulation, cable following, and needle threading,,,,,,.Other methods learn force or compliance control, including RETAF [40], Adaptive Compliance Policy [41], and UMI-FT [42].Source: B. Force and Tactile Feedback for… · p. 2

C. Test-Time Searching

  • Prior test-time methods rank action samples using value functions [48], consistency [49], world models [50], learned dynamics, VLM-based rewards [52], or tactile-informed contact models [53].Here, policy-generated force values guide selection, and the visual force estimator does not score candidates at test time.Source: C. Test-Time Searching · p. 2

III. METHOD

The method trains a visual force estimator, uses it to annotate demonstrations and select a task target, then trains an action–force proposal policy. At inference, it selects a proposed action whose force is closest to the target.

III. METHOD

  • The method collects paired gripper images and reference forces, annotates task demonstrations, selects a target force, and trains an action–force proposal policy.At test time, it samples action–force candidates and executes the action whose predicted force is closest to the task target.Source: §III · p. 2

A. Problem Formulation

  • The robot observes gripper deformation with a wrist camera, global robot–object interaction with a side-view camera, and its proprioceptive state.The proprioceptive state includes end-effector pose and gripper state.Source: A. Problem Formulation · p. 2
  • An end-effector action comprises Cartesian position, orientation quaternion, and gripper command; the associated force is the magnitude of axial interaction force.Source: A. Problem Formulation · p. 2
  • Because force is not directly observed at deployment without force/torque or tactile sensors, the robot must infer it from gripper deformation to select an action near the target.Source: A. Problem Formulation · pp. 2–3

B. Force Calibration and Visual Estimator Training

The visual estimator is trained on paired gripper-deformation images and reference force measurements. Its visual pipeline extracts gripper edges and predicts axial force without deployment force/torque sensing or robot proprioception.

B. Force Calibration and Visual Estimator Training

  • Calibration pairs come from 19 contact shells mounted on a reference sensor, with both a Mitsumi MMS101 resistive sensor and an ATI six-axis force/torque sensor used.The collected data comprise 117 episodes totaling 47.5 minutes; images are recorded at 30 Hz and force at approximately 20 Hz, aligned by timestamp, with Fz used for training.Source: B. Force Calibration and Visual… · p. 3
  • The wrist image is segmented, masked, converted to grayscale, and transformed into a Sobel edge image that emphasizes Fin Ray finger stripes and geometry.The mask is eroded before application to reduce artificial edges at the segmentation boundary.Source: B. Force Calibration and Visual… · p. 3
  • An encoder-only U-Net predicts signed axial force by minimizing mean-squared error against aligned Fz measurements; downstream manipulation uses predicted force magnitude.At deployment, force is inferred solely from gripper deformation, without a force/torque sensor or robot proprioceptive state.Source: B. Force Calibration and Visual… · p. 3

C. Task Demonstration Collection and Force Annotation

Demonstrations are annotated with visual force estimates, and the resulting action–force policy samples candidate pairs for target-based action selection at test time.

C. Task Demonstration Collection and Force Annotation

  • For each task, the frozen visual force estimator labels demonstrated end-effector actions with corresponding force estimates, producing force-augmented demonstrations.A task-specific target force is selected from the estimated forces in that task’s demonstration dataset.Source: C. Task Demonstration Collection and… · p. 3

D. Action–Force Proposal Policy and Inference-Time Selection

  • The policy is trained on force-augmented demonstrations to jointly generate end-effector actions and associated force values from the current observation.During training, actions are supervised by demonstrated actions and forces by visual force labels.Source: D. Action–Force Proposal Policy and… · pp. 3–4
  • At each timestep, 32 sampled action–force pairs are ranked by closeness to the task-specific target, and only the selected action is sent to the robot.The generated force is used for candidate selection, not as a robot control command; deployment does not require force/torque or tactile sensing.Source: D. Action–Force Proposal Policy and… · p. 4

IV. EXPERIMENT

The experiments assess whether visually estimated force can improve policy performance on contact-rich manipulation tasks.

  • The experiments evaluate whether visually estimated force improves policy performance on contact-rich manipulation tasks.Source: §IV · p. 4

A. Experimental Setup

The experiments use separate setups for collecting force-estimator supervision and evaluating robot manipulation, with the reference force sensor absent during deployment.

A. Experimental Setup

  • Visual-force calibration uses a hand-held Fin Ray gripper observed by a D405 camera, with a reference F/T sensor recording ground-truth force through 3D-printed fixtures.The reference sensor supplies supervision for the visual estimator and is not attached to the robot during policy deployment.Source: A. Experimental Setup · p. 4
  • The manipulation platform is an ARX arm with the Fin Ray gripper, a D405 wrist camera for gripper deformation, and a D435 side-view camera for robot–object interaction.Source: A. Experimental Setup · p. 4
  • The robot policy uses the Diffusion Policy architecture and is trained and executed on one NVIDIA GeForce RTX 5090 GPU.Source: A. Experimental Setup · p. 4

B. Force Estimation Evaluation

The visual estimator is evaluated against a motor-current baseline on held-out episodes, with lower reported mean episode MAE for ViewForce and closer force tracking in a representative trajectory.

B. Force Estimation Evaluation

C. Manipulation Evaluation

Visual force estimation outperformed motor-current estimation in accuracy and temporal tracking, while manipulation evaluation tested force-sensitive tasks using force-augmented demonstrations and inference-time candidate selection.

C. Manipulation Evaluation

  • The tasks test can slippage versus deformation, berry crushing within a narrow grasp margin, plate pivoting without slipping or lifting, and plug friction during insertion.Success requires undamaged transfer, upright reorientation, or full insertion.Source: C. Manipulation Evaluation · p. 5
  • Forty demonstrations per task were annotated by a frozen visual estimator to train a stochastic policy that jointly generates end-effector actions and force values.Source: C. Manipulation Evaluation · p. 5
  • Evaluation compared one force-ignored policy sample with 32 candidates selected by target-force proximity, using 15 trials per method and task.Both methods used the same policy checkpoint and initial-condition distribution.Source: C. Manipulation Evaluation · p. 5

A. Force-Estimation Results

  • Visual estimation achieved 0.91N MAE versus 4.20N for motor current, tracked force changes better, and stayed closer to reference measurements across episodes.The current baseline often reported force near zero load and overestimated force during loading.Source: A. Force-Estimation Results · pp. 5–6

C. Manipulation Evaluation

  • Representative force-search rollouts transferred the can and berry without visible damage, pivoted the plate while maintaining contact, and inserted the plug without slippage.The rollouts illustrate suitable force for both gentle handling and friction-dependent tasks.Source: Figure 7 · p. 6

B. Force-Aware Manipulation Results

Force search improved success across all four manipulation tasks; sampled action–force candidates show how inference-time selection chooses a force near the task target.

B. Force-Aware Manipulation Results

  • Can grasping rose from 46.7% to 86.7%, berry picking from 0% to 73.3%, reorientation from 6.7% to 46.7%, and plug insertion from 26.7% to 73.3%.Berry picking benefited most; reorientation remained most challenging.Source: B. Force-Aware Manipulation Results · p. 6
  • For a held-out soda-can observation, 32 sampled force predictions ranged from 4.45 to 6.36N, averaging 5.36N with a 0.69N standard deviation.The selected candidate predicted 4.45N, the lowest force in this example.Source: B. Force-Aware Manipulation Results · p. 6, Figure 8 · p. 6
  • Single-sample failures included a dented can, crushed berry, failed plate reorientation, and plug slippage, illustrating the need to balance grasp security with task contact conditions.Source: B. Force-Aware Manipulation Results · p. 7
  • The four tasks illustrate two roles for gripping force: balancing grasp stability against damage and providing friction for reorientation and insertion.The results indicate visually estimated force can guide action selection across both interaction types.Source: B. Force-Aware Manipulation Results · p. 7

VI. LIMITATIONS

The method has boundaries in target-force selection, force components, and online feedback; the conclusions report consistent gains across four tasks and frame gripper deformation as a visual sensing signal.

VI. LIMITATIONS

  • The method requires a task-specific target force from successful demonstrations; automatic adaptation to object properties, contact state, and task phase remains future work.Source: §VI · p. 7
  • The estimator recovers only normal gripping force Fz, not tangential forces or torques; full-wrench estimation would require richer multi-axis deformation cues.Source: §VI · p. 7
  • Inference-time selection uses policy-generated force predictions without online visual force feedback, limiting correction of prediction errors and unexpected disturbances.Source: §VI · p. 7

VII. CONCLUSIONS

  • The approach estimates normal gripping force from soft-gripper deformation and uses force search without dedicated force or tactile sensors, policy updates, or deployment control-interface changes.Source: §VII · p. 7
  • Across four force-sensitive tasks, force search consistently improved success over single-sample policy inference.The authors conclude visual gripper deformation can guide policy inference as well as estimate force.Source: §VII · p. 7
  • The authors suggest compliant robot structure can serve as a visually observable sensing medium and encourage low-cost systems that recover interaction signals from deformation.Source: §VII · p. 7

Related papers

Loading 2610.04741v1…