PaperlayerView exact revision on arXiv

Source-linked AI summary

FORCA: A Force-Aware Coding Agent for Contact-Rich Robot Manipulation

Xiao Tan, Yixuan Wang, Zhexi Wang, Wenzhen Yuan

arXiv:2610.31841v1cs.RO

TL;DR

Contact-rich manipulation can require responding to geometry and physical properties that vision alone cannot reveal, while force-enabled learned policies may require demonstrations or robot data. FORCA writes and revises programs using visual observations and wrist force/torque feedback; across five real-robot tasks, it improves manipulation outcomes and reduces contact loss versus a vision-only variant.

  • Problem

    Vision and proprioception can be insufficient for contact-rich manipulation when geometry is uncertain, physical properties are not visible, or contact must respond to surface variation and object motion.

    Source: §I · p. 1

  • Method

    FORCA writes Python programs using visual observations and wrist F/T measurements, then revises motion and force-feedback logic from force records and visual outcomes between rollouts.

    Source: §I · p. 1, §III · p. 2

  • Results

    Compared with a vision-only variant, FORCA raises drawer-opening success from 40% to 100% and door-handle success from 10% to 100%, and reduces wiping and peeling contact-loss rates from 40.6–55.9% to at most 2.2%.

    Source: §I · p. 1

  • Takeaways & Limitations

    Across five real-robot tasks, FORCA refines geometric estimates through contact, adapts programs to changed physical conditions, and maintains contact during wiping and peeling without task demonstrations or task-specific model training.

    Source: §VI · p. 7

    Each iteration takes several minutes, with agent reasoning accounting for most of the time, and the current refinement process relies on real-robot rollouts; wrist wrench measurements also do not directly reveal contact location or finger-force distribution.

    Source: §V · p. 7

Abstract

from arXiv · show

Coding agents can translate language instructions into robot programs. For contact-rich manipulation, a key challenge is to generate and refine control logic that responds to physical interaction. We present FORCA, a coding agent that takes visual observations and wrist force/torque (F/T) measurements as input to write Python programs for contact probing and force-responsive motion. Between executions, the agent inspects force records alongside visual outcomes to diagnose failures and revise both motion and force-feedback control logic. FORCA requires no demonstrations, task-specific training, or simulation. Across five real-robot tasks, we demonstrate its effectiveness in three forms of contact-rich manipulation: improving manipulation accuracy, adapting to changes in object or surface properties, and regulating contact under surface variation and object motion. Compared with a vision-only variant of the same agent, FORCA improves grasp accuracy and task success and reduces contact loss in wiping and peeling.

I. INTRODUCTION

Vision and proprioception can be insufficient for contact-rich manipulation when geometry is uncertain or contact must respond to physical variation. FORCA uses wrist force/torque measurements during execution and revisions to address these tasks.

I. INTRODUCTION

  • Vision and proprioception can be insufficient when surfaces are occluded or reflective, mass and friction are not visible, or contact must respond to surface variation and object motion.Force and contact measurements provide evidence about contact states and mechanical responses that vision alone may not reveal.Source: §I · p. 1
  • Learned force- and tactile-feedback policies use demonstrations or robot data, while supporting diverse tasks requires deciding when and how contact information should guide motion and later revisions.Source: §I · p. 1
  • FORCA writes Python programs that use measured contact to refine grasp poses and regulate motion, then revises motion and force-feedback logic using recorded force traces and visual outcomes.The agent relates traces to outcomes to diagnose failures between rollouts.Source: §I · p. 1
  • Compared with a vision-only variant, FORCA raises drawer-opening success from 40% to 100% and door-handle success from 10% to 100%, while reducing wiping and peeling contact loss from 40.6–55.9% to at most 2.2%.The evaluation spans five real-robot tasks across precision manipulation, physical adaptation, and contact regulation.Source: §I · p. 1

II. RELATED WORK

Prior manipulation agents have used visual, proprioceptive, or linguistic feedback, while force-enabled approaches often expose fixed primitives or revise targets after failure. FORCA generates and revises task-specific force-feedback logic at deployment time using real-robot interaction.

II. RELATED WORK

  • FORCA is presented as the first coding-agent framework to jointly generate task-specific force-feedback control logic and revise that logic between real-robot rollouts using recorded F/T traces and visual outcomes.Earlier force-enabled agents expose compliant or thresholded primitives, call force routines, or revise target wrenches after failed attempts.Source: §II · p. 1
  • Unlike approaches that require demonstrations or robot data, FORCA writes force-responsive controllers at deployment time and revises them from recorded F/T traces.Classical methods rely on task-specific thresholds and nominal traces; learned policies require demonstrations or robot data.Source: §II · p. 2

III. METHOD

  • Given language instructions and RGB-D observations, FORCA writes Python code with wrist F/T access, executes it on the robot, and revises it after reviewing the interaction.One execution of a program is a rollout.Source: §III · p. 2

A. Force-Aware Program Synthesis

FORCA generates task programs that combine action sequences with force-dependent logic, using an initial codebase of perception, grasping, motion-planning, and pick-and-place skills. During execution, force measurements support contact-based geometric refinement and regulate ongoing motion.

A. Force-Aware Program Synthesis

  • The initial skill codebase provides perception, grasping, motion planning, and pick-and-place modules; its components include tool localization, grasp proposal and selection, and trajectory planning.The modules are assembled from open-source components under human supervision.Source: A. Force-Aware Program Synthesis · p. 2, Figure 2 · p. 3
  • The agent writes task-specific force-dependent logic, modifying existing routines or adding measurement and control code to specify when motion stops or responds to force error.Source: A. Force-Aware Program Synthesis · p. 2

B. Contact-Responsive Execution

  • A contact probe can update surface-height or grasp-position estimates from the robot pose at contact and tool geometry, and subsequent motions use the estimate in the same rollout.Source: B. Contact-Responsive Execution · p. 2
  • For sustained contact, live force feedback can pause forward motion when contact is insufficient and resume it after contact is restored.The cucumber-peeling illustration shows this kind of condition added after comparing force records with visible peeling gaps.Source: B. Contact-Responsive Execution · p. 3, Figure 3 · p. 3

C. Execution-Grounded Program Revision

After each rollout, FORCA analyzes force and robot-state traces alongside visual observations to identify possible failure causes and revise the program. Revisions are checked before execution and stop at convergence or when a rollout or wall-clock budget is exhausted.

C. Execution-Grounded Program Revision

  • After each rollout, the agent examines visual observations with force and robot-state traces, using analysis code to relate force changes to motion and visible outcomes.Source: C. Execution-Grounded Program Revision · p. 3
  • The agent uses the records to identify possible failure causes and revise motion parameters, contact conditions, control logic, or the action sequence.In the cucumber-peeling example, it adds a wait-for-contact condition where force records align with unpeeled skin.Source: C. Execution-Grounded Program Revision · p. 3
  • Before execution, checks screen revised code and planned motions for errors, reachability, and workspace violations; revision ends at declared convergence or when rollout or wall-clock limits are reached.Source: C. Execution-Grounded Program Revision · p. 3

IV. EXPERIMENTS

The experiments assess how force cues support precise manipulation, adaptation to physical changes, and contact regulation. FORCA is evaluated on a real robot against a vision-only agent with the same foundation model and basic skills but no access to force data.

IV. EXPERIMENTS

  • The experiments test force cues for improving manipulation accuracy, adapting to physical changes, and regulating contact as surfaces vary or objects move.The evaluation combines comparisons with a vision-only agent and case studies of generated programs.Source: §IV · p. 3

A. Experimental Setup and Protocol

  • The robot setup includes a UR5e arm with wrist force/torque sensing, a WSG-50 gripper, and a fixed RealSense camera.The supplied hardware passage identifies the camera as a RealSense D455 RGB-D camera.Source: A. Experimental Setup and Protocol · pp. 3–4
  • Figure 4 illustrates FORCA estimating geometry, refining it through contact probing, and grasping to open a drawer or turn a door handle.The examples contrast successful FORCA outcomes with a missed drawer grasp and a slipping door-handle grasp from the vision-only agent.Source: Figure 4 · p. 4
  • The vision-only baseline shares FORCA’s foundation model, instructions, and basic skills, but receives no force data or force-responsive routines during generation, revision, or execution.Runs are limited to 20 rollouts and four hours; human involvement is limited to execution authorization, scene operations, and final acceptance assessment.Source: A. Experimental Setup and Protocol · p. 4

B. Results and Analysis

On drawer opening and door-handle operation, FORCA has lower grasp errors and succeeds more often than the vision-only agent. In cube-to-target adaptation, a program developed for one object and surface reaches the target band in a few additional iterations after physical conditions change.

1) Tasks Requiring Precise Manipulation

  • 100% success (10/10) on both tasks for FORCA, versus 4/10 on drawer opening and 1/10 on door-handle operation for the vision-only agent, alongside lower grasp errors.Table I reports translational and rotational grasp error as mean ± standard deviation over 10 trials.Source: Table I · p. 4, 1) Tasks Requiring Precise Manipulation · p. 4

2) Tasks Requiring Adaptation to New Physical Conditions

  • The adaptation task varies the object from a 61.9 g cube to a 663.6 g cylinder and varies the surface from a whiteboard to a woodboard, changing travel response while preserving the target objective.Object mass and contact geometry affect impact response, while surface friction affects deceleration and stopping distance.Source: 2) Tasks Requiring Adaptation to New… · pp. 4–5
  • In each of three changed object or surface configurations, the retained cube-on-whiteboard program reaches the target band within a few additional iterations, far fewer than developing from scratch.Figure 6 plots distance to the target band per attempt; Figure 5 shows illustrative landing positions and object/surface configurations.Source: 2) Tasks Requiring Adaptation to New… · p. 5, Figure 6 · p. 5, Figure 5 · p. 5

3) Tasks Requiring Online Contact Regulation

In vase wiping, force-responsive motion maintains more consistent contact when a hand-held vase moves, while a vision-only program tuned on a static vase loses contact and leaves more marks. In cucumber peeling, FORCA follows the uneven surface and produces a more continuous strip, whereas the vision-only program interrupts contact and presses excessively.

3) Tasks Requiring Online Contact Regulation

  • When transferred without revision from static to hand-held wiping, FORCA maintains more consistent contact and high marker removal as the vase moves, unlike the vision-only program’s contact loss and residual marks.Both programs were refined in the static setting before transfer; Figure 7 pairs representative outcomes and execution frames with force traces.Source: 3) Tasks Requiring Online Contact… · p. 5, Figure 7 · p. 6
  • FORCA follows the cucumber surface with force-responsive motion and produces a more continuous peeled strip, while the vision-only program has contact interruptions and excessive pressing that leave gaps.Figure 8 relates representative peeling outcomes to force traces.Source: 3) Tasks Requiring Online Contact… · pp. 5–6
  • The peeling evaluation defines excessive-force rate as the share of active peeling time above the upper contact-force threshold, capturing sustained over-pressing that peak force alone cannot characterize.The force-trace evaluation uses an upper threshold of 10 N; contact-loss intervals are also thresholded by force and duration.Source: 3) Tasks Requiring Online Contact… · p. 6

C. Program Development Cost

Agent reasoning takes most of each iteration, and wiping and peeling use the full iteration budget.

C. Program Development Cost

  • 4–10 min of agent reasoning versus 1–3 min of robot motion per iteration; wiping and peeling each use the full 20-iteration budget.Table IV reports iteration counts and mean time per iteration for reasoning and robot motion.Source: C. Program Development Cost · p. 6

D. Iterative Program Revision

The case study contrasts how FORCA and a vision-only agent revise programs after failure. FORCA uses force evidence to diagnose pressing failures, while the vision-only agent repeatedly loses its grasp during pulling.

D. Iterative Program Revision

  • The section presents one development run for each agent to illustrate iterative program revision using execution feedback.Table IV is titled as reporting development iterations and mean time per iteration, but the supplied passage gives no values.Source: D. Iterative Program Revision · p. 6, Table IV · p. 7
  • Fig. 8 reports 90.6% versus 28.9% of intended length covered in representative peeling passes for FORCA and the vision-only agent, respectively.The figure also shows normal contact force, with contact-loss intervals shaded and a 4.5–7.5 N target band.Source: Figure 8 · p. 6
  • FORCA mainly failed at pressing the handle down, then used a sharp rise in resistance to identify the end stop and address later pulling and releasing failures.After execution 16, its revision added an angle condition for accepting an end stop and a force-drop condition before pulling.Source: D. Iterative Program Revision · pp. 6–7
  • Fig. 9 marks each door-handle execution at its failure stage or at task completion; executions 18–20 repeat FORCA’s final program.The detailed failures and revisions are listed in Table V.Source: Figure 9 · p. 7
  • The vision-only agent mostly failed at grasping and pulling: although it could press the handle down, the handle repeatedly slipped from the gripper before the door opened.Its adjustments to grasp and press angle were based on images alone.Source: D. Iterative Program Revision · p. 7

V. LIMITATIONS

The limitations concern adaptation cost and setting, the information available from wrist force sensing, and the difficulty of more demanding tasks.

Iteration cost

  • Each refinement iteration takes several minutes, mostly because of agent reasoning, and the current process relies on real-robot rollouts.The authors identify reducing reasoning overhead and exploring simulation-based preliminary refinement as future work.Source: §V · p. 7

Contact sensing

  • Wrist wrench measurements capture combined contact effects but do not directly reveal contact location or force distribution across the fingers.This constrains the contact information available for diagnosis and finer manipulation.Source: §V · p. 7

Task complexity

  • For longer action sequences or tighter force constraints, the agent may struggle to diagnose failures and revise programs, even after repeated attempts.The authors suggest investigating whether improved reasoning and tool use could help with such tasks.Source: §V · p. 7

VI. CONCLUSION

FORCA uses force feedback to generate and revise robot programs, and real-robot experiments show deployment-time refinement across contact-rich tasks without demonstrations or task-specific model training. The authors frame this as a step toward executable, inspectable programs that accumulate physical experience, while noting that more reliable reasoning and more efficient revision remain necessary.

VI. CONCLUSION

  • FORCA’s real-robot experiments show it can refine geometric estimates through contact, adapt programs to changed physical conditions, and maintain contact during wiping and peeling without task demonstrations or task-specific model training.The paper describes this as learning at deployment time through force-feedback-based program generation and revision.Source: §VI · p. 7
  • The authors envision executable, inspectable programs accumulating useful contact routines and evidence about when they work, then revising them as conditions change.They identify more reliable reasoning from contact signals and a more efficient revision process as needed for complex tasks with fewer physical trials and manual resets.Source: §VI · p. 7

Related papers

Loading 2610.31841v1…