PaperlayerView exact revision on arXiv

Source-linked AI summary

TAO-Force: Unifying Force-Aware Perception and Fast-Slow Control for Contact-Rich Manipulation

Bohan Gan, Xuanzhang Wen, Yongsheng Zhao, Baoping Cheng, Wenhe Jia, Ye Wang, Gongxin Yao, Han Gao, Jingyao Tang, Lei Zhao, Ji Ge

arXiv:2609.18497v1cs.RO

TL;DR

Contact-rich VLA policies may underuse force information because pretrained visual representations dominate posttraining, while joint motion–force strategies often omit contact-state estimation and phase-transition coordination. TAO-Force uses F-FiLM to incorporate force feedback into a frozen pretrained VLA and contact-gated fast–slow control to regulate interaction. 80.0% ID and 64.2% OOD average success rates across four contact-rich tasks, outperforming evaluated general-purpose VLA baselines on average.

  • Problem

    Contact-rich VLA policies may underuse force information because pretrained visual representations dominate posttraining, while joint motion–force strategies often omit contact-state estimation and phase-transition coordination.

    Source: §I · p. 2

  • Method

    TAO-Force uses F-FiLM to incorporate force feedback into a frozen pretrained VLA and contact-gated fast–slow control to regulate interaction.

    Source: §I · p. 2

  • Results

    80.0% ID and 64.2% OOD average success rates across four contact-rich tasks, outperforming evaluated general-purpose VLA baselines on average.

    Source: §I · p. 2

  • Takeaways & Limitations

    Real-world experiments and ablations support combining force-aware perception with force control for robust contact-rich manipulation.

    Source: E. Ablation Study · p. 8, §VI · p. 8

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have demonstrated strong performance across diverse robotic manipulation tasks, yet their predominantly vision-centric perception and position-controlled execution remain insufficient for contact-rich manipulation. Visual observations alone often provide limited evidence of contact onset and interaction magnitude, while position-control policies cannot respond compliantly to rapidly changing contact dynamics. To bridge both the perception and control gaps, we propose TAO-Force, a force-conditioned VLA framework that combines force-aware policy learning with contact-regulated execution. For force-aware perception, TAO-Force introduces Force-conditioned Feature-wise Linear Modulation (F-FiLM) to inject encoded force feedback into the representations of a frozen pretrained visual-language backbone while preserving its semantic priors. For responsive control, it employs a contact-gated fast-slow architecture, with a slow position-control branch tracking nominal trajectories during non-contact phases and a fast admittance-control branch regulating physical interaction during contact phases. Detailed analyses on a force-perception task and real-world evaluations across four contact-rich manipulation tasks validate the effectiveness and robustness of TAO-Force.

I. INTRODUCTION

TAO-Force addresses perception and control gaps in contact-rich manipulation by adding force feedback to a pretrained VLA and coordinating position and admittance control. Across four contact-rich tasks, it outperforms evaluated general-purpose VLA baselines on average.

I. INTRODUCTION

  • Visual observations can make contact onset, interaction magnitude, and required compliance difficult to infer in tasks such as peeling, pressing, and wiping.The paper argues that contact-rich state transitions are governed by physical interaction, which may correspond to only subtle visual changes.Source: §I · p. 1
  • 80.0% ID and 64.2% OOD average success rates across four contact-rich tasks, outperforming evaluated general-purpose VLA baselines on average.The reported values are average success rates under in-domain and out-of-domain conditions, respectively.Source: Figure 1 · p. 1, §I · p. 2
  • Two challenges remain: pretrained visual features can dominate force learning, and joint motion–force targets do not explicitly estimate contact state or coordinate phase transitions.The latter makes balancing motion accuracy and contact compliance difficult.Source: §I · p. 2
  • F-FiLM uses encoded force histories to modulate a frozen pretrained backbone’s visual–language representations while preserving semantic priors and supporting trajectory and contact-force prediction.The framework also uses contact-causal force supervision, masking force-chunk targets at pre-contact anchors.Source: §I · p. 2
  • A predicted contact probability gates slow nominal position control in non-contact phases and faster admittance control during contact.Trajectory densification and smooth fusion coordinate the branches and reduce discontinuities during transitions.Source: §I · p. 2

A. General-Purpose VLA Models

The section traces VLA development and contact-rich sensing and control approaches, then frames the perception and control gaps TAO-Force addresses. It introduces admittance control as a way to turn force-tracking error into compliant motion.

A. General-Purpose VLA Models

  • VLA models evolved from vision–language-conditioned manipulation toward pretrained backbones, multi-embodiment training, and diffusion- or flow-based continuous action generation.Hierarchical systems also connect semantic reasoning with fine-grained motor execution through intermediate representations and continuous controllers.Source: A. General-Purpose VLA Models · p. 2

B. Contact-Rich Manipulation Models

  • Contact-rich models fuse force or tactile sensing into perception, while other approaches combine motion and force targets or add fast tactile- or force-conditioned action refinement.The section identifies preserving pretrained semantics while converting low-rate predictions into stable high-rate regulation as an ongoing design challenge.Source: B. Contact-Rich Manipulation Models · pp. 2–3

III. PRELIMINARIES AND PROBLEM FORMULATION

  • The preliminaries frame contact-rich execution as requiring both awareness of physical interaction and responsive regulation, and introduce force-aware adaptation of a pretrained VLA policy.The policy formulation adds external-force context to the standard visual, language, and robot-state inputs.Source: §III · p. 3, B. Problem Formulation · p. 3

A. Admittance Control

  • Admittance control maps force-tracking errors to Cartesian motion corrections for compliant interaction on position-controlled robots.The model uses measured external and desired forces, with the measured force compensated for the tool’s gravitational wrench and expressed in the controller frame.Source: A. Admittance Control · p. 3

B. Problem Formulation

  • The perception gap is missing direct observations of robot–environment interaction, while the control gap is the mismatch between low-frequency action prediction and fast contact dynamics.The policy formulation predicts action and desired-force chunks alongside contact probability.Source: B. Problem Formulation · p. 3

IV. THE TAO-FORCE FRAMEWORK

TAO-Force combines a force-conditioned policy with contact-gated fast–slow control. It builds on a frozen pretrained VLA backbone and adds lightweight components for force-aware adaptation.

IV. THE TAO-FORCE FRAMEWORK

  • TAO-Force jointly predicts nominal actions and desired forces, separately estimates contact probability, and uses that probability to gate fast–slow execution.A slow branch tracks nominal motion while a fast controller regulates interaction during contact.Source: §IV · p. 3

A. Pretrained VLA Architecture

  • The framework builds on GR00T N1.5, freezing its Eagle visual–language backbone and adding lightweight force encoders, F-FiLM modulation, and prediction heads.The backbone combines a SigLIP visual encoder and Qwen-based language model to produce multimodal representations for a diffusion transformer action head.Source: A. Pretrained VLA Architecture · p. 3

B. Force-conditioned Policy Learning

TAO-Force conditions a frozen visual-language representation on force feedback, then predicts action and desired-force chunks alongside contact probability. These outputs support contact-gated execution.

B. Force-conditioned Policy Learning

  • Force feedback, current force, recent force history, and robot state condition the pretrained visual-language features through learned feature-wise affine modulation.The force history is encoded with a lightweight 1D-CNN; the visual-language backbone remains frozen while new force encoders and modulation networks are optimized.Source: B. Force-conditioned Policy Learning · p. 3
  • F-FiLM’s zero-initialized modulation initially preserves the pretrained representation, then learns to incorporate physical-interaction feedback.The modulation starts with Δγt = 0 and βt = 0.Source: B. Force-conditioned Policy Learning · p. 4
  • A shared DiT decoder jointly generates temporally aligned action and desired-force chunks, while a separate lightweight head predicts instantaneous contact probability.At inference, the chunk outputs are recovered by integrating learned velocity fields from initial noise.Source: B. Force-conditioned Policy Learning · p. 4
  • The policy’s nominal action, desired-force chunk, and contact probability provide the outputs for contact-gated control.The contact head responds to the current force-conditioned perceptual state while the decoder generates temporally coherent chunks.Source: B. Force-conditioned Policy Learning · p. 4

C. Training Objective

TAO-Force trains action and force generation together with contact classification. Its force-targeting and modality-dropout strategies aim to ground predictions in physical interaction and encourage use of force feedback.

C. Training Objective

  • The composite objective combines action and force flow matching with contact classification.The action loss preserves nominal action generation; the force loss uses contact-causal targets, and the contact loss uses binary cross-entropy.Source: C. Training Objective · pp. 4–5
  • Future force targets containing contact can teach pre-contact predictions to rely on visual or timing correlations, risking mistimed force predictions when contact onset varies.The described deployment risks are predicting nonzero force before contact or delaying force prediction after contact.Source: C. Training Objective · pp. 4–5
  • Contact-causal supervision gates force targets by current contact state, using zero targets before contact and recorded force chunks during contact.The force loss is evaluated over all frames.Source: C. Training Objective · p. 5
  • Force-attentive modality dropout partially masks visual and proprioceptive features in annotated force-critical segments to encourage use of force-conditioned cues.The method targets shortcut reliance on non-force modalities when visually similar observations correspond to different interaction states.Source: C. Training Objective · p. 5

D. Contact-Gated Fast–Slow Control

A predicted contact probability selects between nominal position control and high-frequency admittance control. Reference processing and a smooth transition mechanism coordinate policy chunks with responsive execution.

D. Contact-Gated Fast–Slow Control

  • A thresholded contact probability selects nominal position control in free space or high-frequency admittance control during contact.The controller bridges chunk-level policy predictions and high-frequency robot commands.Source: D. Contact-Gated Fast–Slow Control · p. 5
  • Action chunks are densified and joined with minimum-jerk transitions, while force references are smoothed across chunk boundaries to reduce discontinuities.The force processor aligns each action target index to the force timeline and blends the current reference with the new prediction.Source: D. Contact-Gated Fast–Slow Control · p. 5
  • During contact, admittance control regulates measured external force toward the desired force while using the nominal joint reference as a weak positional anchor.Small stiffness limits accumulated positional drift without dominating force regulation; Cartesian velocity is mapped to joint space and integrated at high frequency.Source: D. Contact-Gated Fast–Slow Control · p. 5
  • When contact ends, a quintic smoothstep gradually removes the force-control command residual to reconnect motion to the nominal trajectory.If contact recurs during the transition, trajectory fusion is canceled and force control resumes immediately.Source: D. Contact-Gated Fast–Slow Control · pp. 5–6

V. EXPERIMENTS

The evaluation is designed to assess TAO-Force’s perception and control contributions through offline analyses and real-world deployments. It asks whether force changes trajectories, whether the method outperforms VLA baselines across conditions, and how its design choices affect success.

V. EXPERIMENTS

  • The evaluation combines offline analyses and real-world robot deployments to assess the perception and control gaps.It is organized around three research questions.Source: §V · p. 6
  • The first question tests whether force feedback meaningfully influences trajectory generation.Source: §V · p. 6
  • The second question asks whether TAO-Force outperforms VLA baselines in task success and robustness under in-distribution and out-of-distribution conditions.Source: §V · p. 6
  • The third question examines how each key design contributes to task success.Source: §V · p. 6

A. Experimental Setups

The experiments evaluate force-aware perception and contact-rich execution on a real-world dual-arm humanoid, using visual and wrist force/torque observations. The setup spans five tasks and compares TAO-Force with two VLA baselines where applicable.

Hardware Platform

  • Experiments use an AgiBot G1 dual-arm humanoid with three RGB cameras and wrist-mounted six-axis force/torque sensors.The VLA runs at 5 Hz; the admittance controller generates compliant commands at 1000 Hz.Source: A. Experimental Setups · p. 6

Experimental Tasks

  • Five real-world manipulation tasks are evaluated with 30 trials per task in each of the in-domain and out-of-domain settings.The tasks cover force perception, contact-rich execution, and OOD robustness.Source: A. Experimental Setups · p. 6

Baselines

  • TAO-Force is compared with GR00T N1.7 and π0.5, but Blind Box Sorting is excluded because those baselines lack force input.Sorting visually identical cans by weight inherently requires force perception.Source: A. Experimental Setups · p. 6

B. Analysis of Force-Aware Trajectory Generation

The force-perception analysis tests whether force feedback changes generated trajectories when vision is held fixed. F-FiLM responds consistently to counterfactual force swaps, unlike the evaluated alternative fusion methods.

Analysis Goal

Evaluation Protocol

  • The counterfactual analysis fixes visual input at the routing-divergence anchor and swaps force signatures between empty and full cans.The evaluation measures the final 16 frames of the Joint 2 action chunk using trajectory error and flip rate.Source: B. Analysis of Force-Aware Trajectory… · pp. 6–7

Results and Analysis

  • F-FiLM redirects generated trajectories toward the routing associated with injected counterfactual force and responds consistently in both swapping directions.DePost responds asymmetrically, while FVLMoE shows limited sensitivity; F-FiLM has the most consistent force-aware adaptation among the evaluated methods.Source: B. Analysis of Force-Aware Trajectory… · p. 7

C. Analysis of Contact-Force Prediction

Contact-force prediction is assessed during cucumber peeling, and the real-world task evaluation reports robust performance across contact-related perturbations. TAO-Force leads the VLA baselines in overall OOD performance, while tool changes remain challenging.

C. Analysis of Contact-Force Prediction

  • The force-prediction analysis samples anchors within intervals where predicted contact probability exceeds 0.8 and evaluates the first frame of each predicted chunk.MSE is reported separately for translational forces Fx, Fy, Fz and rotational torques Tx, Ty, Tz.Source: C. Analysis of Contact-Force Prediction · p. 7
  • Predicted force and torque signals closely follow rapid ground-truth variations during peeling contact, and F-FiLM improves prediction accuracy across force and torque components.Table III reports contact-phase force and torque prediction errors.Source: C. Analysis of Contact-Force Prediction · p. 7, Table III · p. 7

D. Real-World Task Performance

  • TAO-Force achieves 100.0% success on Blind Box Sorting in both ID and OOD settings; baselines without force input are inapplicable to this task.The reported rates use 30 trials per setting.Source: D. Real-World Task Performance · p. 7
  • Tool changes in wiping and cutting shift contact timing and force characteristics, contributing to insufficient pressure, unstable contact, or ineffective cutting and lower OOD success.By contrast, TAO-Force achieves the highest OOD success rates for cucumber peeling and pump pressing, whose OOD perturbations alter object height and contact geometry.Source: D. Real-World Task Performance · p. 7
  • TAO-Force achieves the strongest OOD performance and a smaller ID-to-OOD success-rate drop than GR00T N1.7 and π0.5 across evaluated tasks.The results characterize improved robustness to contact geometry, tool properties, and environmental variation.Source: D. Real-World Task Performance · p. 7

E. Ablation Study

The ablation study finds complementary roles for force-conditioned trajectory generation and closed-loop force control. Their combination yields the strongest success rates, a result consistent with the paper’s conclusion about robust contact-rich manipulation.

E. Ablation Study

  • F-FiLM modulates multimodal representations during trajectory generation, while force control improves execution-time adaptability through closed-loop regulation.Regulating desired contact force is more tolerant of contact-timing and local-interaction variation than directly tracking predicted actions.Source: E. Ablation Study · pp. 7–8
  • Integrating F-FiLM with force control yields the highest success rate in every evaluated setting.The paper concludes that the components contribute complementarily to robust contact-rich manipulation.Source: E. Ablation Study · p. 8, §VI · p. 8

VI. DISCUSSION AND CONCLUSION

  • The conclusion describes a fast–slow architecture in which the slow system generates action and desired-force trajectories and the fast system regulates force during execution.TAO-Force combines this control with F-FiLM force-aware trajectory generation.Source: §VI · p. 8

Related papers

Loading 2609.18497v1…