Source-linked AI summary
TacForcing: Streaming Action Generation with Execution-Time Tactile Feedback
Jianbo Zhou, Boyuan Zhao, Yuzheng Zhang, Yiyang Chen, Wenxin Chen, Qiuyue Li, Xiangyang Gu, Yuhan Cao, Xiao Xia, Yanzhe Hu, Zhijie Deng
TL;DR
Contact-rich manipulation requires adapting to rapidly evolving contact states, but chunk-based policies can leave tactile conditioning stale during execution and reactive alternatives add complexity. TacForcing uses a Streaming Action Expert with Execution-Aware Tactile Attention to incorporate execution-time tactile feedback, achieving higher success across six simulated and three real-world tasks.
Problem
Fixed tactile conditioning can become stale within an action chunk, while separate high-frequency reactive pathways increase architectural and training complexity.
Method
TacForcing progressively generates action blocks during execution and uses Execution-Aware Tactile Attention to condition only the next scheduled block on each tactile update.
Results
65% and 69% average success rates are achieved across six simulated UniVTAC tasks and three real-world tasks, respectively, outperforming strong baselines in both settings.
Takeaways & Limitations
Aligning tactile conditioning with block execution supports contact-rich manipulation without a separate reactive controller.
Abstract
from arXiv · showhide
Contact-rich manipulation requires adapting to contact states that can evolve substantially within an action horizon. However, chunk-based vision-language-action models predict complete action chunks from observations collected before execution, leaving tactile conditioning stale during execution. Existing tactile-reactive approaches typically rely on separate high-frequency controllers, which increase both architectural and training complexity. In this paper, we introduce TacForcing, a streaming action-generation framework that effectively incorporates execution-time tactile feedback. Instead of employing a separate reactive controller, TacForcing replaces the standard action expert with a streaming action expert to generate actions conditioned on the evolving tactile observations acquired during execution. TacForcing also introduces Execution-Aware Tactile Attention (EATA), which restricts tactile conditioning to actions nearing execution, thereby reducing the temporal mismatch between tactile acquisition and action execution. Across six simulated UniVTAC tasks and three real-world contact-rich manipulation tasks, TacForcing achieves average success rates of 65% and 69%, respectively, outperforming strong baselines in both settings.
1 Introduction
Contact-rich manipulation needs continuous adaptation to contact states that vision alone may not reveal, while fixed tactile conditioning can become stale within an action chunk. TacForcing addresses this mismatch with streaming generation and execution-aware tactile conditioning, and improves success across simulated and real-world tasks.
- Visual observations alone cannot reliably reveal force changes, slip events, or other latent contact states, especially under occlusion.
- Tactile representations can change substantially within one action chunk despite nearly unchanged visual representations, making fixed tactile conditioning increasingly misaligned.
- TacForcing progressively generates sequential action blocks during execution, retaining unfinished-block states for refinement with newly acquired tactile feedback.
- 65% and 69% average success rates are achieved in simulation and real-world tasks, respectively, outperforming strong baselines in both settings.
- The framework incorporates execution-time tactile feedback without a separate reactive controller.
- Execution-Aware Tactile Attention conditions each tactile update only on the next scheduled action block, reducing temporal mismatch.
2 Preliminaries and Motivation
Flow Matching generates action chunks by transporting Gaussian noise to a conditional data distribution, but synchronous chunk generation prevents execution-time tactile feedback from updating later actions. A dropper-squeezing analysis shows tactile representations can change substantially within a 40-action horizon despite limited visual variation.
- Flow Matching: Flow Matching learns a conditional velocity field that transports Gaussian-prior samples to the data distribution.During inference, integrating the learned field from τ = 0 to τ = 1 produces a sample from the conditional distribution.
- Policy formulation: The policy conditions an H-step action chunk on visual observations, proprioception, language, and tactile input.The chunk is At = (at, at+1, …, at+H−1), with each action in R^da.
- Synchronous generation: Because all action positions share one flow time, the conventional expert generates the entire chunk synchronously from fixed initial inputs.The action chunk and same-shaped Gaussian noise serve as the data and prior endpoints, respectively.
- Motivation: Consequently, tactile feedback acquired during execution cannot condition the remaining actions within the same chunk.This creates a mismatch between the tactile input used for generation and later contact states.
- Motivation: 0.55 tactile cosine distance versus 0.005 visual cosine distance was measured after 35 control steps within the 40-action horizon.The final sample occurred 1.17 s after the initial observation, indicating substantial tactile change despite limited visual variation.
3 Method
TacForcing generates action chunks progressively in blocks so execution-time tactile feedback can refine actions that remain to be executed. EATA limits each tactile update to the next action block, reducing temporal mismatch while preserving computational efficiency.
- Streaming Action Generation: TacForcing replaces synchronous chunk generation with a Streaming Action Expert that progressively generates and dispatches action blocks during execution.Intermediate states of unfinished blocks are retained and refined using newly acquired tactile feedback.
- Block-wise Flow Scheduling: Position-dependent flow times make blocks complete sequentially, while completed blocks remain fixed and unfinished blocks continue evolving.With H = KB and N = KS, the chunk is divided into K blocks of B actions, separated by S sampling steps.
- Block-wise Tactile Updates: Block-level tactile updates balance responsiveness with computational efficiency by avoiding tactile acquisition, encoding, and inference at every control step.The task context is encoded once per chunk, whereas tactile observations are refreshed after each executed block.
- Execution-Aware Tactile Attention: EATA restricts each current tactile representation to the action block scheduled to execute next, preventing later blocks from using potentially outdated feedback.Later blocks receive updated tactile conditioning when they become the next block to execute, and the same visibility constraint is applied during training.
- Training: Training pairs block-wise intermediate states with tactile feedback from the corresponding execution stage and applies loss only to unfinished actions.This matches the positions updated during streaming inference.
4 Experiments
TacForcing is evaluated across six UniVTAC simulation tasks and three real-world contact-rich tasks against non-tactile, tactile-conditioned, and tactile-reactive baselines. It achieves strong average success rates in both settings, while ablations show benefits from execution-time tactile updates and EATA.
- Benchmarks: TacForcing is evaluated on six UniVTAC simulation tasks and three real-world tasks: Stand Bottle, Transfer Liquid, and Wipe Board.
- Baselines: The baselines cover vision-only, tactile-conditioned, and tactile-reactive policy designs.
- Simulation Results: 65% average success in simulation outperforms π0.5 by 14 percentage points and RDP by 23 percentage points.TacForcing achieves the highest or tied-highest success rate on five of six tasks; Lift Can is the exception, at 63% versus 66% for the best method.
- Real-World Results: 69% average success in the real world outperforms FTP-1, GR00T N1.7, and π0.5 by 17, 27, and 42 percentage points, respectively.TacForcing leads on Stand Bottle and Transfer Liquid and ties FTP-1 on Wipe Board; on Transfer Liquid it reaches 50% while every baseline reaches at most 19%.
- Ablation Study: Execution-time tactile updates raise average success from 42% to 51% in simulation and from 31% to 48% in real-world experiments without EATA.The comparison is between Fixed Tactile and TacForcing without EATA.
- Ablation Study: Adding EATA raises average success from 51% to 60% in simulation and from 48% to 69% in real-world experiments.Compared with Fixed Tactile, TacForcing gains 18 percentage points in simulation and 38 percentage points in the real-world experiments.
5 Related Work
Prior work integrates tactile perception, streaming generation, or separate fast tactile control into manipulation policies. TacForcing instead focuses on how successive tactile observations condition a partially generated action horizon within one streaming generator.
- Streaming Action Generation: Diffusion and flow-based action models commonly generate complete trajectories or receding-horizon chunks before execution begins.
- Streaming Action Generation: Position-dependent schedules and streaming policies allow sequence elements or rolling action buffers to progress under changing observations.
- Tactile-Aware Policies: Tactile methods improve contact information, representation transfer, or multimodal policy integration, while TacForcing targets the temporal use of feedback during action generation.
- Execution-Time Adaptation: Execution-time adaptation often separates slow visuomotor reasoning from fast tactile control, whereas TacForcing retains unfinished action states in one streaming generator and aligns updates with block execution.
6 Conclusion
TacForcing streams action-block generation with execution-time tactile feedback without a separate reactive controller, while EATA aligns each tactile update with the next executable block. Across simulated and real-world tasks, it outperforms strong baselines, and ablations support both design elements.
- TacForcing progressively generates action blocks, retains unfinished-block states, and refines them using newly acquired tactile feedback.
- EATA restricts each tactile update to the next scheduled action block, reducing the temporal mismatch between tactile acquisition and execution.
- 65% and 69% average success rates are achieved across six simulated and three real-world contact-rich tasks, respectively, outperforming strong baselines in both settings.
- Ablations show that execution-time tactile updates and EATA both contribute to performance improvements.
A.1 Simulation Training Parameters
Simulation training uses 50 demonstration trajectories per task for 15,000 steps, with an action horizon divided into ten five-action blocks.
- Simulation training uses 50 demonstration trajectories per task for 15,000 steps with a global batch size of 256.
- The action horizon is H = 50, divided into K = 10 blocks of B = 5 actions.
A.2 Real-World Training Parameters
The real-world training configuration uses 100 demonstrations per task and trains for 30,000 steps with specified optimization settings and an action horizon divided into eight blocks.
- 100 demonstration trajectories per task are used for training over 30,000 steps.
- The optimizer is AdamW with a peak learning rate of 6 × 10−5, cosine decay to 1 × 10−6, and 2,000 warm-up steps.
- The configuration uses 10−5 weight decay and gradient clipping at a global norm of 1.0.
- The action horizon is H = 40, divided into K = 8 blocks of B = 5 actions.
A.3 Representation-Dynamics Analysis
The representation-dynamics analysis examines a dropper-squeezing episode over a 40-action horizon, sampling visual and tactile encodings across time and comparing their changes relative to initial representations.
- A representative dropper-squeezing episode is analyzed over a 40-action horizon with observations sampled every five actions at 30 FPS.
- Visual representations come from top and wrist cameras, while tactile representations come from thumb and index-finger sensors.
- For each sensor and offset k ∈ {0, 5, . . . , 35}, the analysis computes cosine distance between the current representation and its initial representation.
- Modality-level distances average the distances across the two corresponding visual or tactile sensors.
- The real-world platform supplies the analysis with tactile fingertip deformation maps, global and local RealSense views, and two robot arms with dexterous hands.
B.2 Real-World Task Settings
The real-world evaluation covers three contact-rich tasks—Stand Bottle, Transfer Liquid, and Wipe Board—with 16 independent trials per task and distinct grasp, contact, and manipulation requirements.
- Each real-world method is evaluated over 16 independent trials per task.
- Stand Bottle: Stand Bottle requires grasping a horizontal bottle, reorienting it, and placing it upright while maintaining a stable grasp.
- The platform includes two RealMan RM75 arms, Sharpa Wave dexterous hands, Manus Pro data gloves, and top- and wrist-mounted RealSense cameras.
- Transfer Liquid: Transfer Liquid requires drawing liquid from a flask with a transparent dropper and dispensing it into a beaker despite partial visual occlusion.
- Wipe Board: Wipe Board requires moving an eraser across a marked whiteboard while maintaining sufficient surface contact to remove the marks.
C Simulation Task Details
The simulation evaluation uses six UniVTAC contact-rich manipulation tasks, spanning constrained grasping, extraction, placement, and precision insertion scenarios.
- The six simulated tasks are Lift Bottle, Pull-out Key, Lift Can, Put Bottle in Shelf, Insert Hole, and Insert Tube.
- Lift Bottle requires grasping and lifting a bottle near a wall, while Lift Can requires lifting a cylindrical can without dropping it.
- Pull-out Key requires extracting a key from a lock, and Put Bottle in Shelf requires placement into a shelf with limited clearance.
- Insert Hole requires aligning and inserting a peg into a narrow hole, while Insert Tube requires alignment with a mating fixture and adaptation to contact from a constrained opening.
- Figure 6 presents representative execution sequences for the six UniVTAC simulation tasks.