Source-linked AI summary
TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation
Zipei Ma, Xiaofei Wei, Junzhe Jiang, Shunlin Lu, Li Zhang
TL;DR
Contact-rich manipulation requires tactile information that vision-centric world-action models miss, while predicting tactile observations alone leaves a timing gap because plans are fixed before execution. TacPAC caches the predicted contact and planned action representation, then uses a tactile expert to correct unexecuted actions against that cache in real time. Across five real-robot tasks, it leads every task, reaches a 64% average success rate, and performs corrections 20.7× more cheaply than regenerating the chunk.
Problem
Vision-centric world-action models miss local contact cues, and predicting tactile observations alone recovers only about a third of the achievable tactile-sensing gain because predictions precede execution.
Method
TacPAC caches the tactile prediction and the planned chunk’s internal representation, then lets a tactile expert compare each new tactile image against that cache to correct remaining actions.
Results
TacPAC attains the highest success rate on all five tasks, averages 64%, and performs each correction in 30.4 ms, 20.7× cheaper than replanning.
Takeaways & Limitations
Tactile prediction and tactile correction are complementary: interpreting contact against the plan’s expectation supports real-time action correction across contact-rich tasks.
Abstract
from arXiv · showhide
World-action models guide action generation with predicted future observations, but vision-centric predictions miss the local contact cues that decide contact-rich manipulation. However, naively predicting future tactile observations as additional views recovers only a third of the achievable gain in our experiments. This gap reflects a timing mismatch: predictions precede execution, while tactile feedback arrives during it. We introduce TacPAC, which turns tactile prediction into real-time action correction. Once the base model has planned an action chunk, TacPAC caches the predicted contact that plan was conditioned on together with the plan's own representation, and a tactile expert reads each newly observed tactile image against that cache to correct the actions not yet executed. Feedback is thus interpreted against what the plan anticipated rather than in isolation, and one correction is a single pass over that cache, $20.7\times$ cheaper than regenerating the chunk. On five real-robot tasks spanning precision insertion, fragile-object handling, object reorientation, and long-horizon manipulation, TacPAC leads every task and raises the average from 22% for its vision-only base model to 64%. Code is available at https://github.com/LogosRoboticsGroup/TacPAC.
1 Introduction
Contact-rich manipulation needs tactile feedback because vision often misses the local deformation, slip, and insertion mismatch that determine success. TacPAC addresses the timing gap by comparing observed tactile feedback with the contact predicted for a planned action chunk, then correcting remaining actions in real time.
- Motivation: Vision provides global context but often misses local deformation, incipient slip, and small insertion mismatch in contact-rich tasks.These cues matter for precise charger-plug insertion and lifting fragile potato chips without breaking them.
- Motivation: Predicting tactile observations improves success but recovers only about a third of the gain available from tactile sensing.A future-view objective may not make action generation sensitive to temporally sparse contact transitions.
- Method: TacPAC caches the predicted contact and planned action representation, allowing a tactile expert to compare each new tactile image against the plan’s expectation.The corrector outputs per-step delta actions for steps that have not yet been executed.
- Evaluation: TacPAC evaluates contact-rich manipulation across five real-robot tasks covering insertion, transfer, fragile-object handling, and reorientation.The evaluation tests overall capability, the insufficiency of tactile prediction alone, component contributions, and real-time correction speed.
- Contributions: The paper’s contributions are identifying the limitation of fixed tactile prediction, introducing prediction-grounded correction, and training a single-pass corrector on the base model’s plans.The corrector is evaluated across five contact-rich manipulation tasks.
2 Related work
World-action models predict future observations to condition action generation, but their visual feedback remains insufficient for contact changes that vision cannot resolve. TacPAC combines tactile prediction with live tactile correction by retaining the predicted contact as a reference for interpreting incoming tactile images.
- World-action models: World-action models condition action generation on predicted observation trajectories, but their feedback is predominantly visual.This limits their ability to anticipate slip, incipient contact, and slight misalignment in contact-rich manipulation.
- Tactile manipulation and predictive models: Tactile sensing has largely been used as a present-time signal, leaving tactile prediction and action correction as separate abilities.The related systems described include tactile prediction without later correction and correction without a predicted tactile expectation.
- TacPAC: TacPAC retains the predicted tactile keys and values so each incoming tactile image can be interpreted against the contact the plan assumed.This makes prediction a live reference rather than a fixed output used only during initial planning.
- TacPAC: Blocking access to the cached prediction costs 17 points of average success under otherwise identical model and input conditions.The ablation isolates the contribution of reading tactile feedback against the predicted contact.
3 Method
TacPAC combines tactile-predictive planning with prediction-grounded, real-time correction of unexecuted actions. It caches the planned chunk and predicted contact, then uses a tactile expert to update the remaining actions asynchronously.
- Base model: The base model generates an action chunk conditioned on predicted future visual and tactile observations.A video expert and action expert interact through layer-wise attention, with tactile views serving both as predicted observations and conditioning context.
- Prediction-grounded correction: TacPAC corrects the planned chunk when observed contact departs from the contact anticipated during planning.The corrector compares each new tactile image with the cached prediction and the chunk’s internal representation rather than interpreting tactile feedback in isolation.
- Prediction-grounded correction: The cache stores predicted tactile views and action-token states from a clean pass over the planned chunk, while visual keys are omitted.The cache is built once per chunk and reused for every subsequent correction.
- Tactile expert: The tactile expert maps the current tactile image, partially executed plan, execution offset, and cache to a delta action added to the planned chunk.Executed actions are replaced with commands already sent, remaining actions retain planned values, and status embeddings distinguish executed from unexecuted steps.
- Inference: Cache reuse makes each correction a single tactile-expert pass without rerunning the video stream or denoising schedule.This supports asynchronous correction at the tactile-frame arrival rate, unlike refreshing predicted contact through a new full plan.
- Training: Training freezes the base model and trains the tactile expert on the base model’s own plans across multiple execution offsets.The expert is supervised on the residual between planned and ground-truth actions over the still-unexecuted suffix, with one shared plan and cache reused across sampled offsets.
- Inference: Inference applies corrections only from the latency-adjusted offset onward, leaving already executed actions and commands sent during computation unchanged.A tactile frame at offset m produces a correction after d steps, so only steps from m+d through H are revised.
4 Experiments
TacPAC is evaluated on five contact-rich real-robot tasks through matched baselines, component ablations, and inference-efficiency measurements. It achieves the highest success across all tasks while showing that tactile prediction, execution-time correction, and cached prediction access each contribute to performance.
- Evaluation setup: The benchmark covers charger-plug insertion, multi-object fruit transfer, fragile potato-chip transfer, empty-bottle uprighting, and expansion-card insertion.These tasks exercise tight alignment, regulated grip force, fragile-object handling, and multi-stage execution.
- Overall contact-rich manipulation: TacPAC achieves the highest success rate on all five tasks and averages 64%, 16 points above the strongest baseline, T-Rex at 48%.The evaluation uses 20 real-world trials per method per task and reports per-task and average episode success rates.
- Overall contact-rich manipulation: TacPAC combines tactile prediction with prediction-informed action correction and outperforms every prior evaluated method across all five tasks.The comparison includes general VLA, action-chunking, visuo-tactile, vision-only world-action, and tactile world-action baselines.
- Why tactile prediction alone is insufficient: Contact transitions occupy 4.9% of frames but account for 18.6% of total tactile change, and they occur after chunk execution begins.This timing makes tactile events difficult for a prediction objective that conditions a chunk before those events are observed.
- Why tactile prediction alone is insufficient: Predicting tactile observations raises average success from 22% to 37%, while the full system reaches 64%, leaving most of the gain unrealized without correction.The comparison is under matched data, backbone, action interface, optimization steps, and evaluation resets.
- Component ablation: Removing tactile-expert correction lowers average success to 37%, removing tactile prediction lowers it to 33%, and removing cached prediction access lowers it from 64% to 47%.These ablations indicate that tactile prediction, correction, and access to the cached anticipated contact are complementary pathways.
- Inference efficiency: A tactile-expert correction takes 30.4 ms at 32.90 Hz, reducing latency 20.7× versus regenerating an action chunk.Only 1.6% of calls advance one or more control steps during computation, and no correction-induced jerks are observed.
5 Conclusion
TacPAC turns tactile prediction into real-time action correction during execution, rather than leaving predicted contact fixed after planning. Across five real-world contact-rich tasks, it achieves the highest success rate on every task while improving efficiency over replanning.
- TacPAC reads each new tactile image against a layer-wise tactile cache to correct the planned action chunk asynchronously.The cache contains the predicted contact and the plan representation used during planning.
- TacPAC attains the highest success rate on all five tasks and averages 64%, 16 points above the strongest baseline, T-Rex.
- Each correction takes 30.4 ms, making it 20.7× cheaper than replanning.
- Tactile prediction and tactile correction are complementary, with explicit expectations providing a practical route to closed-loop contact-rich manipulation.
A Notation
The notation distinguishes flow time from physical execution time and marks clean quantities, noisy samples, model estimates, vectors, scalars, and indices consistently.
- Flow time τ denotes the conditional-flow noise level, whereas physical time t indexes execution stages along the action horizon.
- Undecorated symbols denote clean quantities, subscript τ marks noisy samples, and a hat marks a model estimate.
- Bold lower-case symbols denote vectors or vector-valued signals, while italic letters denote scalars and indices.
B Implementation details
The implementation shares attention geometry across video, action, and tactile experts so the tactile expert can consume cached base-model states. It generates 48-step action chunks, trains corrections from frozen plans, and evaluates on a multi-camera, tactile-equipped robot.
- Model configuration: Video, action, and tactile experts share 30 Transformer layers and identical 24-head attention geometry, enabling cache consumption by the tactile expert.Their hidden and feed-forward widths differ, with a linear bridge mapping video features to the tactile-expert width.
- Flow schedule and temporal layout: The model generates ten flow steps plus one clean cache-building pass, with H = 48 action steps issued at 30 Hz over 1.6 s.Each action contains seven relative joint commands and one gripper command.
- Stage-2 training: Stage 2 trains only the tactile expert using four correction offsets per frozen plan and a base learning rate of 10^-4.
- Optimization and computing infrastructure: Both training stages use one node with eight NVIDIA H100 GPUs, a global batch size of 64, and 10 epochs.
- Real-world experimental setup: The real-world setup uses a Flexiv Rizon 4 with wrist and third-person RealSense cameras and two InTac S1 tactile sensors.
- Notation: Table 3 groups notation by role and lists fixed configuration values, using “–” for context-dependent symbols.
C Evaluation details
Evaluation records all attempted trials under consistent task resets and counts every binary-success failure. The reported setup also adapts Dream-Tac’s gating hyperparameters to the tactile sensors and contextualizes temporal tactile activity across tasks.
- The tactile-change scale ct differs between tasks because the stored videos have different bitrates.
- Trial accounting and task reset: All 20 attempted trials per method–task pair are included, with no outcome-based exclusions or repeats.Failures include unsuccessful grasping, incomplete insertion or transfer, prohibited object damage, and incomplete task execution.
- Baseline adaptation: Dream-Tac uses its official hyperparameters with adjusted gating hyperparameters for adaptation to the tactile sensors.
D Task descriptions
The section describes five contact-rich manipulation tasks, each requiring precise interaction and task-specific success conditions. The tasks span insertion, transfer, fragile-object handling, and reorientation.
- Charger plug insertion requires grasping an adapter, aligning its prongs with a designated outlet, and seating it in the socket.
- Multi-object fruit transfer requires sequentially moving three fruits into a tray without visible damage.
- Fragile potato-chip transfer requires carrying an intact chip between plates without crushing or visibly damaging it.
- Empty-bottle uprighting requires rotating a compliant bottle upright, placing its base on the table, and releasing it stably.
- Expansion-card insertion requires aligning an edge connector with a motherboard slot and pressing the card until seated.
E Diagnostic for Q2
The diagnostic measures when tactile observations change across execution phases and finds the strongest activity around contact transitions. Its protocol normalizes phase activity within episodes and uses independent gripper-width annotations for phase boundaries.
- The diagnostic covers 296 episodes and retains 284,948 frames after exclusions.
- 4.1 ± 0.1 times the episode mean is the tactile activity during contact-transition windows.This exceeds sustained contact at 1.36 ± 0.04 and free-space approach at 0.58 ± 0.03.
- Contact-transition activity exceeds sustained-contact activity in 99% of episodes and approach activity in 94%.
- Phase contrasts are normalized within each episode because bitrate variation makes absolute tactile activity incomparable across episodes.The analysis divides each phase mean by that episode’s own mean activity.
- Execution phases are segmented from gripper width, with contact-transition windows defined as eight frames on either side of each hold boundary.The annotation deliberately labels some early or environmental contacts as approach or unmarked contact.