PaperlayerView exact revision on arXiv

Source-linked AI summary

RATE: Risk-Aware Tactile Encoding for Contact-rich Robotic Manipulation

Yuyao Jiang, Haichao Liu, Jiarui Zheng, Zihan Ding, Weihao Yuan, Ziwei Wang

arXiv:2610.03538v1cs.RO

TL;DR

Similar tactile observations can reflect interaction conditions with different task-risk implications, leaving a gap in context-grounded risk representation. RATE learns risk-aware tactile features using history-conditioned prediction and alert supervision, then complements conventional features through a residual pathway. Across simulated and real-world evaluations, RATE consistently improves manipulation performance, including a 71.8% macro-average success rate across eight simulated tasks.

  • Problem

    Similar tactile observations can reflect sensor noise, necessary adjustments, or emerging undesirable contact despite having different task-risk implications.

    Source: §I · p. 1

  • Method

    RATE uses history-conditioned prediction and alert supervision to learn a complementary risk-aware representation, integrated with conventional tactile features through a lightweight residual pathway.

    Source: §I · pp. 1–2

  • Results

    Across eight simulated and three real-world tasks, RATE consistently improves manipulation performance; it achieves 71.8% macro-average success across the eight UniVTAC tasks.

    Source: B. Simulation Results on the UniVTAC… · p. 6, §V · p. 8

  • Takeaways & Limitations

    Ablations support complementary effects of alert-guided learning and context, with particularly strong gains on contact-rich interactions.

    Source: §V · p. 8

    Subtle task-relevant tactile variations may become less distinguishable when contact magnitudes vary substantially across interaction stages.

    Source: §V · p. 8

Abstract

from arXiv · show

Tactile sensing is particularly valuable for contact-rich robotic manipulation. Recent work has made substantial progress in tactile representation learning for robotic manipulation. However, similar tactile observations can arise from interaction conditions with very different task-risk implications, such as sensor noise, task-necessary variations, and emerging undesirable contact. Without context-grounded risk information, these cases can be ambiguous to downstream policies, leading to unnecessary corrections to benign variations or delayed responses to genuinely risky contact. To address this limitation, we propose Risk-Aware Tactile Encoding (RATE), which learns tactile representations that encode task-conditioned interaction risk. Specifically, history-conditioned prediction captures interaction context, while alert supervision associates this context with task-conditioned risk. The learned risk-aware representation complements conventional tactile features through a lightweight residual adapter. Experiments in both simulation and the real world demonstrate substantial improvements in task success, with controlled ablations confirming the complementary benefits of alert-guided learning and predictive temporal modeling.

I. INTRODUCTION

Tactile feedback supports contact-rich manipulation, but similar observations can reflect noise, necessary adjustments, or risky contact despite differing task-risk implications. RATE learns a complementary risk-aware representation and evaluates it across simulated and real-robot tasks.

I. INTRODUCTION

  • Sensor noise, task-necessary adjustments, and emerging undesirable contact can produce similar tactile observations despite differing task-risk implications.Without context-grounded risk information, policies may make unnecessary corrections or respond too late.Source: §I · p. 1
  • RATE achieves the best or jointly best success rates on six of eight simulation tasks and the highest rates on all three real-robot tasks.The real-robot tasks are USB insertion, bottle-cap screwing, and plug insertion.Source: Figure 1 · p. 1
  • RATE uses history-conditioned prediction to capture interaction context and alert supervision to associate that context with task-conditioned risk.The learned representation complements conventional tactile features through a zero-initialized residual pathway, with alert-weighted action imitation emphasizing task-critical steps.Source: §I · pp. 1–2
  • The residual pathway updates only 0.59% of model parameters, while pretrained tactile encoders and the policy backbone remain frozen.Alert targets and future observations are used only during training.Source: §I · pp. 1–2
  • Simulation and real-world experiments show gains, while ablations find alert-guided learning and context individually effective and complementary.Source: §I · p. 2

A. Visuo-Tactile Representation Learning for Contact-Rich Manipulation

Prior work advances visuo-tactile representation learning, temporal tactile modeling, and contact-aware manipulation. These approaches provide useful contact and temporal information, while the section identifies task-conditioned risk representation as a distinct gap.

A. Visuo-Tactile Representation Learning for Contact-Rich Manipulation

  • Visuo-tactile methods use cross-modal learning, shared spatial structure, and large-scale simulation or demonstrations to represent contact appearance, geometry, and multimodal structure.Their objectives are not explicitly designed to organize tactile information by task-conditioned risk in an evolving interaction.Source: A. Visuo-Tactile Representation… · p. 2

B. Temporal Tactile Modeling and Contact-Aware Manipulation

  • Temporal tactile methods use sequences or future-prediction objectives to capture contact evolution and model interaction dynamics.Tactile and force feedback also supports reactive correction and predictive action guidance, with temporal information primarily used for dynamics prediction or action correction.Source: B. Temporal Tactile Modeling and… · p. 2
  • Contact-aware approaches use tactile or contact feedback for grasp outcomes, collision detection, force regulation, and trajectory refinement.These signals typically shape event prediction, control constraints, or policy optimization rather than the tactile representation itself.Source: B. Temporal Tactile Modeling and… · p. 2

III. METHODOLOGY

  • The methodology section formulates the problem, describes risk-aware representation learning, and details residual integration into a pretrained manipulation policy.Source: §III · p. 2

A. Problem Formulation and Framework Overview

RATE formulates tactile risk representation as a complement to existing features, using recent tactile interaction and only observations available up to the current time at deployment. It learns the representation from tactile history and integrates it through a residual pathway into a policy conditioned on tactile, visual, and robot-state inputs.

A. Problem Formulation and Framework Overview

  • RATE learns a complementary risk-aware representation from tactile observations available up to time t, augmenting rather than replacing the existing representation.Source: A. Problem Formulation and Framework… · p. 2
  • Short-horizon future contact-change prediction and continuous task-conditioned alert supervision jointly shape the representation.Alert targets are derived from task-relevant interaction signals, normalized to [0, 1], and higher values correspond to increasingly undesirable contact.Source: A. Problem Formulation and Framework… · p. 2
  • A zero-initialized residual pathway fuses the learned representation with conventional tactile features, and the resulting representation conditions the policy alongside camera observations and robot state.At deployment, the policy uses visual, tactile, and proprioceptive observations available up to the current time step.Source: A. Problem Formulation and Framework… · pp. 2–3, Figure 2 · p. 3

B. Learning Risk-Aware Tactile Representations

RATE builds interaction context from bilateral contact, local temporal change, and recent history, then jointly shapes it with future contact-change prediction and future alert supervision. The resulting representation summarizes evolving tactile context in relation to task risk and is computed at deployment without future observations or alert targets.

B. Learning Risk-Aware Tactile Representations

  • The representation combines bilateral contact configuration with local feature transitions and aggregates these transitions across recent interaction history.A two-layer LSTM processes a tactile history of length H = 9 with stride s = 5; its final hidden state is the risk-aware representation.Source: B. Learning Risk-Aware Tactile… · p. 3
  • Auxiliary heads predict short-horizon future changes in contact quantities and the task-conditioned alert at the same future target time.The future-change target uses df = 5 frames; the joint supervision encourages cues to be interpreted within preceding interaction context.Source: B. Learning Risk-Aware Tactile… · p. 3
  • A current-alert predictor achieves an average MSE of 0.0256 across validation episodes from all eight tasks, assessing alert-signal learnability rather than the history-conditioned future-alert objective.Source: B. Learning Risk-Aware Tactile… · pp. 3–4
  • Image-target regression combines a loss over the full target with one over significant-change regions, while alert prediction uses mean squared error.Training-set change statistics use the 90th percentile to construct the significant-change mask and the 99th percentile to normalize the regression target; γ_y controls emphasis on substantial changes.Source: B. Learning Risk-Aware Tactile… · p. 4
  • The joint objective organizes contact configuration, temporal transitions, and recent-history accumulation according to their relevance to task risk.Future observations and alert targets construct pretraining supervision only; deployment computes the representation from current and historical tactile observations.Source: B. Learning Risk-Aware Tactile… · p. 4

C. Lightweight Residual Integration into the Manipulation Policy

RATE adds a learned risk-aware residual to conventional tactile features, preserving the pretrained representation while supplying interaction-context information organized around task risk. During adaptation, alert targets also reweight imitation learning, but deployment uses only observations available up to the current time step.

C. Lightweight Residual Integration into the Manipulation Policy

  • RATE fuses a learned risk-aware residual with conventional tactile tokens rather than replacing the pretrained tactile representation.The same residual, derived from bilateral tactile history, is applied to both tactile tokens.Source: C. Lightweight Residual Integration… · p. 4
  • A projected risk-aware representation is transformed by a two-layer MLP and fused with conventional tactile tokens using a fixed task-level scaling coefficient.The fused tactile tokens join visual observations and proprioceptive state as inputs to the pretrained ACT policy, which predicts a 50-step action chunk.Source: C. Lightweight Residual Integration… · p. 4
  • Zero initialization makes the fused representation initially reproduce the original tactile tokens, while adaptation updates approximately 0.59% of model parameters.The conventional tactile encoder, RATE encoder, visual backbone, and ACT backbone remain frozen; only the input projection and residual adapter are optimized.Source: C. Lightweight Residual Integration… · p. 4
  • Alert targets reweight imitation loss toward action steps associated with more undesirable contact, while neither targets nor predicted alerts become policy inputs.At deployment, the policy uses visual, proprioceptive, and tactile observations available up to the current step.Source: C. Lightweight Residual Integration… · p. 4

IV. EXPERIMENTS

The experiments evaluate RATE on eight simulated manipulation tasks and three real-world contact-rich tasks. They examine overall performance, the contributions of alert-guided learning and interaction context, and task completion alongside contact-safe execution.

IV. EXPERIMENTS

  • The evaluation asks whether RATE outperforms representative manipulation policies overall without relying on large-scale models.This is one of the stated experimental questions.Source: §IV · p. 5
  • A second question examines how alert-guided learning and interaction context contribute individually and jointly, and how the representation integrates through lightweight residual adaptation.The evaluation also asks how effectively the resulting risk-aware representation can be integrated.Source: §IV · p. 5
  • The real-world experiments assess how effectively RATE achieves both task completion and contact-safe execution.The evaluation spans three real-world contact-rich tasks.Source: §IV · p. 5

A. Experimental Setup

The setup covers simulation benchmarks, controlled ablations, and three real-world tasks. It specifies shared evaluation protocols and reports task success alongside a contact-safety measure based on estimated tactile depth.

A. Experimental Setup

  • Simulation uses eight UniVTAC tasks across contact-rich interaction, pose reasoning, and shape perception, with a Franka Panda and two simulated GelSight Mini sensors.The policy receives RGB images, bilateral marker-based tactile images, and an eight-dimensional proprioceptive state.Source: A. Experimental Setup · p. 5
  • RATE and the retrained ACT+UniVTAC baseline use the same demonstrations and camera configuration, with 100 closed-loop rollouts per task.The comparison reports task-wise success, eight-task macro average, and model scale; other baseline results are published figures that may use different configurations.Source: A. Experimental Setup · p. 5
  • Ablations vary alert formulation and history-conditioned predictive context while holding the task-specific tactile base and other evaluation settings constant.Full RATE combines continuous severity alert supervision with context; the variants include alert-only and context-only configurations.Source: A. Experimental Setup · p. 5, Table II · p. 6
  • The real-world evaluation uses USB insertion, bottle-cap screwing, and plug insertion on an xArm 7 platform, with 50 demonstrations and 20 trials per task.Daimon measurements are converted to GelSight-like RGB tactile images, and the tactile encoder is retrained on those images.Source: A. Experimental Setup · p. 5
  • Contact-Safe Success Rate measures successful trials whose maximum estimated tactile depth stays below a preset threshold, and is zero when no trials succeed.Maximum tactile depth is a proxy for excessive tactile deformation, not a direct force measurement.Source: A. Experimental Setup · pp. 5–6

B. Simulation Results on the UniVTAC Benchmark

RATE achieves the highest reported macro-average success across the eight-task benchmark, with especially strong results on contact-rich interactions. It also leads in other task categories and outperforms larger compared policies.

B. Simulation Results on the UniVTAC Benchmark

  • 71.8% macro-average success rate puts RATE first or jointly first on six of eight tasks.It averages 72.0% on Contact-Rich Interaction, 23.0 percentage points above StarVLA-α.Source: B. Simulation Results on the UniVTAC… · p. 6
  • 71.8% versus 47.4% macro-average success rates show RATE outperforming ACT+UniVTAC (Retrained) under the controlled setting.On Pull-out Key, Insert Hole, Insert HDMI, and Insert Tube, RATE improves success by 38, 48, 27, and 41 percentage points, respectively.Source: B. Simulation Results on the UniVTAC… · p. 6
  • RATE achieves a 62.0% category-average success rate on Pose Reasoning and ties for best in Shape Perception with 100% success on Grasp Classify.The reported gains are most pronounced on the contact-rich interactions targeted by RATE.Source: B. Simulation Results on the UniVTAC… · p. 6
  • With approximately 133.2M parameters, RATE exceeds the strongest large-scale baseline, StarVLA-α, by 15.6 percentage points on unrounded macro-average success.The compared large-scale visuomotor policies have 3.3B–5B parameters.Source: B. Simulation Results on the UniVTAC… · p. 6

C. Ablation Studies

Controlled ablations show that alert-guided learning and context each improve performance over the task-specific tactile base. Their combination in Full RATE yields a further gain, including on contact-rich interactions.

C. Ablation Studies

  • The controlled ablation compares three alert formulations, context alone, and their combination in Full RATE from a common task-specific tactile base.The results are evaluated under the protocol defined in Sec. IV-A.Source: C. Ablation Studies · p. 6
  • Alert formulations raise macro-average success from the 44.0% task-specific tactile base to 56.1%, 57.5%, and 56.0%, while context alone reaches 56.0%.The three alert formulations are Discrete Ordinal, Continuous Severity, and Transition-Aware Continuous; context alone improves by 12.0 percentage points.Source: C. Ablation Studies · p. 6
  • 71.8% macro-average success in Full RATE exceeds the task-specific tactile base, continuous-alert, and context-only configurations by 27.8, 14.3, and 15.8 percentage points.The continuous-severity alert formulation is selected for Full RATE after achieving the best result among the three alert formulations.Source: C. Ablation Studies · pp. 6–7
  • On Contact-Rich Interaction tasks, residual adaptation raises average success from 49.3% to 72.0% over the continuous-alert baseline.Insert Hole rises from 32% to 71%, and Insert HDMI from 18% to 42%; only approximately 0.59% of model parameters are updated.Source: C. Ablation Studies · p. 7

D. Real-World Evaluation

Across three real-world contact-rich tasks, RATE achieves higher task success and contact-safe success than comparison methods. The results cover connector insertion and rotational tightening.

D. Real-World Evaluation

  • 73.3% versus 40.0% macro-average SR across the three real-world tasks shows RATE outperforming π0.5 by 33.3 percentage points.RATE achieves 75%, 85%, and 60% SR on USB Insertion, Bottle Cap Screwing, and Plug Insertion, respectively.Source: Table III · p. 7, D. Real-World Evaluation · p. 7
  • 92.22% versus 58.40% macro-average CSSR across the three tasks shows RATE outperforming π0.5 on contact-safe successful executions.CSSR is the proportion of successful trials satisfying the prescribed tactile-deformation limit.Source: Table III · p. 7, D. Real-World Evaluation · p. 7
  • Fig. 3 depicts USB Insertion, Bottle Cap Screwing, and Plug Insertion, with successive task stages arranged from left to right.Rows correspond to the three tasks in that order.Source: Figure 3 · p. 7
  • RATE achieves higher task success on every real-world task and a higher macro-average proportion of successful executions within the tactile-deformation limit.The evaluation covers USB Insertion, Bottle Cap Screwing, and Plug Insertion.Source: D. Real-World Evaluation · p. 7

V. CONCLUSION

The conclusion presents RATE as a lightweight way to ground task-conditioned risk in evolving tactile interaction. Across simulation and real-world tasks, it reports improved manipulation performance and contact-safe execution, while noting a limitation when contact magnitudes vary substantially.

V. CONCLUSION

  • RATE combines history-conditioned prediction and alert supervision to ground task-conditioned risk in evolving tactile context, then integrates the representation through a lightweight residual pathway.The conclusion reports improved performance across eight simulated and three real-world tasks, with particularly strong gains on contact-rich interactions; real-world evaluations also show a higher proportion of contact-safe successful executions.Source: §V · pp. 7–8
  • Substantial variation in contact magnitude across interaction stages may make subtle task-relevant tactile variations less distinguishable despite explicit modeling of local temporal changes.The stated future direction is scale-adaptive tactile encoding that preserves fine-grained relative variations and their absolute interaction magnitude.Source: §V · p. 8
Loading 2610.03538v1…