TACTIC: Understanding Tactile Encoders and Conditioning for Contact-rich Robot Manipulation Policies

A. Tactile Sensing for Robot Manipulation

• Tactile feedback improves contact-rich and visually ambiguous manipulation across several types of single-task imitation policies,,,,.Source: A. Tactile Sensing for Robot… · p. 2
• Slow-fast and hierarchical policies couple global action generation with higher-rate feedback, while VLA work explores heterogeneous tactile modalities, representation alignment, and contact-aware reasoning,,,,,,,.Source: A. Tactile Sensing for Robot… · p. 2

B. Encoding Tactile Information

• Tactile encoders range from sensor-general transformers and temporal representations to self-supervised ViTs, compact VQGAN latents, and spatially aware ResNet-18 models,,,,, [25].The study evaluates a representative subset of these approaches; selection criteria are detailed in Section II.Source: B. Encoding Tactile Information · p. 2
• Single-task imitation policies use fusion strategies including concatenation, FiLM conditioning, cross-attention, learned gating, and contrastive visuo-tactile representation learning,,,, [27],,,,.Source: B. Encoding Tactile Information · p. 2

C. Benchmarking Tactile Manipulation

• ManiFeel, TacO, and CONTACT benchmark tactile sensing and representation choices, and AnyTouch2 includes downstream policy ablations, but backbone–fusion interaction remains comparatively underexplored in a common policy and evaluation environment,,,.Source: C. Benchmarking Tactile Manipulation · p. 2
• The study addresses this gap with controlled ablations of tactile encoders and fusion mechanisms using ACT, excluding VLAs and hierarchical systems to isolate representation and fusion effects.ACT is widely used in tactile imitation learning, and prior work suggests tactile-driven gains may follow similar trends under Diffusion Policy.Source: C. Benchmarking Tactile Manipulation · p. 2

A. Data Collection and Inference Setup

• Two Franka FR3 arms enable bilateral leader–follower teleoperation with haptic feedback.Source: A. Data Collection and Inference Setup · p. 2
• The teleoperator controls the leader in gravity compensation mode, while the follower tracks the leader’s joint positions.Source: A. Data Collection and Inference Setup · p. 2
• The follower’s external torque readings are added to the leader’s joint torques to provide haptic feedback.Source: A. Data Collection and Inference Setup · p. 2
• Haptic feedback gains are adjusted between 1 and 1.5 depending on the task.Source: A. Data Collection and Inference Setup · p. 2
• The follower’s custom Franka Hand fingers rigidly hold two DIGIT sensors, and a Realsense D405 serves as its wrist camera.Source: A. Data Collection and Inference Setup · p. 2
• A diagonally positioned Realsense D435 provides a full view of the follower and workspace; inference uses all modalities, including sensor images and proprioceptive joint and gripper state.Source: A. Data Collection and Inference Setup · p. 2

B. Tasks

• Four contact-rich tasks are evaluated, with success tracked across initialization, grasp, task attempt, task completion, and task termination.The tasks are depicted in Figure 2.Source: B. Tasks · p. 2, Figure 2 · p. 3
• For each task, the study collects approximately 100 demonstrations at 30 Hz using four pairs of DIGIT cartridges.The cartridge variety is intended to increase tactile-appearance diversity and reduce overfitting to cartridge-specific characteristics.Source: B. Tasks · p. 2

1) Board wiping:

• Board wiping tests sensitivity to normal contact forces as the robot wipes a chalk line and returns the eraser to the board.Demonstrations vary the chalk line’s pose and orientation and the board height.Source: 1) Board wiping · p. 2
• Wiping succeeds when no residue remains on the board or vase, while screwing succeeds when the nut is fully tightened.Source: Figure 2 · p. 3

B. Tasks

• For USB-C insertion, success means the USB is inserted in the correct position, but it need not be forced in.Source: Figure 2 · p. 3

2) Screwing:

• Screwing requires tactile feedback to detect shear/slip characteristics of a fully tightened nut because there is no clear visual completion signal.The required number of turns varies across demonstrations.Source: 2) Screwing · p. 3

3) USB-C insertion:

• USB-C insertion tests whether touch aids precise insertion and recovery from small alignment errors caused by plug-pose variation within the gripper.The robot inserts the plug into a fixed port and returns to its initial pose; the environment remains fixed.Source: 3) USB-C insertion · p. 3

4) Vase wiping:

• Vase wiping requires controlling normal force and relating grasp pose to eraser contact geometry on a curved surface.The robot wipes a marked line from a 3D-printed vase whose line position varies around its circumference, then returns the eraser.Source: 4) Vase wiping · p. 3

C. Encoder Backbone

• Five tactile backbones share weights across the left and right sensors, while wrist and side-camera streams use a shared ImageNet-initialized ResNet18,.Unless otherwise noted, a background image estimated from 50 blank tactile frames is subtracted to reduce cartridge-specific appearance sensitivity.Source: C. Encoder Backbone · p. 3
• The tactile baseline is an ImageNet-initialized ResNet-18 fine-tuned with the action expert; SARL [25] is pretrained from scratch on tactile demonstrations, while UniT is adapted from released weights.SARL uses a ResNet-based self-supervised teacher-student formulation with spatially aware objectives; UniT learns a quantized tactile latent space using a VQ-GAN objective.Source: C. Encoder Backbone · p. 3
• Sparsh-DINO, is fine-tuned with frames from t and t −4 concatenated for temporal context, whereas T3 uses its medium DIGIT variant with released weights as-is.T3 uses sensor-specific encoders with a shared latent trunk and processes raw DIGIT images without background subtraction to match its pretraining configuration.Source: C. Encoder Backbone · p. 3
• The selected backbones span end-to-end CNN training, tactile-specific CNN self-supervision, discrete generative pretraining, temporal-context ViT pretraining, and heterogeneous multi-sensor pretraining.These categories correspond respectively to ResNet-18, SARL, UniT, Sparsh-DINO, and T3.Source: C. Encoder Backbone · p. 3
• Encoder pretraining or adaptation uses only demonstration tactile images, with 95% of sampled frames from closed-gripper periods and 5% from open-gripper periods.Pretrained tactile encoders are frozen during action-expert training; only the baseline ResNet tactile encoder is optimized end-to-end with the policy.Source: C. Encoder Backbone · p. 3

D. Multimodal Feature Fusion

• Concat joins visual and tactile features, while FiLM uses gripper state to modulate tactile channels with learned scale and shift parameters.When needed, a 1D convolution projects tactile features to the visual feature dimension; FiLM uses a two-layer GELU MLP and supplies an explicit contact prior because meaningful DIGIT deformation primarily occurs with the gripper closed.Source: D. Multimodal Feature Fusion · p. 3
• Gated cross-attention lets tactile tokens query visual tokens, then adds the update to the tactile representation through a learned gate without proprioceptive conditioning.It uses a single multi-head cross-attention layer, with visual tokens as keys and values.Source: D. Multimodal Feature Fusion · p. 3
• The CLIP-style variants use symmetric contrastive training with modality-specific GELU projectors, while the vision encoder stays frozen and tactile-backbone training depends on initialization.The ResNet-18 tactile baseline is optimized during pretraining; with pretrained tactile backbones, the backbone stays frozen and only a 1 × 1 convolution projection layer is learned. Following prior works,,, the projectors are discarded and the tactile encoder and projection layer are frozen during ACT training.Source: D. Multimodal Feature Fusion · pp. 3–4
• CLIP-R uses 256 randomly sampled examples across four task datasets, whereas CLIP-T uses 32 per batch from same-task trajectories at least 30 frames apart.Both batches are task-balanced and contain half closed-gripper samples and half samples uniformly drawn from full trajectories; the comparison tests whether temporally structured sampling adds benefit over random cross-modal alignment. CLIP-T follows VITaL.Source: D. Multimodal Feature Fusion · p. 4

E. Action Expert

• ACT is a transformer policy that predicts action chunks from camera observations and sensor data; each instance specializes to one dataset, and all policies used a 120-future-step horizon.Each action step specifies delta joint positions for the next T timestamps.Source: E. Action Expert · p. 4
• ACT temporally averages past action predictions to smooth motion; the coefficient was 0.01 for all tasks except USB-C, which used 0.1 to prevent the policy from getting stuck.Stronger averaging reduces small differences between consecutive chunks, while insufficient averaging can produce jerky, less accurate motion.Source: E. Action Expert · p. 4

III. EXPERIMENTS

• The comparison used 20 real-world inference rollouts per ablation condition and 2,180 total rollouts across five backbones, five fusion schemes, and four tasks, including out-of-domain experiments.A vision-only policy using side- and wrist-camera encoders provided a tactile-free comparison; the experiments asked which tactile encoders generalize across tasks and how fusion affects downstream policy performance.Source: §III · p. 4

A. In-domain Experiments

• Across in-domain ablations, Sparsh-CLIP variants averaged nearly 50% successful termination, but UniT-Concat slightly led completion; CLIP variants struggled on board, and ResNet18-Concat surpassed them on USB-C.Strong average performance concealed task-level variation, with tactile encoders acting as task specialists when paired with suitable fusion strategies.Source: A. In-domain Experiments · p. 4

1) Board Wiping:

• On board wiping, CLIP variants often failed beyond grasp because wiping depends on wrist-camera visibility and tactile feedback for maintaining contact and pressure.Many CLIP-R policies released the eraser after lifting it; those that wiped tended to repeat similar trajectories over the same board region.Source: 1) Board Wiping · p. 4
• Non-CLIP variants adapted their wiping trajectories; T3 performed well with all non-CLIP fusion strategies, while SARL, Sparsh, and UniT worked best with Concat.Except for ResNet18, Concat had the highest completion rate; the authors hypothesize that shared projection restricts alignment of similar tactile representations with diverse visual embeddings, reducing sensitivity to spatial variation.Source: 1) Board Wiping · p. 4

• Many FiLM and CLIP variants and the vision-only policy dropped the eraser after lifting it, possibly because occlusion makes grasping and release representations hard to distinguish.ACT lacks closed-loop feedback and task memory to track prior states or actions, which may contribute to confusing grasping with releasing.Source: 2) Vase Wiping · p. 4
• Concat and GCA overcome ambiguous representations; GCA can also selectively modulate tactile information to increase its relevance during contact.The results suggest useful task-relevant information is already present in encoder representations, without requiring a more complex fusion mechanism.Source: 2) Vase Wiping · pp. 4–5
• UniT-CLIP-T matched UniT-GCA at 55% termination and was more stable than UniT-Concat, while Sparsh favored CLIP-T and degraded little with CLIP-R.Source: Figure 3 · p. 5, 2) Vase Wiping · p. 5

3) USB-C Insertion:

• On USB-C insertion, ResNet18-Concat performs best at an 80% task termination rate, while more complex fusion strategies consistently reduce ResNet18’s performance.Source: 3) USB-C Insertion · p. 5
• For USB-C insertion, FiLM raises T3 to a 55% termination rate and 75% completion rate, with similar improvements for Sparsh and UniT but a performance drop for SARL.Source: 3) USB-C Insertion · p. 5
• On USB-C insertion, CLIP-R and CLIP-T produce jerky, sporadic motions; CLIP-R brings the plug near the port, but CLIP-T overshoots and cannot recover.VITaL reported a 20% improvement for CLIP-T over vision-only with four cameras, but this study did not reproduce it and hypothesizes that its two-camera setup cannot fully exploit CLIP-T.Source: 3) USB-C Insertion · p. 5
• Because vision-only already reaches a 55% termination rate on USB-C insertion, tactile information is not uniformly beneficial and poorly integrated tactile features may interfere with vision.Source: 3) USB-C Insertion · p. 5

4) Screwing:

• On screwing, Sparsh-CLIP-T achieves the best-in-task 90% termination rate, and nearly every encoder benefits from CLIP-T except SARL.CLIP-T encourages temporal feature alignment, which helps capture slip and shear-force cues once the cap is tight; Sparsh explicitly incorporates temporally separated tactile observations.Source: 4) Screwing · p. 6
• On screwing, T3-CLIP-T also achieves a 90% termination rate despite not being fine-tuned on the dataset.The paper suggests T3’s material-texture-classification and pose-estimation pretraining may preserve shear-force and object-slip cues that emerge when the cap is fully tightened.Source: 4) Screwing · p. 6
• Probing confirms that T3 and Sparsh better detect when the nut is tightened.Table I reports linear probing of tactile encoders to distinguish the last from the second-to-last turn on the screw task [36].Source: 4) Screwing · p. 6, Table I · p. 6

5) Effect of CLIP Fusion:

• Raw tactile embeddings and CLIP counterparts differ only by a linear transformation, except ResNet18, which is optimized during contrastive pretraining.CLIP-based methods have the most substantial impact on performance among the fusion strategies considered.Source: 5) Effect of CLIP Fusion · p. 6
• On screw, CLIP methods increase performance from 40% up to 90%, whereas on vase most encoders collapse except Sparsh.The analysis focuses on vase and screw as tasks representing contrasting directions of CLIP's influence.Source: Table II · p. 6, 5) Effect of CLIP Fusion · p. 6
• The PCA analysis examines whether CLIP's 1-D projection expands or collapses tactile signal, with distributed embeddings potentially helping against premature stopping on screw.PCA offers insight into how CLIP modifies the tactile signal, not a prediction of downstream performance.Source: 5) Effect of CLIP Fusion · p. 6
• CLIP transforms the underlying tactile signal but cannot add information, and downstream performance still depends crucially on the task and tactile signal.The PCA analysis cannot ultimately predict downstream performance.Source: 5) Effect of CLIP Fusion · p. 6

B. Out-of-Domain Analysis

• The evaluation tested each task’s best-performing policy variants 10 times under visual shifts created by distractor objects and disco lights.Source: B. Out-of-Domain Analysis · pp. 6–7
• Under disco-light shift, both non-CLIP variants failed to grasp, while Sparsh-CLIP-T matched in-distribution performance on the screw task.The authors hypothesize that Sparsh-CLIP-T is less dependent on vision embeddings and relies more on proprioception and tactile embeddings.Source: Figure 6 · p. 7, B. Out-of-Domain Analysis · p. 7
• With distractors, non-CLIP variants grasped but failed termination; Sparsh-CLIP-T achieved 50% vase-task termination and matched in-distribution performance on the screw task.UniT-Concat could not localize the board task’s chalk line; ResNet18-Concat attempted USB-C insertion, succeeded in two cases, and degraded significantly.Source: Figure 6 · p. 7, B. Out-of-Domain Analysis · p. 7
• Under disco lights, Sparsh-CLIP-R had no successful termination, yet grasped the eraser and attempted the board task in more than half of cases; on USB-C, it inserted 3 out of 10 times.These additional tests used the best-performing CLIP variants for the board and USB-C tasks.Source: Figure 6 · p. 7, B. Out-of-Domain Analysis · p. 7
• Attention analysis found ResNet18-Concat primarily attended to the scene camera under normal lighting, but that signal was destroyed by disco lights; Sparsh-CLIP-R attended less to cameras and maintained its approach trajectory with higher probability.The analysis tracked the USB-C approach phase up to the first grasp under nominal and disco-light conditions.Source: Figure 7 · p. 7, B. Out-of-Domain Analysis · p. 7

IV. CONCLUSION

• Across 2,180 rollouts on four contact-rich tasks, TACTIC found no encoder–fusion combination best across all tasks, with the right choice depending on each task’s tactile signal.The study compared five tactile encoders and five multimodal fusion strategies for ACT policies.Source: §IV · p. 7
• Sparsh and T³ variants performed best on average; temporal information produced richer signals for tasks requiring shear forces and dynamic tracking.The results suggest tactile representations benefit from large-scale pretraining involving different sensors.Source: §IV · p. 7
• Simple concatenation remained competitive with more complex conditioning mechanisms.Source: §IV · p. 7
• Contrastive-based strategies induced specialist behavior rather than generalizing performance across tasks; PCA showed fusion redistributes variance or promotes an existing cue but cannot add information.Source: §IV · p. 7
• In out-of-distribution experiments, attention maps provided evidence that contrastive alignment lets policies attend more evenly to vision and tactile.Source: §IV · p. 7
• The study aims to guide future work toward conditioning mechanisms that generalize across different tactile requirements in contact-rich tasks.Source: §IV · pp. 7–8