PaperlayerView exact revision on arXiv

Source-linked AI summary

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

Yoonhyung Park, Minji Kim, Sungwon Moon, Jiyoung Lee

arXiv:2607.00302v1cs.CVcs.MMcs.RO

TL;DR

Compact MLLMs need tactile reasoning while preserving established vision-language capabilities. Splash isolates tactile updates to dormant parameters, and achieves state-of-the-art visuo-tactile performance while preserving general-purpose capabilities without added inference cost.

  • Problem

    Compact MLLMs are needed for tactile reasoning under edge-robot constraints, but existing computationally intensive backbones limit practicality while vision alone struggles with intrinsic material properties.

  • Method

    Splash estimates parameter importance, freezes critical vision-language parameters, and updates only a dormant subspace alongside a negligible tactile front-end.

  • Results

    Splash achieves state-of-the-art performance across three visuo-tactile-language benchmarks while safeguarding pretrained general-purpose capabilities without additional inference overhead.

  • Takeaways & Limitations

    Splash expands tactile capability in small MLLMs through unified single-stage training while preserving their original vision-language reasoning and model scale.

  • Takeaways & Limitations

    The static dormant subspace may not capture temporally varying parameter utilization during long-horizon visuo-tactile reasoning.

Abstract

from arXiv · show

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

1 Introduction

Splash addresses the challenge of adding tactile reasoning to compact MLLMs without sacrificing pretrained vision-language capabilities. It isolates tactile updates to dormant parameters while freezing critical parameters as stable anchors, enabling single-stage adaptation without additional inference overhead.

  • Motivation: Existing visuo-tactile representations support retrieval and classification but lack higher-level semantic reasoning and instruction following.These capabilities are required for complex embodied decision-making.
  • Motivation: Compact MLLMs face a trade-off: tactile training can distort pretrained visual features, causing catastrophic forgetting and degraded vision-language reasoning.This limitation is especially acute for ≤3B-scale models intended for edge deployment.
  • Splash Framework: Splash estimates parameter importance from weight and activation statistics, freezes critical parameters, and restricts tactile adaptation to the dormant subspace.The frozen critical parameters serve as stable vision-language anchors while gradients update the isolated dormant parameters.
  • Splash Framework: Splash preserves the original MLLM scale and adds zero additional inference overhead, requiring only a negligible tactile front-end rather than external modules such as adapters.The framework uses unified single-stage training to optimize the tactile front-end and dormant LLM subspace concurrently.
  • Contributions: Splash integrates tactile sensing into small MLLMs while mitigating vision-related catastrophic forgetting.The contribution targets compact models including Qwen2.5-VL-3B and InternVL-1B.

2 Related Work

Related work establishes visuo-tactile integration as a way to resolve single-modality ambiguities, while recent MLLM research emphasizes vision-tactile-language alignment. Splash also builds on pruning-based multi-task learning and selective parameter updates to reduce task interference and catastrophic forgetting.

  • Visuo-tactile and MLLM alignment: Visuo-tactile integration addresses ambiguities in individual modalities across force estimation, cross-modal synthesis, and robotic manipulation.With the rise of MLLMs, research has shifted toward aligning tactile sensing with vision and language.
  • Parameter-efficient learning: Pruning-based multi-task learning assigns disjoint parameter subsets to different tasks to mitigate cross-task interference.This provides the underlying principle shared by Splash.
  • Parameter-efficient learning: Updating only a small fraction of LLM parameters can match competitive full-finetuning performance while curbing catastrophic forgetting.These findings motivate selectively exploiting model capacity without disturbing existing capabilities.

3 Splash

Splash adds tactile alignment to pretrained MLLMs by identifying a dormant LLM parameter subspace, freezing critical visual-language weights, and updating only the dormant subspace alongside the tactile front-end. This mask-isolated, unified training preserves pretrained visual-language pathways while enabling tactile description generation.

  • Framework: Splash generates a natural-language tactile description from an RGB image, tactile contact image, and text prompt.The model computes Y with an LLM over visual and tactile front-end features plus the text prompt.
  • Framework: The method initializes the LLM and visual front-end from a pretrained sMLLM, while using an ImageNet-pretrained ViT as the tactile front-end.This initialization transfers general-purpose reasoning and visual understanding, while the tactile encoder extracts geometric contact-image features.
  • Locating the Dormant Subspace: Splash identifies dormant LLM parameters with visual-relative importance scores and partitions them into frozen critical weights and trainable dormant weights.Scores combine weight magnitude and input-activation norm; first and last transformer blocks remain frozen to preserve grounding and high-level semantics.
  • Mask-Isolated Tactile Alignment: During unified single-stage training, the mask gives critical parameters zero gradient, updates dormant LLM parameters, and trains the tactile branch fully.The tactile front-end and masked LLM parameters are optimized together under the autoregressive language-modeling objective.
  • Mask-Isolated Tactile Alignment: Unlike additive PEFT methods such as LoRA, Splash confines updates exclusively to the dormant subspace, using frozen critical weights as stable anchors for pretrained visual-language reasoning.This structural isolation is intended to maintain the integrity of the multimodal manifold during tactile adaptation.

4 Experiments

Splash is evaluated on visuo-tactile benchmarks using compact multimodal models and achieves strong tactile reasoning while retaining general vision-language capabilities. Ablations and qualitative analyses support its robustness, while experiments identify current limitations in dynamic masking and cross-sensor generalization.

  • Experimental Setup: Splash trains on SSVTP and TVL, then evaluates robustness under distribution shift using the DIGIT-sensor subset of TacQuad.SSVTP contains 4.5K aligned visuo-tactile pairs, while TVL contains 44K pairs with natural-language tactile descriptions.
  • Analysis: Splash consistently outperforms baselines in F1 score and Top-5 Accuracy, whereas TVL-LLaMA and UniTouch often produce visually plausible but tactually inconsistent attributes.The qualitative mismatches lead to lower GPT-4o semantic-similarity scores for the baselines.
  • VL Capability Retention: 50.0 to 55.3 on MMMUval and 52.8 to 65.3 on MathVista show Splash-3B improves or preserves standard vision-language capability after tactile adaptation.Splash-3B also achieves higher MMBench-EN and MMBench-CN scores than TVL, while maintaining comparable MMEsum performance.
  • Ablation Study: 60% sparsity achieves a 4.91 tactile-semantic average while obtaining the best MMMUval reasoning performance and competitive MathVista and MMBench results.Moderate sparsity provides the strongest trade-off between tactile-semantic performance and VL capability retention.

5 Conclusion … A.1 Vision Forgetting Problem

Splash expands small MLLMs with tactile alignment by updating a dormant parameter subspace while anchoring critical visual-language parameters, addressing catastrophic forgetting in visual understanding. Qualitative examples show that tactile alignment without mask guidance can degrade visual descriptions and hallucinate unsupported attributes.

  • 5 Conclusion: Splash is a mask-isolated tactile alignment framework that expands small MLLMs’ sensory capabilities without catastrophic forgetting.It isolates dormant parameters using a visual-relative importance metric.
  • 5 Conclusion: Splash partitions parameters into dormant and critical subspaces using a visual-relative importance metric.Critical parameters preserve foundational vision-language reasoning, while dormant parameters provide capacity for tactile alignment.
  • 5 Conclusion: During training, Splash updates only the dormant subspace alongside the tactile front-end while keeping critical parameters anchored.This selective update is designed to expand modality capabilities without disrupting foundational visual-language reasoning.
  • 5 Conclusion: Extensive experiments across diverse visuo-tactile and vision-language benchmarks demonstrate Splash’s effectiveness.The supplied conclusion passage states broad benchmark evaluation but does not provide individual scores here.
  • A.1 Vision Forgetting Problem: The visual forgetting problem arises because tactile-aligned models can replace correct visual understanding with inaccurate descriptions or hallucinated attributes.The original Qwen2.5-VL recognizes objects and scenes correctly, whereas TVL frequently produces unsupported visual content.
  • A.1 Vision Forgetting Problem: 8/10 was scored by Qwen2.5-VL-3B, compared with 5/10 for Qwen2.5-VL-3B + Tactile branch w/o Mask Guided Alignment.The qualitative comparison illustrates degradation in visual descriptions after tactile alignment without mask-guided alignment.

A.2 Sparsity Analysis for SPLASH-1B

For Splash-1B, a 60% sparsity ratio best balances visuo-tactile and vision-language performance. Its VL performance is more sensitive to sparsity than Splash-3B’s, whose VL benchmarks remain relatively stable from 30% to 60%.

  • 60% sparsity provides a favorable balance between VTL and VL performance for Splash-1B.
  • Splash-1B’s VL performance varies more noticeably across sparsity levels than Splash-3B’s.
  • Splash-3B’s VL benchmarks remain relatively stable between sparsity ratios of 30% to 60%.

A.3 Adaptive Sparsity Allocation · A.4 Dormant Subspace Importance Scores

The appendix evaluates adaptive versus uniform sparsity allocation under a fixed global budget and examines importance shifts within Splash’s trainable dormant subspace. Uniform sparsity performs better overall, while dormant parameters show distinct shifts in weight magnitude and activation responsiveness.

  • A.3 Adaptive Sparsity Allocation: Splash explores layer-wise sparsity allocation under a fixed global sparsity budget, motivated by differing transformer-layer redundancy and sensitivity.The strategy optimizes layer-wise sparsity while keeping the global budget fixed.
  • A.3 Adaptive Sparsity Allocation: LOD-based allocation assigns lower sparsity to layers with more activation outliers and higher sparsity to layers with fewer outliers.LOD is estimated from activation-weight importance statistics on a small vision-language calibration set, following OWL.
  • A.3 Adaptive Sparsity Allocation: Under a global 60% sparsity setting, uniform sparsity consistently dominates adaptive sparsity on VTL metrics and overall performance.Adaptive sparsity provides only marginal gains, whereas uniform sparsity is selected to ensure balanced tactile flow in dormant spaces.
  • A.3 Adaptive Sparsity Allocation: The adaptive allocation visualization uses LOD bars and an adaptive sparsity-ratio curve across transformer layers under the global 60% budget.Green bars denote LOD, while the orange curve denotes each layer’s assigned adaptive sparsity ratio.
  • A.4 Dormant Subspace Importance Scores: Dormant-subspace analysis compares pretrained Qwen2.5-VL 3B with Splash-3B at s = 60% on a TVL subset using sampled dormant parameters from intermediate linear layers.The analysis excludes the first and last transformer blocks and uniformly samples 15 million dormant parameters.
  • A.4 Dormant Subspace Importance Scores: 48.8% of dormant weights increase in magnitude, while 67.1% of corresponding activation responsiveness shifts show a significant majority increase.The visual-relative importance score is decomposed into weight magnitude, |W_i,j|, and activation ℓ2-norm, ∥x_j∥2, components.

A.5 Freezing Boundary Layers

Splash freezes the first and last transformer layers during visuo-tactile adaptation because they are sensitive to distribution shifts and important for preserving pretrained representations. Unfreezing both layers reduces the average VTL score, so they remain frozen for stable adaptation.

  • Boundary-layer freezing: The first and last transformer layers are frozen during visuo-tactile adaptation because they are sensitive to distribution shifts and tied to preserving pretrained representations.This boundary-layer choice follows prior work and applies throughout adaptation.
  • Boundary-layer freezing: 0.18 points: Unfreezing both boundary layers reduces the average VTL score, indicating harm to tactile–visual-language balance.The observed decrease motivates keeping these layers frozen throughout all experiments.

A.6 Inference cost

Splash is evaluated for end-to-end inference latency under fixed hardware, input, prompt, and generation settings. The reported findings indicate that it adds no inference overhead in the LLM part and is suitable for real-time robotic perception loops.

  • Latency measurement: End-to-end latency is measured on a single NVIDIA RTX PRO 6000 Blackwell GPU with batch size 1, identical inputs and prompts, and up to 6 generated tokens.The protocol uses 50 warm-up iterations and 200 inference runs, discarding the first 50 measured runs to account for initialization overhead.
  • Efficiency: Splash integrates into compact VLMs without compromising inference efficiency, supporting real-time robotic perception loops.The passage characterizes Splash as suitable for real-time deployment without additional inference overhead in the LLM part.

A.7 Failure Case Analysis

The failure analysis identifies tactile perceptual aliasing as a source of material misclassification, particularly when minimally deformed fabric surfaces produce nondistinctive tactile imprints. This reflects difficulty integrating fine-grained surface geometry that high-resolution sensors such as GelSight can capture.

  • Material misclassification under tactile perceptual aliasing: Tactile perceptual aliasing causes material misclassification when minimally deformed fabric surfaces produce ambiguous, nondistinctive tactile imprints.This failure appears in an SSVTP case involving textile material properties.
  • Material misclassification under tactile perceptual aliasing: The error indicates inadequate integration of fine-grained surface geometry, which micron-resolution GelSight sensors can capture.The passage links the limitation to tactile signals containing detailed surface-geometry information.

A.8 Additional Qualitative Results · B LLM Judge Prompt

Additional qualitative examples across TacQuad, SSVTP, and TVL show that the proposed models generate more diverse and detailed tactile descriptions than baseline methods. For VTL evaluation, GPT-4o judges tactile descriptions using a prompt containing the user question, assistant response, and correct response.

  • A.8 Additional Qualitative Results: The additional examples cover the TacQuad, SSVTP, and TVL benchmarks.They are presented in Fig. A5, Fig. A6, and Fig. A7, respectively.
  • A.8 Additional Qualitative Results: The proposed models produce more diverse and detailed tactile descriptions than baseline methods.This conclusion is drawn from the additional qualitative examples across the three benchmarks.
  • B LLM Judge Prompt: Following TVL, GPT-4o serves as an LLM judge for tactile descriptions on VTL benchmarks.The judge evaluates tactile descriptions using the specified evaluation prompt.
  • B LLM Judge Prompt: The VTL prompt supplies the user question, assistant response, and correct response for evaluation.It asks the judge to assess how closely the assistant response matches the correct response describing tactile feelings.
  • A.8 Additional Qualitative Results: Fig. A5 provides additional qualitative comparisons on TacQuad.Each example shows a natural RGB image, a tactile contact image, predicted texts from baselines and the proposed models, and evaluation scores out of 10.
  • A.8 Additional Qualitative Results: Fig. A6 provides additional qualitative comparisons on SSVTP.The passage identifies the figure as additional qualitative comparisons on SSVTP without further methodological detail.
  • A.8 Additional Qualitative Results: Fig. A7 provides additional qualitative comparisons on TVL.The passage identifies the figure as additional qualitative comparisons on TVL without further methodological detail.
Loading 2607.00302v1…