PaperlayerView exact revision on arXiv

Source-linked AI summary

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

Yoonhyung Park, Minji Kim, Sungwon Moon, Jiyoung Lee

arXiv:2607.00302v1cs.CVcs.MMcs.RO

TL;DR

Vision-based systems often struggle to infer intrinsic material properties, while compact MLLMs must gain tactile reasoning without losing established vision-language capabilities. Splash confines tactile adaptation to dormant parameters and keeps critical parameters frozen. Experiments report state-of-the-art visuo-tactile benchmark performance while preserving general-purpose capabilities.

  • Problem

    Vision-based systems often struggle to infer intrinsic material properties, while computationally intensive MLLM backbones limit practical use on resource-constrained edge robots.

    Source: §1 · pp. 1–2

  • Method

    Splash estimates parameter importance, freezes the critical subspace, and restricts tactile adaptation to the dormant subspace.

    Source: §1 · p. 2

  • Results

    Splash-3B achieves state-of-the-art performance on SSVTP, TVL, and TacQuad, outperforming previous 7B-scale methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Splash adds tactile reasoning with zero additional inference overhead while safeguarding pretrained general-purpose capabilities.

    Source: §1 · p. 3, §5 · p. 15

    Because the dormant subspace remains static after offline splitting, it may not capture changing parameter use during long-horizon visuo-tactile reasoning.

    Source: §4.9 · p. 14

Abstract

from arXiv · show

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

1 Introduction

Touch reveals friction and compliance that vision may not resolve, yet adding touch to compact MLLMs risks degrading pretrained visual reasoning. Splash confines tactile adaptation to dormant parameters while freezing critical parameters.

1 Introduction

  • Vision often struggles to infer friction and compliance when properties are occluded or visually indistinguishable, motivating compact MLLMs for tactile reasoning on resource-constrained edge robots [11].The paper presents touch as physical feedback important for precise interaction and robotic dexterity.Source: §1 · pp. 1–2
  • Earlier shared visuo-tactile-language representations support retrieval and classification but lack higher-level reasoning and instruction following, and generalize poorly to novel objects with visually ambiguous textures.The authors argue that embodied decision-making requires models that reason over multimodal sensory inputs.Source: §1 · p. 2
  • In small MLLMs of 3B parameters or fewer, tactile optimization can distort pretrained visual features, causing catastrophic forgetting and a trade-off between sensory expansion and preserved vision-language reasoning.The introduction illustrates this failure with tactile training on TVL and Qwen2.5-VL-3B.Source: §1 · p. 2, Figure 1 · p. 2
  • Splash estimates parameter importance using a VL-relative metric from weight and activation statistics, updates only the dormant subspace, and freezes critical parameters as stable vision-language anchors.This selective optimization is intended to preserve pretrained reasoning while incorporating tactile information.Source: §1 · pp. 2–3
  • Splash supports unified single-stage training of the tactile front-end and dormant LLM subspace, maintaining the original MLLM scale without additional inference overhead or external modules beyond a negligible front-end.It avoids multi-stage alignment and fine-tuning procedures.Source: §1 · p. 3
  • Splash-3B achieves state-of-the-art performance on SSVTP, TVL, and TacQuad, outperforming prior 7B-scale methods while safeguarding pretrained general-purpose capabilities across small MLLMs.The reported models include Qwen2.5-VL-3B and InternVL-1B.Source: §1 · p. 3

2 Related Work

Prior work combines vision and touch to resolve single-modality ambiguity and aligns tactile sensing with vision-language representations, while compact MLLM integration and modality-level parameter isolation remain open challenges.

2.1 Visuo-Tactile-Language (VTL) Alignment

  • VTL research progressed from robotic sensor integration to contrastive alignment, scalable datasets, and cross-sensor representations, but tactile integration in compact MLLMs remains underexplored.Earlier integration addressed force estimation [6], cross-modal synthesis, and robotic manipulation [23].Source: §2.1 · p. 3

2.2 Pruning-based Multi-Task Learning

  • Pruning-based multi-task learning isolates parameter subsets to limit interference, but prior work focuses on intra-modality tasks rather than modality-level tactile alignment.LLM studies report that updating a small parameter fraction can be competitive with full finetuning while curbing catastrophic forgetting.Source: §2.2 · p. 4

3 Splash

Splash addresses the risk that tactile tuning overwrites pretrained visual knowledge by training selected dormant LLM weights alongside a tactile front-end while preserving critical visual-language weights.

3.1 Motivation and Problem Formulation

  • Splash formulates tactile-description generation from an RGB image, tactile contact image, and text prompt, initializing the LLM and visual front-end from a pretrained sMLLM.An ImageNet-pretrained ViT extracts geometric features from contact images; visual parameters are frozen while tactile parameters and a sparsity-selected subset of LLM parameters are updated.Source: §3.1 · p. 4

3.2 Locating the Dormant Subspace

  • Splash scores LLM weights by magnitude times input-activation L2 norm, then marks the lowest-scoring fraction dormant and excludes the first and last transformer blocks.Importance scores use a small visual-language calibration set, such as 128 CC12M samples; the resulting mask separates trainable dormant weights from frozen critical weights to stabilize tactile adaptation.Source: §3.2 · pp. 5–6

3.3 Mask-isolated Tactile Alignment

  • Splash jointly trains the tactile front-end and dormant LLM weights in one stage, freezing critical weights as anchors to prevent forgetting without added inference latency.The method replaces updates in the dormant subspace rather than adding a PEFT module, eliminating a separate alignment stage and simplifying training.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate Splash’s tactile alignment and retention of pretrained vision-language capabilities. Results, ablations, and qualitative examples assess its performance and the conditions affecting adaptation.

4.1 Datasets; 4.2 Evaluation Protocol

  • Evaluation uses SSVTP (4.5K pairs), TVL (44K pairs), and DIGIT TacQuad (72K samples), with GPT-4o semantic scoring plus keyword F1 and Top-5 Accuracy.TacQuad evaluation uses each contact trajectory’s middle frame; standard VL benchmarks assess retained general-purpose capabilities.Source: §4.1 · p. 7, §4.2 · p. 7

4.3 Implementation Details; 4.4 Baselines

  • Experiments use Qwen2.5-VL-3B and InternVL2.5-1B backbones with a 5.8M-parameter tactile encoder; a matched Qwen2.5-VL-3B LoRA baseline isolates adaptation strategy.Splash is also compared with TVL-LLaMA, UniTouch, and (IA)3; the tactile encoder is a ViT-Tiny initialized from ImageNet weights.Source: §4.3 · pp. 7–8, §4.4 · p. 8

4.5 Performance Comparison on VTL Tasks

  • Splash-1B reaches a 5.01 average tactile-semantic score, while Splash-1B and -3B outperform UniTouch’s 3.74; Splash also leads baselines in F1 and Top-5 Accuracy.The comparison covers SSVTP, TVL, and TacQuad; Splash-1B’s 5.01 average is reported despite its smaller backbone.Source: Table 1 · p. 9, §4.5 · pp. 9–10, Figure 3 · p. 9

4.6 Performance Comparison for Preservation of VL Capability

  • Splash-3B raises MMMUval from 50.0 to 55.3 and MathVista from 52.8 to 65.3 versus TVL baselines, while remaining competitive on other VL benchmarks.TVL-(IA)3 leads on MMEsum, MMBench-EN, and MMBench-CN but scores 3.84 on TacQuad versus 4.86 for Splash-3B; Splash remains comparable to zero-shot Qwen2.5-VL-3B on MMMUval (53.1) and MathVista (62.3).Source: §4.6 · pp. 10–11, Table 2 · p. 10

4.7 Ablation Study

  • At 60% sparsity, Splash-3B achieves its highest tactile-semantic average, 4.91, and best MMMUval reasoning; 100% sparsity degrades VL reasoning, while 30% slightly limits tactile adaptation.Calibration results remain stable across sample sizes and corpora, including unpaired noise data; with matched tactile-pretrained frontend initialization, Splash outperforms TVL on VTL and VL benchmarks.Source: §4.7 · pp. 11–12, Table 3 · p. 11, Table 4 · p. 12

4.8 Qualitative Results

  • Across TVL, SSVTP, and TacQuad examples, Splash generates more accurate, diverse, and physically consistent tactile descriptions than baselines.TVL-LLaMA and UniTouch sometimes produce visually plausible but tactually inconsistent or generic attributes, which receive lower GPT-4o semantic similarity scores.Source: §4.8 · pp. 12, 14, Figure 4 · p. 13, Figure 5 · p. 14

4.9 Limitation and Future Work

  • Splash uses a static dormant subspace, which may not reflect changing parameter use during long-horizon interaction; experiments use one DIGIT sensor, leaving cross-sensor generalization for future work.The frontend-agnostic design can be extended to alternative tactile encoders.Source: §4.9 · pp. 14–15

5 Conclusion … A.1 Vision Forgetting Problem

Splash expands small MLLMs with tactile alignment by updating dormant parameters while anchoring critical visual-language reasoning, achieving state-of-the-art tactile reasoning while preserving general-purpose ability. The appendix illustrates the visual forgetting that motivates this approach.

5 Conclusion

  • Splash achieves state-of-the-art tactile reasoning across diverse VTL and VL benchmarks while fully preserving general-purpose ability.Source: §5 · p. 15
  • Splash updates visually low-importance dormant parameters alongside the tactile front-end while anchoring critical parameters for foundational VL reasoning.It identifies the dormant subspace using a visual-relative importance metric.Source: §5 · p. 15
  • Splash uses unified single-stage training and adds zero inference cost, offering a practical path toward high-level tactile intelligence for resource-constrained edge robots.Source: §5 · p. 15

A.1 Vision Forgetting Problem

  • The motivation for Splash is catastrophic forgetting, especially in the visual domain, during tactile alignment of MLLMs.Source: §A.1 · p. 19
  • The original Qwen2.5-VL recognizes visual objects and scenes correctly, whereas TVL often gives inaccurate descriptions or hallucinates attributes unsupported by visual input.Source: §A.1 · p. 19
  • Figure A1 presents additional qualitative examples of catastrophic forgetting in tactile alignment for MLLMs.Source: Figure A1 · p. 19, §A.1 · p. 19

A.2 Sparsity Analysis for SPLASH-1B

For Splash-1B, 60% sparsity balances VTL and VL performance, but the smaller backbone is more sensitive to masking ratio than Splash-3B.

A.2 Sparsity Analysis for SPLASH-1B

  • At 60% sparsity, Splash-1B balances VTL and VL performance, but its VL performance is slightly lower than at 30%-40%; Splash-3B VL benchmarks remain relatively stable from 30% to 60%.The analysis attributes Splash-1B’s greater sensitivity to its smaller capacity.Source: §A.2 · p. 20

A.3 Adaptive Sparsity Allocation · A.4 Dormant Subspace Importance Scores

Splash compares adaptive and uniform sparsity allocation, finding uniform allocation stronger on VTL metrics and overall performance. Analysis of dormant parameters shows tactile alignment is associated with increased activation responsiveness and importance scores.

A.3 Adaptive Sparsity Allocation

  • Prior work reports that transformer layers vary in redundancy and pruning sensitivity, motivating adaptive layer-wise sparsity under a fixed global budget.The paper also examines the default uniform allocation as a comparison.Source: §A.3 · p. 20
  • Following OWL, Splash estimates layer-wise outlier distributions from activation-weight importance statistics on a small vision-language calibration set.Layers with higher Layerwise Outlier Distribution (LOD) scores receive lower sparsity; layers with lower scores receive higher sparsity, subject to bounded layer-wise ratios and global rescaling.Source: §A.3 · p. 20
  • Uniform sparsity consistently dominates adaptive sparsity on VTL metrics and overall performance, while adaptive allocation yields only marginal gains.The comparison uses the same global sparsity budget of 60%.Source: §A.3 · p. 20, Table A2 · p. 21
  • Fig. A2 visualizes the resulting layer-wise sparsity distribution for Splash-3B at a global 60% sparsity budget.Green bars show LOD and the orange curve shows the adaptive sparsity ratio assigned to each transformer layer.Source: §A.3 · p. 20, Figure A2 · p. 21

A.4 Dormant Subspace Importance Scores

  • The dormant-subspace analysis samples 15 million parameters from intermediate transformer linear layers, excluding the first and last blocks, in Qwen2.5-VL 3B and Splash-3B.Both models are evaluated on a subset of TVL; the Splash model uses s = 60%.Source: §A.4 · p. 21
  • 67.1% of corresponding dormant-weight activations show a strong positive shift in ℓ2-norm, indicating increased responsiveness along the dormant pathway.The analysis evaluates a subset of TVL and decomposes visual-relative importance-score shifts into weight-magnitude and activation-norm changes.Source: §A.4 · p. 21
  • 63.0% of dormant weights show increased visual-relative importance scores after tactile alignment.The authors interpret this as evidence that Splash accommodates new modalities by amplifying input-driven activations along selectively trainable dormant pathways.Source: §A.4 · p. 21
  • 48.8% of dormant weights increase in magnitude, while the activation ℓ2-norm shift is positive for a majority of corresponding activations.The decomposition separates structural shifts in weight magnitude from responsiveness shifts in activation norm.Source: §A.4 · p. 21

A.5 Freezing Boundary Layers

Splash freezes the first and last transformer layers during visuo-tactile adaptation to preserve pretrained representations; unfreezing them lowers the average VTL score by 0.18 points.

A.5 Freezing Boundary Layers

  • Following prior work, Splash freezes the first and last transformer layers because they are sensitive to distribution shifts and closely tied to preserving pretrained representations.Source: §A.5 · pp. 21–22
  • Unfreezing both boundary layers reduces the average VTL score by 0.18 points, so Splash keeps them frozen throughout experiments for stable adaptation.The layers are the first and last transformer layers.Source: §A.5 · p. 22

A.6 Inference cost

Splash matches the Qwen2.5-VL-based TVL baseline in inference latency and control frequency, while both are faster than the LLaMA-7B-based TVL baseline. The results support integrating tactile reasoning into compact VLMs without compromising inference efficiency for real-time robotic perception loops.

A.6 Inference cost

  • 134.55 ms P99 latency and 7.50 Hz control frequency are shared by TVL-Qwen2.5-VL and Splash, versus 202.37 ms and 4.97 Hz for the LLaMA-7B-based TVL baseline.Measurements used a fixed TVL sample, batch size 1, and up to 6 output tokens on one NVIDIA RTX PRO 6000 Blackwell GPU; the Qwen2.5-VL-based baseline was measured after merging LoRA weights.Source: §A.6 · p. 22
  • Splash integrates tactile reasoning into compact VLMs without compromising inference efficiency, making it suitable for real-time robotic perception loops.Source: §A.6 · p. 23

A.7 Failure Case Analysis

The failure analysis identifies two limits: tactile perceptual aliasing can make fabric resemble smooth rigid material, and competitive models can fail to integrate tactile cues about fine surface geometry.

Case 1: Material misclassification under tactile perceptual aliasing

  • Tactile perceptual aliasing makes minimally deformed fabric resemble smooth rigid surfaces, leading Splash-3B to predict “metallic” and “smooth” and Splash-1B to predict “uneven, patterned”.The repetitive linear fabric texture can visually resemble brushed metal patterns.Source: §A.7 · p. 23

Case 2: Micro-texture misinterpretation and Visual Dominance

  • All competitive models, including Splash, misinterpret surface micro-geometry, indicating insufficient integration of fine-grained tactile signals.High-resolution tactile sensors such as GelSight can capture this geometry at micron-level resolution.Source: §A.7 · pp. 23–24

A.8 Additional Qualitative Results · B LLM Judge Prompt

Additional examples show that Splash models produce more diverse and detailed tactile descriptions than baseline methods across TacQuad, SSVTP, and TVL; the appendix also specifies a GPT-4o judge prompt for VTL evaluation.

A.8 Additional Qualitative Results

  • Across TacQuad, SSVTP, and TVL, additional examples illustrate that the proposed models produce more diverse and detailed tactile descriptions than baseline methods.Source: §A.8 · p. 24

B LLM Judge Prompt

  • For VTL benchmark evaluation, GPT-4o serves as an LLM judge of tactile descriptions.Source: B LLM Judge Prompt · p. 24
  • The judge compares response and correct-response semantics for tactile feelings, scores overall performance from 1 to 10, and returns the score on one line followed by an explanation.Source: B LLM Judge Prompt · p. 24

A.8 Additional Qualitative Results

  • TacQuad examples compare baseline and model predictions alongside RGB and tactile contact images and evaluation scores out of 10.Source: Figure A5 · p. 25
  • Additional qualitative comparisons are provided for SSVTP.Source: Figure A6 · p. 26
  • Additional qualitative comparisons are provided for TVL.Source: Figure A7 · p. 27

Related papers

Loading 2607.00302v1…