PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs

Jiayue Jin, Jingwei Zhang, Chen Wang, Jing Liu, Longteng Guo

arXiv:2608.08107v1cs.CLcs.AI

TL;DR

Multimodal instruction tuning can degrade pretrained language intelligence, raising the question of how to add multimodal capabilities without losing language performance. NeuPAT allocates neuron-wise update flexibility based on heterogeneous adaptation patterns, recovering 90.0% of lost performance across four language reasoning benchmarks while preserving comparable multimodal performance across diverse LLMs.

  • Problem

    Multimodal instruction tuning can substantially degrade pretrained language intelligence, motivating methods that expand multimodal capabilities while preserving language-based reasoning.

  • Method

    NeuPAT probes heterogeneous neuron adaptation patterns and selectively regulates neuron-wise update flexibility to protect language-sensitive neurons while enabling multimodal adaptation.

  • Results

    90.0% of lost performance was recovered across four language reasoning benchmarks, with consistent preservation across six LLMs and comparable multimodal performance.

  • Takeaways & Limitations

    NeuPAT provides an efficient, architecture-agnostic approach for preserving language intelligence during multimodal expansion.

  • Takeaways & Limitations

    NeuPAT’s effectiveness on substantially larger backbones remains unverified, and extending it beyond vision-language expansion may require modality-specific strategies.

Abstract

from arXiv · show

Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.

1 Introduction

Multimodal instruction tuning can substantially degrade pretrained language capabilities, challenging the goal of additive capability expansion. NeuPAT addresses this through neuron-aware update constraints, recovering most language performance while retaining multimodal ability across diverse models and benchmarks.

  • 1 Introduction: 90.0% of lost performance is recovered across 4 language reasoning benchmarks, while NeuPAT generalizes across 6 LLMs, 11 language benchmarks, and 5 multimodal benchmarks.NeuPAT substantially reduces language degradation while maintaining comparable multimodal performance across model families and scales.
  • 1 Introduction: Preserving language capability is fundamental because language-grounded MLLMs rely on it for reasoning, knowledge organization, and multimodal generalization.The introduction frames language preservation as necessary for capability evolution rather than merely preventing regression.
  • 1 Introduction: Multimodal instruction tuning causes an average 39.8% performance drop across key language evaluations by interfering with pretrained language representations.The degradation shows that multimodal expansion is not purely additive and can overwrite capabilities needed for language-based reasoning.
  • 1 Introduction: Pretrained LLM neurons exhibit heterogeneous adaptation patterns: some preserve language intelligence, whereas others more flexibly absorb multimodal knowledge.This internal adaptation dynamic motivates selective regulation during multimodal learning.
  • 1 Introduction: NeuPAT uses lightweight probing with diagnostic samples to estimate neuron-wise adaptation and dynamically protect language-sensitive neurons while promoting multimodal adaptation.The framework requires no additional training data or architectural modification, making it architecture-agnostic and efficient.

2 Related Work

MLLMs commonly connect visual encoders to pretrained LLMs through cross-attention, Q-Formers, or lightweight projections, while recent systems improve perception and reasoning through stronger encoders, larger corpora, and post-training. Existing language-preservation strategies mix text-only data, use parameter-efficient or continual-learning methods, modify architectures, or merge adapted models; NeuPAT instead leverages neuron modality preferences.

  • MLLM architectures: MLLMs typically map visual features into the language space using cross-attention, Q-Former, or lightweight projection modules (Alayrac et al., 2022; Li et al., 2023a; Dai et al., 2023; Zhu et al., 2024; Liu et al., 2023b, 2024a).This encoder-connector-LLM architecture is the prevailing design described in the related work.
  • Recent MLLMs: Recent MLLMs, including Qwen-VL, DeepSeek-VL, InternVL3, and LLaVA-OneVision, improve multimodal perception and reasoning through stronger encoders, larger corpora, and advanced post-training (Bai et al., 2023; Wang et al., 2024; Bai et al., 2025; Lu et al., 2024; Wang et al., 2025b; Zhu et al., 2025; Li et al., 2024).Despite these advances, most MLLMs still adapt pretrained LLM backbones through multimodal learning.
  • Alternative approaches: Alternatives use parameter-efficient or continual-learning methods (Hu et al., 2022; Kirkpatrick et al., 2017), modify architectures (Zhang et al., 2024; Wang et al., 2025a; Lu et al., 2025), or merge adapted models with the original LLM (Ratzlaff et al., 2024; Yu and Ananiadou, 2025; Wang et al., 2026; Li et al., 2025).NeuPAT instead leverages neuron modality preferences to preserve language capabilities during multimodal adaptation.

3 Neuron-Level Modality Preference Analysis

The analysis shows that pretrained LLM neurons have heterogeneous modality preferences, importance, and layer-dependent plasticity during multimodal adaptation. These patterns motivate neuron-aware update constraints instead of uniform backbone updates.

  • Neuron-level analysis: The method estimates neuron importance from lightweight text-only and visual probing sets using activation strength and output contribution, then normalizes scores within each layer.The resulting modality preference distinguishes visual-associated, language-associated, and balanced neurons, while overall importance captures importance across both modalities.
  • Observations: Neurons exhibit diverse preferences for language and visual inputs, so uniform updates may perturb language-critical representations while underusing neurons suited to multimodal adaptation.This heterogeneity indicates that the pretrained backbone does not participate uniformly in multimodal adaptation.
  • Observations: Different neurons require different update constraints: highly important neurons may support shared cross-modal functions, whereas low-importance neurons may offer flexibility for acquiring multimodal knowledge.Balanced modality preference does not imply homogeneous neuron function or identical protection requirements.
  • Observations: Plasticity patterns vary substantially across Transformer layers, making fixed global allocation strategies insufficient for layer-specific adaptation characteristics.Both modality preference and overall importance show layer-dependent distributions.
  • Motivation: These findings motivate protecting language-critical neurons while allocating greater update flexibility to neurons better suited for multimodal adaptation.Uniform freezing or regularization could preserve language capabilities but unnecessarily restrict multimodal knowledge acquisition.

4 Methods

NeuPAT allocates neuron-wise update flexibility during multimodal instruction tuning by estimating modality-associated importance, assigning adaptation roles, and applying role-specific constraints. It protects language-sensitive neurons while preserving adaptive capacity for multimodal learning through selective updates and constrained optimization.

  • Importance Estimation: NeuPAT estimates visual- and language-associated neuron importance from lightweight visual and text-only probing sets, then normalizes scores within each layer.The resulting modality-associated distributions characterize each neuron’s contribution under different input modalities and guide adaptation-role allocation.
  • Adaptation-Role Allocation: Using predefined coverage ratios, NeuPAT selects modality-important neuron subsets and derives four adaptation roles from their overlap: language, multimodal, shared, and reserve.Language neurons are primarily associated with language inputs, multimodal neurons with visual inputs, shared neurons with both modalities, and reserve neurons provide additional flexibility.
  • Role-Specific Update Constraints: NeuPAT freezes language-associated neurons, fully updates multimodal-adaptive and reserve neurons, and constrains shared-neuron updates to balance language preservation with multimodal adaptation.For shared neurons, input-side constraints limit changes to activation behavior, while output-side cosine constraints preserve the direction of neuron contributions to the residual stream.
  • Optimization Objective: During optimization, NeuPAT masks language-neuron gradients while applying the assigned constraints to other roles, using L = Lori + Rshared as the final training objective.Lori is the original autoregressive language-modeling objective.

5 Experiments

Experiments show that NeuPAT substantially restores language performance after multimodal tuning while preserving multimodal capability, generalizes across LLM backbones, and benefits from neuron-wise, layer-adaptive update allocation.

  • Main Results: NeuPAT recovers 5.14 language-average points after Vanilla Tuning, reaching 49.06 versus 48.99 for the original LLM, while multimodal average performance remains comparable at 61.05 versus 60.98.Vanilla Tuning lowers the language average from 48.99 to 43.92, with drops exceeding 10 points on BBH, bAbI, and MATH-500; NeuPAT improves language and logical-reasoning tasks while maintaining multimodal performance.
  • Backbone Generalization: NeuPAT consistently improves language performance over Vanilla Tuning across five additional LLM backbones and sometimes surpasses the original LLM average.The evaluation replaces Qwen3-4B-Instruct-2507 with Qwen3-0.6B, Phi-4-Mini-Instruct, Qwen2.5-7B/14B-Instruct, and Llama3.1-8B-Instruct, showing improvements across model families and scales.
  • Global vs. Neuron-Aware Update: NeuPAT achieves the best global-versus-neuron-aware trade-off, obtaining language and multimodal averages of 54.41 and 72.73.Freeze limits multimodal adaptation, uniform backbone updates provide insufficient language preservation, and global shared-neuron regularization fails to balance both objectives.
  • Neuron Allocation Strategy: Importance-guided, layer-adaptive neuron allocation consistently outperforms Random and Fixed-ratio role assignment baselines.Random preserves category sizes but assigns roles randomly, whereas Fixed-ratio uses the same selection ratio across layers.
  • Neuron-Wise Update Constraint: Role-specific constraints support freezing language neurons, updating multimodal and reserve neurons, and regularizing shared neurons.Updating language-associated neurons decreases text performance, freezing vision-associated or reserve neurons limits multimodal adaptation, and either fully updating or freezing shared neurons harms one capability.

6 Conclusion

Multimodal expansion can compromise pretrained language intelligence because neurons in the LLM backbone adapt heterogeneously. NeuPAT addresses this by regulating neuron update flexibility to balance language preservation with multimodal adaptation.

  • 6 Conclusion: NeuPAT regulates neuron update flexibility to preserve language intelligence while enabling multimodal adaptation.The framework is motivated by heterogeneous adaptation behaviors within pretrained LLM backbones.

7 Limitations

NeuPAT’s effectiveness on substantially larger backbones remains unverified, and its applicability beyond vision-language expansion requires further investigation.

  • NeuPAT’s effectiveness on substantially larger backbones remains unverified because larger models may exhibit more distributed and complex adaptation patterns.The neuron-wise role allocation mechanism may require further investigation at greater scale.
  • The experiments focus on vision-language expansion, leaving NeuPAT’s extension to modalities such as audio and video for future investigation.

8 Ethical Considerations … B Additional Related Work

The paper uses public datasets and benchmarks, frames neuron allocation as a functional approximation rather than a causal explanation, and requires standard safety evaluations before deployment. NeuPAT’s complete procedure is provided in Algorithm 1.

  • 8 Ethical Considerations: NeuPAT’s neuron allocation strategies are response-based functional approximations, not causal explanations of model behavior.This limitation qualifies how the method’s internal adaptation patterns should be interpreted.
  • 8 Ethical Considerations: All training data and evaluation experiments use publicly available datasets and benchmarks.
  • 8 Ethical Considerations: Models trained with NeuPAT should undergo standard safety evaluations before deployment.The passage specifically calls for safety, fairness, and robustness evaluation.
  • 8 Ethical Considerations: Safety, fairness, and robustness checks are especially important for high-stakes applications.
  • B Additional Related Work: The supplied passages do not provide substantive content for B Additional Related Work.
  • A.1 Overall Algorithm: The complete NeuPAT pseudocode is provided in Algorithm 1.
  • A Method Details: The supplied passages do not provide substantive content for A Method Details or B Additional Related Work.

B.1 Neuron-Level Analysis … C.2 Training Details

The paper contrasts prior neuron-level analyses with NeuPAT’s focus on neuron-wise adaptation during general multimodal learning, then details probing sets and a two-stage training pipeline. Probing uses activation statistics only, while training proceeds from adapter-only alignment to plasticity-guided visual instruction tuning.

  • B.1 Neuron-Level Analysis: NeuPAT studies neuron-wise adaptation dynamics during general multimodal learning, unlike prior work focused on knowledge editing, emotion control, unlearning, image translation, or vulnerability analysis.Prior studies associate neurons with modality-specific knowledge, task behaviors, and model capabilities, including ultra-sparse critical neurons whose perturbation can impair language ability.
  • C.1 Probing Sets: NeuPAT estimates modality-specific neuron responses from separate text-only and vision probing sets containing 2,048 samples each.Vision samples come from LLaVA-NeXT-780K, while text prompts come from general-domain datasets disjoint from evaluation benchmarks.
  • C.1 Probing Sets: Text probing covers instruction following, factuality, and general reasoning using evenly allocated samples from four public datasets.The sources include CodeAlpaca-20k, MetaMathQA, databricksdolly-15k, and HaluEval.
  • C.1 Probing Sets: Both probing sets are used only for forward-pass activation statistics and do not affect the training objective or parameter updates.The same processor and chat-template pipeline is used for both sets, with images supplied only to vision samples.
  • C Detailed Experimental Setup: The two-stage pipeline aligns images and text on LLaVA-558K for 2,500 steps using only the multimodal adapter, then tunes LLaVA-NeXT-780K for 3,500 steps with the language model, adapter, and vision encoder optimized.Stage 1 uses a global batch size of 8 and peak learning rate 1 × 10−4; Stage 2 uses batch size 224, 28 accumulation steps, and peak learning rate 1 × 10−5.
  • C.2 Training Details: Stage 1 uses eight A100 GPUs, micro-batch size 1, cosine decay from 1 × 10−4 to 1 × 10−6, and approximately five warmup steps.The warmup ratio is 0.002 during image-text alignment on LLaVA-558K.
  • C.2 Training Details: Stage 2 applies plasticity-guided multimodal tuning to language-backbone neurons during visual instruction tuning on LLaVA-NeXT-780K.Training uses eight A100 GPUs, micro-batch size 1, 28 gradient-accumulation steps, and minimum learning rate 1 × 10−6.

C.3 Baseline Implementations · D Additional Experimental Results

The baseline implementations configure LoRA, EWC, WINGS, TIES, Locate-then-Merge, and PlaM using their specified training, architectural, or merging procedures and recommended hyperparameters.

  • C.3 Baseline Implementations: LoRA (Hu et al., 2022) inserts trainable low-rank adapters into selected linear layers while freezing original parameters, using r = 32, α = 64, and learning rate 1 × 10−4.LoRA is trained on the same multimodal instruction data as Vanilla.
  • C.3 Baseline Implementations: EWC (Kirkpatrick et al., 2017) estimates parameter importance with diagonal Fisher information from 2,048 disjoint text-only samples and penalizes changes only to language-model parameters.The search selects λEWC = 10 according to text-multimodal performance, with remaining settings identical to Vanilla Tuning.
  • C.3 Baseline Implementations: WINGS (Zhang et al., 2024) uses parallel visual and textual learners in attention layers, fuses them with original attention through a learned router, and follows recommended settings.
  • C.3 Baseline Implementations: TIES (Ratzlaff et al., 2024) performs training-free task-vector merging between the visually tuned model and original LLM, following the original density, coefficient, and procedure.
  • C.3 Baseline Implementations: Locate-then-Merge (Yu and Ananiadou, 2025) identifies high-impact neurons, suppresses low-impact updates, and restores selected neurons through replacement or rescaling using recommended hyperparameter ranges.
  • C.3 Baseline Implementations: PlaM (Wang et al., 2026) locates a plateau layer through layer-wise vision-token masking, leaves earlier visual-alignment layers unchanged, and linearly merges selected later layers with the original backbone.The plateau layer and merge coefficients follow the original paper's procedure and recommended range.

D.1 Sensitivity Analysis … D.4.1 Neuron Distribution across LLM Backbones

NeuPAT remains robust across hyperparameter settings, model backbones, ablations, and layer-wise neuron distributions. Complete evaluations show improved language preservation over Vanilla Tuning while maintaining comparable multimodal performance, with balanced defaults and consistent neuron-role structure.

  • D.1 Sensitivity Analysis: The default target importance mass τa = 0.8 provides a favorable balance: larger τa improves language preservation but gradually reduces multimodal performance.Larger τa retains more cumulative response mass, expands selected neuron sets, and assigns more neurons to text-related or high-response groups.
  • D.1 Sensitivity Analysis: The default λin = λout = 0.1 offers the best text–multimodal trade-off, whereas weaker regularization favors multimodal performance and stronger values reduce it.A coefficient of 0.05 achieves the highest multimodal average but lower language performance; values above 0.1 provide no further text improvement.
  • D.2 Complete Cross-Backbone Results: NeuPAT improves average language performance over Vanilla Tuning by 2.64, 5.20, 4.12, 3.64, and 4.76 points on five tested backbones while retaining comparable multimodal performance.The gains correspond to Qwen3-0.6B, Phi-4-Mini-Instruct, Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, and Qwen2.5-14B-Instruct, respectively, and generalize across model families and scales.
  • D.1 Sensitivity Analysis: Probing-set size has limited impact, with text and multimodal average variations of only 0.52 and 0.36 points; Na = 2048 achieves the best averages on both.Increasing Na from 512 to 2048 generally improves performance, whereas 4096 provides no additional benefit.
  • D.3 Complete Ablation Results: The complete ablations confirm that global updates, neuron allocation, and neuron-wise plasticity strategies trade off language preservation against multimodal adaptation.Alternative partitioning verifies the importance of adaptation-role allocation, while update-strategy ablations establish the role of each neuron set.
  • D.3 Complete Ablation Results: Among high-response-neuron regularization designs, l2-cos achieves the highest text average while maintaining near-best multimodal performance.The results suggest magnitude constraints benefit input-side parameters, whereas output-side parameters are better regularized with cosine constraints.
  • D.4.1 Neuron Distribution across LLM Backbones: Across backbones and layers, text-critical, vision-critical, high-response, and low-response neurons consistently coexist, with high-response neurons generally forming the largest group.Proportions vary across model families, scales, and layers; early layers fluctuate more in some models, while later layers tend to stabilize.
Loading 2608.08107v1…