PaperlayerView exact revision on arXiv

Source-linked AI summary

PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs

Zijing Wang, Yongkang Liu, Mingyang Wang, Ercong Nie, Deyuan Chen, Zhengjie Zhao, Shi Feng, Daling Wang, Xiaocui Yang, Yifei Zhang, Hinrich Schütze

arXiv:2601.07645v1cs.CL

TL;DR

Multimodal instruction fine-tuning can degrade the text reasoning inherited from base language models, weakening multimodal inference and visual grounding. PlaM uses layer-wise vision-token masking to identify late-layer plateaus, then merges base language-model parameters into those layers without training. Across five MLLMs and nine benchmarks, it reports consistent improvements and more focused attention on task-relevant visual regions, while the merging position requires dataset-dependent tuning.

  • Problem

    Multimodal instruction fine-tuning can degrade base language-model text reasoning, which supports interpreting and leveraging visual signals.

    Source: §1 · p. 1

  • Method

    PlaM profiles visual-information use with layer-wise vision-token masking and selectively injects base language-model parameters into plateau-phase late layers.

    Source: §1 · p. 2, §3 · p. 3, §5.1 · p. 4

  • Results

    Across five backbones and nine benchmarks, PlaM shows consistent improvements over original models and alternative merging schemes, with more focused late-layer visual attention.

    Source: §5.2 · p. 5, §7 · p. 8

  • Takeaways & Limitations

    Training-free late-layer merging can preserve earlier multimodal alignment while restoring semantic decision structure and improving visual grounding.

    Source: §1 · p. 2, §7 · p. 8

    The optimal merging position k0 varies across datasets and requires additional experiments and cost to determine.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) rely on strong linguistic reasoning inherited from their base language models. However, multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability, undermining multimodal performance. To address this issue, we propose a training-free framework to mitigate this degradation. Through layer-wise vision token masking, we reveal a common three-stage pattern in multimodal large language models: early-modal separation, mid-modal alignment, and late-modal degradation. By analyzing the behavior of MLLMs at different stages, we propose a plateau-guided model merging method that selectively injects base language model parameters into MLLMs. Experimental results based on five MLLMs on nine benchmarks demonstrate the effectiveness of our method. Attention-based analysis further reveals that merging shifts attention from diffuse, scattered patterns to focused localization on task-relevant visual regions. Our repository is on https://github.com/wzj1718/PlaM.

1 Introduction

MLLM fine-tuning can weaken inherited text reasoning while leaving models overly dependent on textual signals and poorly grounded visually. PlaM analyzes layer-wise visual use and selectively restores late-layer language capability through training-free merging.

Motivation

  • MLLMs can rely disproportionately on textual signals, while multimodal instruction fine-tuning can degrade the base model’s text reasoning ability.This degradation can impair interpretation and use of visual signals, contributing to scattered attention over irrelevant image regions.Source: §1 · p. 1
  • Existing remedies often require additional training, curated data, or architectural changes, leaving training-free solutions comparatively underexplored.Source: §1 · p. 2

Approach

  • PlaM uses layer-function analysis to identify early modal separation, middle modal alignment, and late-layer performance plateaus linked to weakened language reasoning.Layer-wise vision token masking reveals diminishing returns from additional visual access in late layers.Source: §1 · p. 2
  • PlaM selectively injects base language model parameters into underutilized late layers via linear interpolation, without additional training.Source: §1 · p. 2

Contributions

  • Across five MLLMs and nine benchmarks, experiments report consistent improvements, while attention analysis links gains to focused localization of task-relevant visual evidence.Source: §1 · p. 2

2 Related Work

Prior work mainly addresses language-reasoning degradation through training, auxiliary components, curated data, or representation steering. PlaM instead restores language-model functionality in underutilized late layers without training.

Language-reasoning degradation

  • Multimodal instruction tuning often degrades pretrained language models’ original text-only reasoning capabilities.Source: §2 · pp. 2–3

Training-based approaches

  • Existing approaches use auxiliary components, text-only data, preference-based alignment, or inference-time optimization, but require training, curated datasets, or architectural changes.Source: §2 · p. 3

Training-free approaches

  • Training-free alternatives include compressing non-text tokens or steering modality preference through latent representation offsets, with the latter requiring externally specified modality preference.Source: §2 · p. 3

PlaM

  • PlaM explicitly restores language-model functionality in underutilized late layers while improving visual grounding.Source: §2 · p. 3

3 Method

PlaM first profiles visual-information use across decoder depth with controlled masking, then merges base language-model parameters into selected layers while leaving visual components fixed.

Method overview

  • PlaM combines layer-wise vision-token masking with selective parameter merging between a base language model and its fine-tuned MLLM.The vision encoder and projector remain fixed during merging.Source: §3 · p. 3

Multimodal representation

  • Visual features are produced by a vision encoder, projected into language-model embedding space, and concatenated with embedded text tokens.The resulting multimodal sequence has total length N = Nvis + Ntxt and hidden dimension d.Source: §3 · p. 3

Decoder processing

  • The decoder processes the multimodal sequence through L Transformer layers, builds a KV cache, and generates output tokens autoregressively.Source: §3 · p. 3

Vision token masking

  • Vision-token masking blocks later layers from attending to visual keys and values while preserving the textual pathway, producing a layer-wise profile of visual contributions.Sweeping the masking layer identifies where visual information helps or becomes redundant.Source: §3 · p. 3

Model merging

  • For a target middle-to-late layer set, PlaM constructs merged parameters from the fine-tuned MLLM and corresponding base language model, preserving original parameters outside that set.The merging coefficient controls how much base-model capability is reintroduced.Source: §3 · pp. 3–4

4 Experiment settings

Experiments test PlaM across five MLLMs spanning LLaMA- and Qwen-based architectures and 3B–8B scales, using nine diverse vision-language benchmarks. Hyperparameters are selected by grid search and masking-based layer analysis.

Vision-language models

  • Experiments evaluate five representative MLLMs spanning LLaMA- and Qwen-based architectures and model scales from 3B to 8B parameters.Source: §4 · p. 4

Evaluation benchmarks

  • The benchmark suite covers reasoning, perception, multilingual understanding, visual grounding, and hallucination across MMStar, MMMU, MME, MMBench-EN/CN, GQA, RealWorldQA, SEED-Bench-2-Plus, and POPE.Source: §4 · p. 4

Hyperparameters

  • A unified simple merging strategy is used across models, with merging coefficients grid-searched from 0.0 to 1.0 in steps of 0.1.Source: §4 · p. 4
  • Target-layer starting points are selected using masking analysis, typically at the transition where visual information changes from beneficial to redundant.Reported results generally use optimal hyperparameters for each model-task combination.Source: §4 · p. 4

5 Visual Degeneration

Layer-wise masking reveals a consistent progression from limited early visual use through middle-layer alignment to late-layer performance plateaus, motivating selective late-layer merging. PlaM restores language-model parameters in plateau layers and consistently improves performance across architectures and benchmarks.

5.1 Visual Information Absorption

  • Across models and tasks, masking exposes three stages: early feature separation with limited gains, middle feature alignment with rapid improvement, and late feature degradation or absorption.The middle stage maps visual objects, attributes, and spatial relations into linguistic tokens, whereas late performance plateaus or slightly declines.Source: §5.1 · pp. 4–5

5.2 Plateau-Guided Late-Layer Merging

  • PlaM identifies the plateau onset from performance-versus-cut-layer curves and merges base-language-model parameters into all subsequent decoder layers.The merge start layer is selected near the elbow where rapid gains transition to a stable plateau; only self-attention Q/K/V/O projections are merged.Source: §5.2 · p. 5, Figure 2 · p. 5
  • PlaM consistently outperforms the base model across 5 MLLM architectures and 9 benchmarks, while early-, mid-, and full-layer merging do not achieve the best overall performance.On LLaVA-v1.5-7B, PlaM improves MMStar from 33.77 to 35.29, MMMU from 36.44 to 37.00, and RealWorldQA from 56.08 to 56.34.Source: §5.2 · p. 5, Table 1 · p. 6
  • Late-layer merging restores linguistic representations after multimodal fine-tuning while preserving earlier visual alignment and multimodal capability.Early or middle merging can disrupt incomplete cross-modal alignment, whereas late merging reintroduces linguistic abstractions and stable reasoning trajectories.Source: §5.2 · p. 5, LLaVA_v1.5_7B · p. 6

LLaVA_v1.5_7B

  • PlaM’s largest task-dependent gains occur on MMStar, MME, and MMMU, including MME_C improvements of +2.50 ∼+22.86 and MME_P improvements of +6.41 ∼+19.20.These benchmarks depend strongly on late-stage semantic decision making and visual evidence integration.Source: LLaVA_v1.5_7B · pp. 6–7
  • Gains are minimal on GQA (+0.04 ∼+0.38) and POPE (+0.19 ∼+0.97), whose limitations are described as lying mainly beyond late-layer semantic decisions.GQA relies more on early-to-middle compositional and spatial grounding, while POPE emphasizes evidence verification and calibration.Source: LLaVA_v1.5_7B · p. 7

6 Attention-based Mechanistic Analysis

The analysis measures layer-wise attention between token types and finds that PlaM increases visual-token use in late layers, with heatmaps showing more focused attention on task-relevant regions.

Measuring Vision Token Attention

  • Attention mass is computed over target and source token sets to track how different token types are emphasized across decoder layers.The method uses instruction tokens as targets during prefilling and response tokens during decoding, with vision tokens among the source choices.Source: §6 · p. 7

Layer-wise Attention Comparison

  • PlaM substantially increases attention mass allocated to vision tokens across architectures, especially in late layers.For Qwen2.5-VL-3B-Instruct, instruction-to-vision attention rises from near 0.1 to 0.3 in late layers, while LLaVA shows a similar increase.Source: §6 · pp. 7–8
  • Higher vision-token attention indicates greater reliance on visual information but does not by itself guarantee improved reasoning quality.The authors interpret the consistent shift toward vision-token use alongside benchmark improvements as mechanistic support for improved multimodal grounding.Source: §6 · p. 8

Case Study

  • In a ring-comparison case study, the original LLaVA shows diffuse attention and answers incorrectly, whereas PlaM focuses on ring contours and the interlocking region and answers correctly.The merged model concentrates multiple heads on both rings, particularly the decision-critical boundaries and interlocking area.Source: §6 · p. 8

7 Conclusion

PlaM is a training-free late-layer merging method that restores degraded semantic decision structure while preserving earlier multimodal alignment. Across five backbones and nine benchmarks, it consistently improves performance and produces more focused visual attention.

7 Conclusion

  • PlaM consistently improves performance over original models and alternative merging schemes across five backbones and nine benchmarks.Attention-mass and heatmap analyses indicate that these gains coincide with more focused attention on semantically relevant visual regions.Source: §7 · p. 8

Limitations

The method's merging position requires dataset-dependent tuning, and the simple merging strategy may not be optimal.

Limitations

  • The merging position k0 varies across datasets, so selecting its optimal configuration requires additional experiments and cost.The authors also note that more complex model-merging methods might yield better results.Source: Limitations · p. 9

A Experiment settings

The experiments evaluate PlaM across diverse multimodal benchmarks and five MLLM backbones, with supplementary tables and figures documenting masking results, hyperparameters, and attention heatmaps.

Evaluation Benchmarks

  • The evaluation suite spans perception, cognition, multilingual understanding, spatial and compositional reasoning, real-world visual reasoning, and hallucination detection.It includes MMStar, MMMU, MME, MMBench-EN/CN, GQA, RealWorldQA, SEED-Bench-2-Plus, and POPE.Source: A Experiment settings · p. 12

Hyperparameters

  • Table 2 reports the best PlaM hyperparameter settings across five backbones and nine benchmarks.The table is scoped to the benchmark and backbone combinations used for tuning and reporting.Source: Table 2 · p. 13

Vision-Language Models

Vision Token Masking

  • Tables 3–7 provide raw benchmark scores for five backbones under depth-controlled vision-token masking at different cut layers.These tables supply the score data used to generate the masking analysis in Figure 2.Source: Table 3 · p. 14, Table 4 · p. 15, Table 5 · p. 15

Related papers

Loading 2601.07645v1…