Source-linked AI summary
Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders
Qingtao Xia, Jiahua Bao, Siyao Cheng, Jie Liu
TL;DR
PEFT layer placement remains sensitive and largely heuristic, motivating a pre-fine-tuning way to identify promising layers. The paper probes vision-encoder Q/K/V statistics and perturbation robustness, then compares these signals with layer-wise PEFT outcomes. Across the evaluated settings, more robust layers tend to yield better PEFT outcomes, often alongside larger norms and higher condition numbers.
Problem
PEFT performance is sensitive to where trainable modules are inserted, while layer selection remains largely heuristic.
Source: §I · p. 1Method
The study measures Q/K/V weight statistics and perturbation robustness in vision-encoder layers before tuning, then compares these indicators with layer-wise gains across five PEFT methods.
Source: §I · p. 1Results
Takeaways & Limitations
In limited-data regimes, robustness-guided selection of a small subset of layers can reduce trainable parameters while remaining competitive with broader tuning.
Source: §V · p. 6
Abstract
from arXiv · showhide
We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision--language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once---where layer selection often relies on heuristic rules---we focus on the vision encoder and directly evaluate the "adaptability'' of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (ii) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters.
I. INTRODUCTION
PEFT layer placement remains sensitive and largely heuristic. The introduction proposes probing vision-encoder weight statistics and perturbation robustness before tuning to identify layers likely to yield stronger gains.
I. INTRODUCTION
- PEFT insertion positions are sensitive, and choosing layers can substantially affect final performance while requiring costly trial and error.Source: §I · p. 1
- The study links Q/K/V weight statistics and perturbation robustness with layer-wise PEFT gains across five PEFT methods and two vision encoders.It reports three-trial medians and run-to-run variance around 0.3pp across both backbones.Source: §I · p. 1
- Layers with larger norms and higher condition numbers often coincide with greater perturbation robustness and better layer-wise PEFT gains.The introduction presents these patterns as pre-tuning signals for layer selection.Source: §I · p. 2
- Selective tuning of a small subset of layers can remain competitive with broader tuning, offering a practical pre-screening signal for PEFT layer selection.Source: §I · p. 2
II. RELATED WORK
The related-work section describes VLMs as systems that connect vision encoders to language models, commonly through lightweight connectors.
II. RELATED WORK
- VLMs connect a vision encoder to an LLM through a lightweight connector; cited examples include LLaVA, InternVL2, and Qwen2.5-VL.Source: A. VLMs · p. 2
B. PEFT with VLMs
The paper studies how perturbation robustness relates to layer-wise PEFT effectiveness in VLM vision encoders, a connection that prior work had left underexplored. It probes layers before tuning and evaluates the setup on a diverse training mixture and seven downstream benchmarks.
B. PEFT with VLMs
- Prior work studied perturbation and robustness in CNNs and Transformers, while their connection to layer-wise PEFT effectiveness in VLM vision encoders remained underexplored.Source: B. PEFT with VLMs · p. 2
III. DATASETS AND PERTURBATION SETTINGS
- The probing hypothesis is that robustness to controlled perturbations before fine-tuning correlates with later PEFT effectiveness; layers are probed without updating the base model.Source: §III · p. 2
A. Datasets Preparation
- The study samples 27,000 image-text pairs from 1.8 million pairs and evaluates PEFT on seven benchmarks, with MMStar as the primary comparison benchmark.The 48-dataset preparation spans capabilities including OCR, chart and table understanding, and scientific and math question answering.Source: A. Datasets Preparation · p. 2
B. Perturbation Settings
The section specifies controlled perturbations to QKV outputs and a score-drop probing procedure for prioritizing layers before PEFT.
B. Perturbation Settings
- The comparison covers LoRA, VeRA, IA3, LayerNorm-Tuning, and Partial Fine-Tuning using InternVL2-1B and Qwen2.5-VL-3B backbones.AddNoise, MulNoise, and ActNoise use α=0.05; PathNoise sweeps α=0.1–1.0 from a base α=0.5.Source: B. Perturbation Settings · pp. 2–3
- The five probes are AddNoise, MulNoise, LinearNoise, ActNoise, and SVD-based PathNoise; PathNoise combines original and low-rank bypass paths.Figure 2 summarizes the operators and their injection points.Source: B. Perturbation Settings · p. 2, Figure 2 · p. 3
- LinearNoise is omitted from reported tables after its trainable linear layer collapsed the VLM, producing 0.0 scores on all benchmarks in preliminary tests.Source: B. Perturbation Settings · p. 3
- Perturbations modify the target layer's QKV output, and robustness is measured by downstream-score drop, with smaller drops indicating greater robustness.Source: B. Perturbation Settings · p. 3
- For budget k, layers are ranked on held-out D by ActNoise and PathNoise drop scores, selecting the k smallest Rℓ for joint PEFT.Source: B. Perturbation Settings · p. 3
A. Distribution of Model Parameters
The section compares Q/K/V projection statistics across two VLM vision encoders and finds depth-related patterns in weight metrics, while condition-number fluctuations are concentrated earlier.
A. Distribution of Model Parameters
- The analysis covers Q/K/V projection matrices in the 24-layer InternVL2-1B and 32-layer Qwen2.5-VL-3B vision encoders.For coarse cross-backbone comparisons, the encoders are segmented into contiguous 8-layer blocks, with higher-index layers treated as deeper.Source: A. Distribution of Model Parameters · p. 3
- Six statistics are computed per layer and projection matrix: variance, L1, L2, Frobenius, and infinity norms, plus condition number.Condition number is computed on raw weights as the ratio of the largest to smallest singular values, without attention-time scaling or additional normalization.Source: A. Distribution of Model Parameters · p. 3
- Deeper layers generally have larger variance and norm-related metrics, while condition-number fluctuations are more concentrated in earlier layers.Metrics are min–max scaled across layers within each backbone for visualization and cross-layer comparison.Source: A. Distribution of Model Parameters · pp. 3–4
B. Segmented Analysis of Vision Encoder Layers
The block-level perturbation analysis evaluates predefined 8-layer blocks on MMStar and six additional benchmarks, and finds robustness varies by perturbation type and depth.
B. Segmented Analysis of Vision Encoder Layers
- Predefined 8-layer blocks are perturbed and evaluated on MMStar plus six benchmarks for InternVL2-1B and Qwen2.5-VL-3B.The block-level MMStar results are reported in Tables I(a) and II(a), respectively.Source: B. Segmented Analysis of Vision… · p. 4
- Across both backbones, ActNoise and PathNoise cause the largest degradation, while AddNoise and MulNoise are much milder.Block-level robustness is not monotonic with depth, motivating finer layer-wise analysis.Source: B. Segmented Analysis of Vision… · p. 4
- The block-level pattern motivates examining individual layers rather than assuming robustness follows encoder depth.This follows from robustness varying non-monotonically with depth across blocks.Source: B. Segmented Analysis of Vision… · p. 4
C. Layer-Wise Analysis of Vision Encoder
Layer-wise perturbation results reveal substantial robustness differences among layers within the same block.
C. Layer-Wise Analysis of Vision Encoder
- Several middle InternVL2-1B layers are consistently more fragile under ActNoise and PathNoise, supporting layer-level rather than contiguous-block PEFT selection.The within-block variation is reported in Table III.Source: C. Layer-Wise Analysis of Vision Encoder · p. 4
D. PEFT Analysis of Vision Encoder
Across PEFT methods, the strongest layer-level gains cluster in early and late vision-encoder layers, broadly overlapping with robust, high-statistics layers. Pooled tests report significant gains for selected layers, while same-budget heuristics show marginal or unstable outcomes.
D. PEFT Analysis of Vision Encoder
- Layer-wise PEFT is evaluated on InternVL2-1B with LoRA (r = 32/128/512), VeRA, IA3, LayerNorm-Tuning, and Partial-Tuning, and on Qwen2.5-VL-3B with LoRA (r = 32) plus the same baselines.Source: D. PEFT Analysis of Vision Encoder · p. 4
- The best PEFT gains cluster in early and late layers, broadly overlapping perturbation-robust, high-statistics layers.Figure 4 compares all-layer and per-layer tuning; Table IV reports MMStar results for LayerNorm-Tuning, LoRA (r = 32), and Partial-Tuning by layer and all-layer tuning.Source: D. PEFT Analysis of Vision Encoder · p. 4, Figure 4 · p. 5, Table IV · p. 5
- With k = 5 on InternVL2-1B, first-k LoRA reaches 43.70% versus 43.63% base; on Qwen2.5-VL-3B, middle-k LoRA reaches 55.24% versus 54.90% base.The passage characterizes budget-matched heuristics as yielding only marginal and unstable gains; attention-only and MLP-only tuning on Qwen2.5-VL-3B reach 54.70% and 54.50%, respectively.Source: D. PEFT Analysis of Vision Encoder · p. 4
E. Paired Significance Testing on Aggregated Benchmarks
- Pooled significance testing uses a two-sided McNemar test and paired percentile-bootstrap 95% confidence intervals with 27k resamples across six benchmarks.The pooled set comprises ChartQA, HallusionBench, MMBench, OCRVQA, RealWorldQA, and ScienceQA.Source: E. Paired Significance Testing on… · p. 4
- VeRA on layer 1 improves pooled accuracy by ∆= +2.14pp, and LoRA on layer 2 by ∆= +2.08pp; probing-selected layers yield statistically significant pooled gains.The VeRA 95% CI is [1.63, 2.64]pp (p = 1.2 × 10−16), and the LoRA 95% CI is [1.57, 2.58]pp (p = 8.236 × 10−16); gains are concentrated on HallusionBench and RealWorldQA.Source: E. Paired Significance Testing on… · p. 4
V. CONCLUSION
The study probes VLM vision-encoder layers before PEFT using Q/K/V weight statistics and perturbation robustness. Across two backbones, more perturbation-robust layers tend to yield better PEFT outcomes, suggesting a way to select a small, competitive subset in limited-data settings.
V. CONCLUSION
- Across two backbones, layers more robust to perturbations tend to yield better PEFT outcomes.The association often accompanies larger weight norms and higher condition numbers in the evaluated settings.Source: §V · p. 6
- The pre-PEFT probing examines VLM vision-encoder layers through Q/K/V weight statistics and perturbation tests.Source: §V · p. 6
- Robustness-guided selection of a small layer subset can reduce trainable parameters while remaining competitive with broader tuning in limited-data regimes.Source: §V · p. 6