PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Linguistic Reasoning Vectors Enhance Multimodal Reasoning Ability?

Ziyi Wang, Li Li, Aolin Zhou, Yankun Shen, Chonghan Liu, Shuxia Lin, Xu Yang

arXiv:2609.31140v1cs.AI

TL;DR

Multimodal scaling may weaken access to language-side reasoning in aligned VLMs, even when the base LLM retains usable reasoning. LIFT transfers reasoning-conditioned hidden-state differences from the base LLM into VLM language-side activations; across two VLMs, base-LLM vectors consistently outperform VLM-derived vectors.

  • Problem

    Multimodal alignment may alter language-side reasoning representations, leaving open whether representations formed in the base LLM remain usable for intervention in its corresponding VLM.

    Source: §1 · pp. 1–2

  • Method

    LIFT extracts answer-token hidden-state differences between matched reasoning-trace and no-trace paths, then injects the vectors into the target VLM without updating its backbone.

    Source: §1 · p. 2

  • Results

    Base-LLM Reasoning Vectors consistently improve performance and outperform VLM-derived vectors; average score rises from 66.6/67.8 to 68.3/69.0 on Qwen2.5-VL/InternVL2.5.

    Source: §1 · p. 2, §4.2.1 · p. 6

  • Takeaways & Limitations

    Reasoning-related directions formed in the base LLM can remain useful after multimodal scaling, and lightweight language-side interventions improve VLM reasoning without updating the backbone.

    Source: §5 · p. 9

    Experiments cover two VLM families, require support examples, and are sensitive to intervention-layer selection; language-side vectors cannot directly correct inaccurate visual perception or grounding.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Most Vision-Language Models (VLMs) are built by extending pretrained Large Language Models (LLMs) with visual modules and multimodal alignment. However, this multimodal scaling often degrades the language-side reasoning ability originally encoded in the base LLM. While the base LLM retains usable reasoning after scaling, the aligned VLM itself cannot reliably access this ability. Therefore, recovering the degraded reasoning capability in VLMs would benefit more from seeking help from the base LLM than from the VLM alone. Motivated by this, we propose LIFT (Language-side reasonIng Facilitation and Transfer), a lightweight vector-intervention method that transfers reasoning capability from the base LLM to the VLM without retraining the backbone. LIFT defines Reasoning Vectors as answer-token hidden-state differences between a Reasoner path with an explicit reasoning trace and a Solver path without it, and injects these vectors into language-side activations of the target VLM. LIFT further supports learnable vector adaptation while keeping the VLM backbone frozen. We evaluate LIFT on two VLMs across six reasoning benchmarks, comparing Reasoning Vectors extracted from the base LLM and from the aligned VLM under matched protocols. Results show that LLM-derived vectors consistently outperform VLM-derived vectors, confirming that the base LLM is a more effective source for recovering reasoning. LIFT partially recovers degraded reasoning through lightweight language-side interventions. Further analyses show that Reasoning Vectors influence intermediate reasoning behavior rather than merely altering final answers. The source code will be released soon.

1 Introduction

The introduction asks whether reasoning representations formed in a base LLM remain usable for intervention after multimodal alignment. LIFT extracts reasoning-conditioned answer-state shifts and transfers them to VLMs, with base-LLM vectors reported stronger than VLM-derived vectors.

1 Introduction

  • Multimodal scaling may change language-model behavior and internal representations, raising the question of whether base-LLM reasoning representations can still intervene on the corresponding VLM.The motivation includes cases where the VLM starts in the right direction but makes later reasoning errors, such as copying a number incorrectly or dropping a condition.Source: §1 · pp. 1–2
  • LIFT defines Reasoning Vectors as answer-token hidden-state differences between matched Reasoner paths with a reasoning trace and Solver paths without one, then injects them into the target VLM.The paths share the same question and final answer, so the difference approximates a reasoning-induced shift in answer representations.Source: §1 · p. 2
  • The study compares vectors extracted from the base LLM using text-only data, from the target VLM using text-only data, and from the target VLM using multimodal data.For VLM-MM, images are included as context, but vectors are computed from language-side answer representations.Source: §1 · p. 2
  • Across Qwen2.5-VL-7B-Instruct and InternVL2.5-8B, base-LLM Reasoning Vectors consistently improve VLM performance and outperform vectors extracted from the VLMs.The authors also report that learnable adaptation further improves performance, especially when initialized from base-LLM vectors.Source: §1 · pp. 2–3
  • The authors analyze source comparisons and layer-wise behavior, reporting that Reasoning Vectors influence the reasoning process rather than merely changing final answers and often help maintain alignment with the question target.The introduction presents LIFT as a lightweight intervention that transfers language-side reasoning vectors without updating model parameters.Source: §1 · pp. 2–3

2 Related Work

Prior work represents task-relevant knowledge with compact parameter or activation vectors and uses diverse training and inference strategies to improve VLM reasoning. LIFT instead probes whether reasoning-conditioned activation differences from a base LLM remain useful after multimodal alignment.

2 Related Work

  • Task and activation vectors summarize behavior-relevant information through parameter differences or activation differences induced by prompts, contexts, or demonstrations.Prior uses include parameter-efficient adaptation, model editing, in-context learning, and controllable activation steering.Source: §2 · p. 3
  • Prior VLM reasoning work includes inference-time search and verification, multimodal CoT or rationale-rich tuning, reinforcement-learning post-training, and cross-modal transfer methods.LIFT differs by comparing base-LLM and corresponding-VLM Reasoning Vectors under the same extraction and intervention protocol to study whether representations remain recoverable after scaling.Source: §2 · p. 3

3 Methodology

LIFT transfers reasoning-induced answer-representation shifts into selected language-model layers without updating the VLM backbone. It defines these shifts through Reasoner–Solver comparisons, supports several vector sources and learnable adaptation, and selects injection layers on held-out data.

3 Methodology

  • LIFT is a lightweight vector-intervention method that extracts Reasoning Vectors from Reasoner–Solver hidden-state differences and transfers them to the VLM without backbone retraining.The overview depicts per-layer difference averaging, optional vector-only adaptation, and inference-time injection into selected VLM layers.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Reasoning Vector Formulation

  • At each layer, LIFT approximates the reasoning-induced answer-representation shift by comparing aligned answer-token hidden states from Reasoner and Solver paths, then averaging differences across the answer span.The paths share the question and final answer; only the Reasoner path includes the reasoning trace.Source: §3.1 · p. 4

3.2 Reasoning Vector Sources and Extraction

  • LIFT compares base-LLM text-only vectors, target-VLM text-only vectors, and target-VLM vectors extracted with multimodal reasoning support sets.For each source, extraction uses samples that source model answers correctly; all vectors come from answer-token states and are injected into the language backbone, not the vision encoder.Source: §3.2 · pp. 4–5

3.3 Learnable Vector Adaptation

  • Learnable adaptation initializes a vector from an extracted Reasoning Vector and optimizes only that vector to bring intervened Solver representations closer to Reasoner representations.The VLM backbone stays frozen, and no separate scaling coefficient is used because optimization learns the vector magnitude.Source: §3.3 · p. 5

3.4 Layer Selection and Injection

  • LIFT injects static or learnable Reasoning Vectors into selected language-model layers during decoding, modifying language-token hidden states while keeping VLM parameters frozen.A held-out subset selects the injection layer using the same protocol across vector sources and adaptation settings.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate LIFT on two VLMs across six text-only and multimodal reasoning benchmarks. Base-LLM-derived vectors improve VLM performance over VLM-derived vectors, and learnable adaptation is strongest when initialized from the base-LLM vectors.

4.1 Setup and Implementation Details

  • The evaluation covers Qwen2.5-VL-7B-Instruct and InternVL2.5-8B across six text-only and multimodal reasoning benchmarks, using corresponding support sets for vector extraction.The suite includes GSM8K, CommonsenseQA, StrategyQA, MathVista, MathVision, and ScienceQA; LIFT variants retain the baseline zero-shot CoT setup and differ by injected vector and intervention layer.Source: §4 · p. 5, §4.1 · p. 6

4.2.1 Reasoning Vectors Transfer Effectively Across Modalities

  • 68.3/69.0 average accuracy for LLM-derived vectors, up from 66.6/67.8, on Qwen2.5-VL/InternVL2.5; they also outperform 4-shot CoT on both models.The comparison uses the same zero-shot CoT setup as the baseline, except for vector injection, and does not update model parameters.Source: Table 1 · p. 6, §4.2.1 · p. 6
  • Text-only extracted LLM-derived vectors improve all three multimodal benchmarks on both VLMs, including MathVista/MathVision scores rising from 68.5/25.0 to 69.1/26.6 and 62.8/22.7 to 64.0/25.0.MathVista and MathVision use GSM8K vectors; ScienceQA uses CommonsenseQA vectors.Source: §4.2.1 · p. 7

4.2.2 LLM-derived Vectors Provide More Reliable Interventions than VLM-derived Vectors

  • LLM-derived vectors outperform VLM-derived vectors on both VLMs, reaching 68.3/69.0 versus 67.5/68.3 average scores and scoring higher on most datasets.In representative MathVision cases, LLM-derived vectors repair intermediate errors in repeated-hit counting, gear-chain reasoning, and frog counting rather than merely changing final answers.Source: §4.2.2 · p. 7, Figure 3 · p. 7

4.2.3 LLM-derived Vectors Outperform VLM-MM Vectors

  • 57.0/60.6 versus 56.8/60.1 average scores show VLM-MM vectors outperform VLM-derived vectors, but LLM-derived vectors remain best at 57.4/60.9.The averages cover MathVision and ScienceQA on Qwen2.5-VL/InternVL2.5; VLM-MM uses support sets from the same multimodal benchmarks used for evaluation.Source: §4.2.3 · pp. 7–8, Table 2 · p. 8

4.2.4 Meaningful Initialization Matters for Learnable Vector Adaptation

  • LLM-learnable reaches the best average scores, 69.1/69.4 versus 68.3/69.0 for static LLM-derived vectors, while outperforming VLM-learnable at 67.2/66.6.Replacing LLM-derived initialization with random initialization lowers the average from 69.1/69.4 to 66.7/68.3; on InternVL2.5-8B, VLM-learnable falls below baseline.Source: §4.2.4 · p. 8, Table 3 · p. 8

4.3 Representation Alignment and Layer-wise Analysis

  • The GSM8K cross-layer cosine similarity shows stronger diagonal alignment for InternVL2.5-8B (0.866) than for Qwen2.5-VL-7B-Instruct (0.531), with Qwen also showing stronger middle-layer off-diagonal similarities.Similar-depth vectors align better, suggesting layer-wise organization is partly preserved but can be redistributed after multimodal scaling.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
  • Across six datasets, injection effects fluctuate by layer, but LLM-derived vectors generally outperform VLM-derived vectors across layers.This layer sensitivity indicates the effect depends on the VLM’s layer-specific organization rather than applying uniformly at all depths.Source: Figure 5 · p. 9, §4.3 · p. 9

5 Conclusion and Limitations

LIFT transfers language-side Reasoning Vectors from a base LLM to its corresponding VLM, improving reasoning without updating the VLM backbone. Its evaluation and language-side intervention impose limits on the current evidence and on the errors it can directly address.

Conclusion

  • Across text-only and multimodal reasoning benchmarks, LIFT improves VLM reasoning without updating the VLM backbone.The results suggest that reasoning-related directions formed in the base LLM can remain usable after multimodal scaling, even if alignment may partially weaken or reorganize them in the VLM.Source: §5 · p. 9

Limitations

  • Experiments cover only two VLM families, and generalization to more architectures, alignment procedures, and multimodal tasks remains to be evaluated.LIFT also requires support examples to extract vectors, and its effectiveness is sensitive to the selected intervention layer.Source: §5 · p. 9
  • Because LIFT operates on language-side hidden states, it cannot directly correct errors caused by inaccurate visual perception or grounding.The authors identify combining vector intervention with stronger visual grounding methods as a promising direction.Source: §5 · p. 10

A Derivation of Reasoning Vector

The derivation treats reasoning-trace tokens as an additional influence on answer-token attention. In the simplified setting, this influence becomes an additive shift relative to attention without the trace.

Attention with and without a reasoning trace

  • With a reasoning trace, an answer token depends on question, reasoning, and answer tokens; without the trace, it attends only to question and answer tokens.The passage names the question, trace, and answer as Q, T, and A, respectively.Source: A Derivation of Reasoning Vector · p. 15

Additive reasoning shift

  • The reasoning-conditioned attention output is expressed as standard attention plus an additive shift associated with the reasoning trace.The derivation describes the shift as a weighted difference between trace-only attention and standard attention; its coefficient is µ = ZT / Ztotal.Source: A Derivation of Reasoning Vector · p. 15

Reasoning Vector

  • LIFT operationalizes the additive shift as answer-token hidden-state differences between the Reasoner and Solver paths.This formulation approximates the reasoning-induced change in answer representations.Source: A Derivation of Reasoning Vector · p. 15

B.1 Datasets and Evaluation Protocols

The appendix documents evaluation protocols, prompting and generation settings, support-set construction, and learnable vector adaptation. Across these procedures, support data is used offline, while evaluation uses a consistent zero-shot CoT setup.

B.1 Datasets and Evaluation Protocols

  • The evaluation covers six text-only and multimodal reasoning benchmarks, uses held-out test splits, and compares methods under a shared zero-shot CoT protocol.The main comparison studies language-side Reasoning Vector sources as an analysis and intervention tool, rather than aiming to replace training-based adaptation or reinforcement learning.Source: §B.1 · pp. 15–16

B.2 Prompting and Generation Settings

  • Baseline and LIFT evaluations use the same prompts, decoding configuration, and answer parser; support examples and traces are excluded from evaluation prompts and test sets.Deterministic decoding uses beam size 1 with sampling disabled.Source: §B.2 · p. 16

B.3 Support Set Construction and Vector Extraction

  • Vector extraction uses 500 support examples by default and keeps only examples answered correctly by each source model, so source vectors can come from different subsets.Math tasks use GSM8K support examples, commonsense and strategy tasks use their corresponding training splits, and ScienceQA uses CommonsenseQA support examples.Source: §B.3 · p. 17

B.4 Learnable Vector Adaptation

  • Learnable adaptation freezes the VLM backbone and optimizes only injected vector parameters, initialized from extracted Reasoning Vectors unless otherwise specified.The optimization uses the offline support set and the alignment objective, without a language-model cross-entropy loss.Source: §B.4 · p. 17

B.5 Reproducibility Details

The reproducibility details specify the random seed, decoding approach, implementation precision, hardware, and broad runtime differences across benchmark types.

B.5 Reproducibility Details

  • Experiments use seed 42, deterministic decoding, PyTorch, and NVIDIA RTX 3090 or A6000 GPUs; text-only evaluations are faster than multimodal ones.The reported runtime distinction reflects 512-token limits for text-only benchmarks and 4096-token limits for multimodal benchmarks.Source: §B.5 · p. 18

C Statistical Uncertainty of Main Accuracy Differences

The section uses paired bootstrap resampling to assess uncertainty in accuracy differences, with macro-average deltas computed equally across model–benchmark pairs. Under this analysis, both vector sources improve over baseline, and LLM-derived vectors outperform VLM-derived vectors.

C Statistical Uncertainty of Main Accuracy Differences

D.1 Additional-seed Robustness Check

An additional support-set seed preserves the reported source ranking on both VLM backbones. A same-norm random-vector control does not reproduce the gains of LLM-derived Reasoning Vectors.

D.1 Additional-seed Robustness Check

  • With a different support-set and layer-selection seed, LLM-derived vectors still improve over baseline and exceed VLM-derived vectors on both backbones.The evaluation split remains fixed; Table 10 reports the additional-seed robustness check.Source: §D.1 · p. 19, Table 10 · p. 19

D.2 Same-norm Random-vector Control

  • The random-vector control uses the same selected layer, scaling coefficient, decoding setting, and evaluation protocol as the corresponding LIFT intervention.Its perturbation magnitude matches the LLM-derived Reasoning Vector.Source: §D.2 · p. 19
  • Random-vector accuracy was 66.4 versus 66.6 baseline and 68.3 LLM-derived on Qwen2.5-VL, and 66.1 versus 67.8 baseline and 69.0 LLM-derived on InternVL2.5.The control preserves perturbation magnitude while replacing the extracted vector direction with a random one.Source: §D.2 · p. 19, Table 11 · p. 19

E Additional Attention-level Injection Results

Attention-level injection applies head-wise Reasoning Vectors to attention outputs during inference while keeping the VLM backbone frozen. It improves average performance on both backbones, with LLM-derived vectors outperforming VLM-derived vectors.

E Additional Attention-level Injection Results

  • For each attention head, the vector is extracted from head outputs under Reasoner and Solver inputs.The intervention is applied to the corresponding attention-head outputs during inference.Source: E Additional Attention-level Injection… · p. 20
  • The attention-level intervention is applied to each generated language-token position, while the vision encoder remains unmodified and the VLM backbone stays frozen.The injection layer is selected on a held-out development subset of 100 training examples and fixed for evaluation.Source: E Additional Attention-level Injection… · p. 20
  • Attention-level vectors improve average performance over baseline on both VLM backbones, and LLM-derived vectors achieve higher average performance than VLM-derived vectors.Table 12 reports additional attention-level injection results.Source: E Additional Attention-level Injection… · p. 20, Table 12 · p. 20

F Additional Case Studies with Full Reasoning Traces

These case studies compare baseline, VLM-derived-vector, and LLM-derived-vector outputs under the same zero-shot CoT prompt to examine how Reasoning Vector interventions affect full reasoning traces and final answers. Across varied examples, LLM-derived vectors correct early reasoning decisions and keep subsequent calculations aligned with the question target, while VLM-derived vectors more often preserve baseline mistakes or drift to another target.

Case 7: Object-subtraction

Case 9: Kim’s sleep duration

  • The LLM-derived prediction is 452 minutes, preserving the overnight interval and subtracting only the 28 minutes Kim was awake.The trace counts 480 minutes from 10 p.m. to 6 a.m., then subtracts 5 minutes awake before 6 a.m. and 23 minutes sleepwalking.Source: F Additional Case Studies with Full… · pp. 26–27

Case 10: DMV rate-of-change

  • For the DMV question, the LLM-derived trace gives the signed rate of change as −3 minutes per month, matching the gold answer.It calculates (14 − 17) / 1, preserving the direction of change rather than reporting an unsigned difference.Source: F Additional Case Studies with Full… · p. 27

Overall observation

  • Across the cases, LLM-derived vectors correct early reasoning decisions more reliably, while VLM-derived vectors more often preserve mistakes or drift to a different target.The examples indicate that interventions change the intermediate reasoning process, not merely the final answer.Source: F Additional Case Studies with Full… · p. 27

Related papers

Loading 2609.31140v1…