PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Reasoning Models Enhance Embedding Models?

Wun Yu Chan, Shaojin Chen, Huihao Jing, Kwun Hang Lau, Elton Chun-Chai Li, Zihao Wang, Haoran Li, Yangqiu Song

arXiv:2601.21192v1cs.AIcs.CL

TL;DR

The paper asks whether RLVR-tuned reasoning backbones yield superior embedding representations, given their improved complex problem-solving. It introduces HRSA to analyze representation, geometry, and function, finding performance parity after contrastive learning and a pattern called Manifold Realignment. The results suggest RLVR traverses an existing semantic landscape rather than fundamentally rewriting it.

  • Problem

    The paper investigates whether RLVR-tuned reasoning models improve embedding geometry or retrieval when used as backbones for embedding models.

  • Method

    The study introduces HRSA, a hierarchical similarity framework spanning representation, geometry, and function, and systematically benchmarks matched RLVR-tuned and base backbones for text embeddings.

  • Results

    Across settings, RLVR largely preserves global geometry and linear readout while reorganizing local geometry, and subsequent contrastive learning produces strong base–reasoning realignment.

  • Takeaways & Limitations

    RLVR primarily optimizes trajectories through an existing semantic landscape rather than fundamentally redrawing the landscape itself.

  • Takeaways & Limitations

    The mechanisms governing persistent local reorganization and the onset and reversibility of coordinate-basis drift remain incompletely explained.

Abstract

from arXiv · show

State-of-the-art embedding models are increasingly derived from decoder-only Large Language Model (LLM) backbones adapted via contrastive learning. Given the emergence of reasoning models trained via Reinforcement Learning with Verifiable Rewards (RLVR), a natural question arises: do enhanced reasoning translate to superior semantic representations when these models serve as embedding initializations? Contrary to expectation, our evaluation on MTEB and BRIGHT reveals a **null effect**: embedding models initialized from RLVR-tuned backbones yield no consistent performance advantage over their base counterparts when subjected to identical training recipes. To unpack this paradox, we introduce **H**ierarchical **R**epresentation **S**imilarity **A**nalysis (HRSA), a framework that decomposes similarity across representation, geometry, and function levels. HRSA reveals that while RLVR induces irreversible latent manifold's local geometry reorganization and reversible coordinate basis drift, it preserves the global manifold geometry and linear readout. Consequently, subsequent contrastive learning drives strong alignment between base- and reasoning-initialized models, a phenomenon we term **Manifold Realignment**. Empirically, our findings suggest that unlike Supervised Fine-Tuning (SFT), RLVR optimizes trajectories within an existing semantic landscape rather than fundamentally restructuring the landscape itself.

1 Introduction

Embedding models built from decoder-only LLMs raise whether RLVR-enhanced reasoning produces better semantic representations. Controlled comparisons instead find no consistent advantage, while HRSA explains how RLVR changes and preserves different aspects of the latent manifold.

  • Decoder-only LLMs are increasingly adapted as embedding backbones to leverage stored semantics and world knowledge.
  • RLVR’s gains in complex reasoning motivate testing whether reasoning-tuned backbones create superior text embedding spaces.
  • Across MTEB(Multilingual, v2), MTEB(Code, v1), and BRIGHT, RLVR-initialized embedding models perform statistically identically to base-initialized models after contrastive learning.
  • HRSA diagnoses model similarity hierarchically across representation, geometry, and function levels.
  • RLVR preserves global manifold geometry and linear readout while irreversibly reshaping local geometry; prolonged RLVR also produces stronger coordinate-basis drift.
  • Contrastive learning subsequently produces strong alignment between base- and reasoning-initialized embedding models, a pattern termed Manifold Realignment.

2 Embedding Model Performances

The study compares matched base and RLVR-tuned backbones under identical embedding-model training and evaluates them across diverse benchmarks. RLVR-initialized models maintain performance parity with base-initialized models, unlike the larger shifts associated with SFT.

  • The study defines matched base and reasoning backbones, with reasoning models fine-tuned directly on base models; SFT pairs provide an explicit control.
  • The near-zero RLVR performance gaps indicate preserved semantic effectiveness, contrasting with larger shifts under SFT.
  • Embedding models remove the language-modeling head, pool final-layer hidden states, and use InfoNCE training with identical recipes within each pair.
  • The evaluation trains multiple matched pairs and tests retrieval, clustering, and semantic similarity across MTEB(Multilingual, v2), MTEB(Code, v1), and BRIGHT.
  • RLVR-initialized embedding models maintain performance parity with base-initialized models across all reported benchmarks.

3 The HRSA Framework: Dissecting Model Similarity

HRSA investigates why base- and reasoning-initialized embedding models look similar by comparing representations at representation and geometry levels, while respecting each level’s invariances. It combines feature correspondence, orthogonal alignment, global geometry, and local-neighborhood analyses to distinguish restructuring from realignment.

  • Common Setup and Notation: HRSA compares shared token-level representation matrices produced by paired base and reasoning models.Each row represents a token embedding on the latent manifold induced by mapped inputs.
  • Representation Level: Representation-level analysis treats coordinate bases as meaningful and tests whether corresponding features align without or with global orthogonal mixing.Dimension-wise correlation tests direct axis alignment, while Orthogonal Procrustes tests a single orthogonal mapping.
  • Representation Level: A near-diagonal or near-permutation Procrustes matrix indicates localized one-to-one feature correspondence, whereas a dense matrix indicates distributed feature mixing.The framework quantifies this distinction using inverse row entropy of O∗.
  • Interpretive Framework: The HRSA summary distinguishes SFT’s fundamental restructuring from RLVR’s trajectory optimization and contrastive learning’s manifold realignment.Linear CKA is used specifically as a global geometry descriptor, not as a direct proxy for task equivalence or linear separability.
  • Geometry Level: Geometry-level analysis ignores coordinate rotations and permutations, focusing instead on global manifold shape and local neighborhood preservation.Linear CKA measures global geometry, while k-NN overlap measures local geometry through nearest-neighbor agreement.
  • Motivation: Benchmark parity motivates HRSA’s structured investigation of where differences between base and reasoning models reside.The framework is introduced because aggregate benchmark similarity can conceal internal representational shifts.

4 Evaluation Setups

The evaluation compares base and reasoning backbones across LLM and downstream embedding adaptations using HRSA, with matched model pairs and shared analysis procedures.

  • Evaluation dimensions: HRSA compares base and reasoning models across LLM backbones and downstream embedding adaptations.The analysis also includes SFT-tuned reasoning models alongside RLVR-tuned models.
  • Datasets: The study uses Chain-of-Thought data for latent-manifold evaluation and AG’s News for linear-readout analysis.The Chain-of-Thought evaluation uses the hard-level subset.
  • Models: SFT compares Qwen2.5-Math-1.5B with DeepSeek-R1-Distill-Qwen-1.5B, while RLVR compares DeepSeek-R1-Distill-Qwen-1.5B with Nemotron-Research-Reasoning-Qwen-1.5B.The models are abbreviated as Qwen2.5, DS, and ProRL; embedding variants use the “-Emb” suffix.
  • HRSA procedure: HRSA collects complete hidden states at every layer before pooling to preserve per-layer, per-token representational structure.For each matched model pair, hidden-state matrices are collected across all layers.
  • Function-level evaluation: Cross-model probes use base or base-embedding representations for evaluation on both models, with smaller ∆ indicating stronger transfer.Results are reported for train, dev, and test splits.

5 Results

HRSA finds that RLVR preserves global geometry and functional readout more than SFT while reorganizing local geometry; contrastive learning then strongly realigns embedding models.

  • Representation-level results: RLVR retains higher dimension-wise correlations than SFT, while contrastive learning largely restores axis alignment between embedding models.The clearest deviation from diagonal structure occurs under prolonged RLVR.
  • Representation-level results: Prolonged RLVR yields a nearly permutative orthogonal map, which becomes strongly permutative after contrastive learning.SFT instead produces a dense orthogonal map implying greater feature mixing.
  • Geometry-level results: Linear CKA remains high under RLVR but drops under SFT, indicating preserved global manifold geometry under RLVR.After contrastive learning, embedding models move even closer in CKA.
  • Geometry-level results: RLVR preserves more local structure than SFT, but k-NN overlap remains below 1, indicating irreversible local-geometry reorganization.This local gap persists after contrastive learning pulls the embedding manifolds closer.
  • Function-level results: RLVR yields stronger cross-model probe transfer than SFT, while embedding pairs retain consistently high functional alignment after contrastive learning.The training trajectory shows rapid manifold realignment during Steps 0–200 before similarity stabilizes.

6 Related Works

Related work covers RLVR update behavior, decoder-only embedding models, and representation-similarity analysis; the paper combines these threads through HRSA.

  • RLVR and representations: Prior RLVR analyses suggest updates remain close to pretrained solutions and avoid large principal-subspace changes.These studies do not directly characterize the representation changes examined here.
  • Embedding models: Decoder-only LLMs are widely used as embedding backbones with bidirectional attention and contrastive training, but RLVR’s effect on embedding geometry remains unclear.This study directly tests whether RLVR-tuned reasoning models improve embedding geometry or retrieval.
  • Similarity analysis: Existing RSA and CKA work typically reports single-level alignment without organizing changes across abstraction levels or linking RLVR updates to geometry.HRSA addresses this gap by separating coordinate basis, manifold geometry, and readout-direction changes.

7 Discussion and Conclusion

The paper concludes that RLVR preserves global structure while reorganizing local geometry, and that contrastive learning realigns base- and reasoning-initialized embedding models.

  • Conclusion: HRSA and systematic benchmarking identify manifold realignment as a consistent pattern across RLVR-tuned and base backbones.The analysis diagnoses how training reshapes latent manifolds and compares resulting embedding models.
  • Conclusion: RLVR largely preserves global geometry and linear readout while irreversibly reorganizing local geometry; prolonged RLVR primarily produces coordinate drift.Contrastive learning corrects this drift and reinstates strong alignment.
  • Discussion: The findings support viewing RLVR as optimization within an existing semantic landscape rather than a rewrite of that landscape.RLVR mainly changes nearby-state relationships while leaving large-scale organization and linear readout mostly intact.
  • Future training design: A proposed training direction is SFT with geometry- and basis-aware regularization to constrain global distances or excessive coordinate drift.The paper presents this as an actionable hypothesis requiring follow-up testing.
  • Limitations and open questions: The mechanism behind persistent local reorganization and stable global geometry remains unresolved, motivating controlled interventions with HRSA.Suggested interventions include reward shaping, curriculum, and KL or entropy constraints.
  • Scope: The experiments focus on text embeddings, while the authors expect HRSA’s geometric signature to extend to vision and audio.The cross-modal claim is positioned as a hypothesis to verify across modalities and objectives.
  • Training details: Embedding models use InfoNCE to increase query-positive similarity while reducing similarity to negative passages.The objective uses in-batch passages, hard negatives, and cosine similarity.
  • Training details: The embedding pipeline removes the language-modeling head, mean-pools final-layer states, and enables bidirectional attention.The models are trained with full parameters rather than LoRA.

A.2 Training Data Statistics

This section describes the datasets and formalizes HRSA as a hierarchy of representation-, geometry-, and function-level analyses distinguished by invariance properties.

  • Training Data: The training data comprise 11 separately composed datasets, with three mined hard negatives per query and a 95% positive-score margin.Hard negatives were mined with Qwen3-Embedding-0.6B using positive-aware hard-negative mining.
  • Framework Extensibility: The HRSA framework is defined by invariance properties rather than specific metrics, allowing alternative valid measures for each analytical level.The framework’s extensibility is summarized in Table 8.
  • Representation Level: Representation-level analysis examines the latent manifold’s explicit coordinate basis and requires sensitivity to rotations, permutations, and general linear transformations.Dimension-Wise Correlation and related measures target coordinate-basis correspondence.
  • Geometry Level: Geometry-level analysis measures intrinsic point arrangement independently of coordinates, remaining invariant to orthogonal transformations and isotropic scaling but not anisotropic transformations.Linear CKA and cosine k-NN overlap are analyzed under these invariance requirements.
  • Linear CKA: Linear CKA is invariant to similarity transformations but generally changes under anisotropic linear transformations because they reweight representation-space principal components.The anisotropic transformation inserts a non-isotropic matrix into the HSIC computation.

B.2.2 k-NN Overlap

Cosine k-NN overlap preserves neighbor rankings under similarity transformations but can change under anisotropic transformations, while cross-model probes test fixed-readout compatibility.

  • k-NN Overlap: Cosine k-NN overlap is based on rankings of pairwise cosine similarities between embedding vectors.The score depends on the resulting top-k neighbor sets.
  • Similarity Transformations: Under orthogonal rotation and isotropic scaling, cosine similarities remain unchanged, so top-k neighbor sets and overlap scores are invariant.The transformed vectors produce exactly the same pairwise similarity rankings.
  • Anisotropic Transformations: Anisotropic transformations distort angles and can reverse neighbor rankings, changing the composition of k-nearest-neighbor sets and the overlap score.Vectors aligned with large versus small eigenvalues are pulled together or pushed apart angularly.
  • Cross-Model Linear Probes: Cross-model linear probes evaluate whether a fixed readout learned on one representation transfers to another model’s representation.The analysis concerns shared readout directions rather than geometric equivalence alone.
  • Evaluation Data: The additional analyses use a broad set of mathematical reasoning traces spanning easy, moderate, and hard difficulty levels.The CoT dataset draws from GSM8K, MATH-500, NuminaMath, and LiveMathBench.

C.3 Quality Control and Evaluation

The quality-control pipeline validates generated mathematical reasoning traces with a deterministic external LLM judge and retains traces labeled correct.

  • Verification: Generated reasoning traces are filtered through an LLM-as-a-Judge verification pipeline to exclude hallucinated or invalid trajectories.The pipeline evaluates whether each answer correctly solves its associated problem.
  • Evaluator Configuration: DeepSeek-V3.2-exp evaluates traces with greedy decoding and disabled internal thinking for deterministic outputs.The evaluator uses temperature 0 and enable_thinking set to false.
  • Validity Criterion: A trace is marked valid only when the evaluator response contains the token “1”; otherwise it is discarded.The resulting correctness label is binary.
  • Dataset Statistics: Table 11 reports total prompts, verified traces, and effective yield as Valid divided by Total.Acc. denotes the effective yield rate.

D Additional Results

Additional HRSA experiments extend the analysis across more model pairs, datasets, and embedding models, with reported embedding-model alignment values generally high for RLVR pairs.

  • Additional Model and Dataset Coverage: The additional experiments apply HRSA to more base–reasoning model pairs and use both CoT Activations and MMLU-Pro datasets.MMLU-Pro broadens analysis beyond the mathematics domain.
  • Dimension-Wise Correlation: Inverse row entropy quantifies axis-aligned correspondence, with higher values indicating more axis-aligned and lower values indicating more globally mixed features.Table 12 reports this measure for base versus reasoning models.
  • Embedding Results: The corresponding reported SFT embedding-model pair values are 0.7105 and 0.9164.These values are listed for the two SFT comparisons.
  • Embedding Results: 0.9794, 0.9978, 0.9814, and 0.9933 are the reported RLVR embedding-model pair values for the listed comparisons.These values correspond respectively to the four RLVR pairs in the reported results.
Loading 2601.21192v1…