PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM2Vec-Gen: Generative Embeddings from Large Language Models

Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt, Nicolas Chapados, Marius Mosbach, Siva Reddy

arXiv:2603.10913v3cs.CL

TL;DR

Existing contrastive embedders reshape LLM representations into a new space and discard response semantics. LLM2VEC-GEN instead compresses the LLM’s potential response into embeddings with trainable tokens, teacher alignment, and reconstruction while freezing the backbone; it achieves state-of-the-art self-supervised MTEB performance and preserves safety, reasoning, and interpretability properties.

  • Problem

    Contrastive fine-tuning maps inputs and outputs into a new representational space, discarding the LLM’s response-space semantics.

  • Method

    LLM2VEC-GEN appends trainable special tokens and optimizes their response representation with unsupervised teacher alignment and reconstruction while keeping the LLM backbone frozen.

  • Results

    LLM2VEC-GEN achieves superior self-supervised MTEB performance, surpassing its embedding teacher by up to 8.8% while improving safety and reasoning retrieval.

  • Takeaways & Limitations

    Output-centric embeddings preserve capabilities acquired during LLM training, including safety and reasoning, and can be decoded into natural language.

  • Takeaways & Limitations

    Compared with LLM2VEC-GEN, HyDE requires explicit generation at inference time, incurring substantial computational overhead.

Abstract

from arXiv · show

Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.

1 Introduction

LLM2VEC-GEN replaces input-centric encoding with response-centric representations that preserve semantics acquired by the underlying LLM. It compresses potential responses into embeddings using unlabeled queries, and achieves strong benchmark, safety, and reasoning results.

  • Text embeddings support semantic search, textual similarity, and retrieval-augmented generation, while LLM-based encoders achieve strong embedding-benchmark performance.
  • Contrastive input-centric encoders represent input content in a new shared space, discarding the LLM’s response-space semantic structure.
  • LLM2VEC-GEN encodes the LLM’s potential response rather than the input, keeping embeddings closer to response-space semantics.
  • Response-centric embeddings can represent safe refusals for harmful queries and transfer reasoning capabilities emerging in the LLM’s responses.
  • LLM2VEC-GEN uses trainable suffix tokens, teacher alignment, and response reconstruction while keeping the LLM backbone frozen.
  • LLM2VEC-GEN substantially outperforms unsupervised and self-supervised baselines on MTEB, surpasses its embedding teacher by up to 8.8%, and improves safety and reasoning retrieval.

2 Related work

Prior work adapts decoder-only LLMs for embeddings through supervised or unsupervised input-centric training, while newer methods explore output-centric and latent-token representations. LLM2VEC-GEN differs by distilling potential responses without requiring explicit answer generation at inference.

  • Decoder-only LLM embedding approaches improve benchmark performance, but methods such as GritLM still use supervised contrastive learning with curated labeled datasets.
  • Unsupervised methods including LLM2Vec and Echo Embeddings reduce labeling needs but remain input-centric and can struggle with query-document lexical and conceptual gaps.
  • Latent-token methods such as xRAG and CLaRa compress documents into special tokens for efficient retrieval or joint retrieval-generation training.
  • HyDE encodes generated answers but requires multiple answers at inference, whereas later methods internalize generative foresight into embedding representations.
  • LLM2VEC-GEN combines teacher-based response-representation prediction with reconstruction, linking representation-space prediction to natural-language grounding.

3 LLM2VEC-GEN

LLM2VEC-GEN compresses an LLM-generated response into fixed-length embeddings using appended special tokens and two complementary objectives. The frozen LLM supplies the response and remains unchanged while lightweight components are trained.

  • LLM2VEC-GEN aims to represent the response an LLM would generate without generating that response during inference.
  • Training generates target responses from unlabeled queries and obtains their embeddings from an off-the-shelf teacher model.
  • Special compression tokens are added to the vocabulary, appended to queries, and used as an information bottleneck for response content.
  • The embedding-alignment objective matches projected, pooled compression-token representations to embeddings of the generated responses from an unsupervised encoder teacher.
  • The reconstruction objective feeds projected compression-token representations as soft prompts so the frozen LLM reconstructs the target response with next-token prediction.
  • The final loss combines alignment and reconstruction, while training updates only special tokens and two MLPs; inference uses one forward pass to produce the embedding.

4 Experiments

LLM2Vec-Gen is evaluated on general embedding, malicious retrieval, and reasoning-intensive retrieval across multiple model families and sizes. It achieves strong self-supervised MTEB performance while transferring safety and reasoning behavior into the embedding space.

  • Evaluation setup: LLM2Vec-Gen is evaluated on MTEB, AdvBench-IR, and BRIGHT across Qwen-3, Qwen-2.5, Llama-3.2, and Llama-3.1 models.MTEB measures general text embedding, AdvBench-IR measures malicious retrieval, and BRIGHT measures reasoning-intensive retrieval.
  • Evaluation setup: LLM2Vec-Gen uses unlabeled queries and an unsupervised encoder teacher, with generative instructions adapted to its output-centric embeddings.The training corpus contains 160K single-turn questions, while the LLM backbone remains frozen and only special tokens and projection layers are trained.
  • MTEB results: 61.9: Qwen-3-8B establishes a new self-supervised MTEB state of the art, with largest category gains in clustering, classification, and semantic textual similarity.The reported category improvements are +22.7% for clustering, +7.0% for classification, and +9.8% for semantic textual similarity.
  • Safety and reasoning: 22.6% reduction: LLM2Vec-Gen lowers unsafe retrieval for Qwen-3-1.7B from 46.7 to 36.2 on AdvBench-IR.Across all Qwen-3 sizes, LLM2Vec-Gen achieves lower, safer retrieval scores than teacher models by encoding refusals rather than malicious query intent.
  • Safety and reasoning: 7.7% to 35.6%: LLM2Vec-Gen improves BRIGHT reasoning-intensive retrieval over its LLM2Vec teacher across all model sizes.The gains increase with model scale, while the authors note that output-centric embeddings are particularly beneficial beyond surface-level lexical matching.

5 Ablations

Ablations show that both objectives and design choices shape LLM2VEC-GEN’s performance, with alignment driving embedding quality and reconstruction enabling decodability. Ten special tokens are a practical default, while same-family teachers and a frozen backbone offer favorable trade-offs.

  • Training objective: Embedding alignment drives embedding quality, while reconstruction grounds embeddings in the LLM’s language space and enables decoding.Removing alignment reduces performance from 67.9 to 43.1; reconstruction makes embeddings decodable into natural language.
  • Special tokens: 66.1 to 68.5 performance generally improves as the number of special tokens increases, but gains become marginal beyond 10 tokens.This supports the model’s default choice of 10 compression tokens.
  • Response generator: 67.3, 67.4, and 67.1 show that Tulu, Qwen-3-8B, and Gemini-3-flash responses do not improve performance over the default response generator.The authors hypothesize that in-distribution responses are easier for the frozen LLM to compress.
  • Embedding teacher: Same-backbone embedding teachers yield the best performance, whereas cross-family teachers slightly degrade results because their embedding spaces may be misaligned.The analysis also finds that output-centric embeddings generalize to the BGE teacher family.
  • Frozen backbone: 68.3 is achieved by LoRA with r = 8, but higher LoRA capacity reduces performance and requires separate embedding and generation weights.The frozen-backbone method instead supports using the same model for both purposes.

6 Interpretability of LLM2VEC-GEN embeddings

LLM2VEC-GEN embeddings can be decoded and inspected through Logit Lens, revealing semantic content oriented toward the model’s response. Qualitative analyses show refusal, instruction-following, and factual-response semantics rather than merely input-query meaning.

  • Decoding and analysis: LLM2VEC-GEN embeddings are decoded with next-token prediction, while Logit Lens projects compression-token representations into vocabulary space.The analysis reports meaningful tokens among the top five predictions for each compression token.
  • Qualitative results: Instruction-following examples map to response-related tokens such as “psychiatric” and “access,” indicating that embeddings capture response content.
  • Reconstruction objective: The reconstruction objective is necessary for interpretability: alignment-only training preserves embedding quality but produces nonsensical decoded outputs.Reconstruction grounds compression tokens in the LLM’s natural-language manifold.

7 Conclusion

The paper concludes that LLM2VEC-GEN creates response-space embeddings through frozen-backbone training with alignment and reconstruction objectives. It reports strong self-supervised performance and interpretable, decodable representations, while identifying teacher dependence and standard-retrieval limitations as boundaries.

  • 7 Conclusion: LLM2VEC-GEN encodes the LLM’s potential response rather than the input query, producing response-space embeddings through self-supervised training.
  • 7 Conclusion: Freezing the backbone and training special tokens with alignment and reconstruction objectives yields embeddings that are both high-performing and decodable into natural language.
  • 7 Conclusion: The output-centric paradigm is presented as preserving capabilities acquired during LLM training, including safety and reasoning, within the embedding space.
  • B Limitations: Embedding quality can inherit limitations from the unsupervised LLM2Vec teacher used to provide alignment targets.
  • B Limitations: Standard retrieval performance marginally declines for Qwen-3-4B, suggesting output-centric embeddings may miss lexical matching cues rewarded by that benchmark.
  • C Open frontiers: A full JEPA variant could remove the external teacher by using the same frozen LLM as response generator and target encoder, but its value remains an open empirical question.

D Implementation details

The implementation uses frozen pretrained LLMs, lightweight projection networks, special-token compression, and teacher-based training evaluated across MTEB-related settings. Experiments also vary instructions, response generators, and teacher models to assess robustness and performance.

  • Training details: Training uses AdamW, model-family-specific learning rates, 100 warmup steps, batch size 32, one epoch over 160K samples, and 512-token query and response limits.Responses exceeding the maximum sequence length are truncated.
  • Architecture: Each projection MLP has one layer matching the backbone hidden dimension, with output size set to the teacher embedding dimension when needed.
  • Efficiency: Only 13M parameters are trained for Qwen3-4B, and Qwen-3-8B training takes approximately 3.5 hours on two 80GB NVIDIA-H100 GPUs.Mixed-precision bfloat16 training is used to reduce memory consumption.
  • Evaluation: A 10-task MTEB-Lite subset preserves the full benchmark’s category distribution for ablations and analysis.The subset is identified in Table 6.
  • Instructions: Generative instructions improve performance, while LLM2VEC-GEN still outperforms the LLM2Vec teacher under identical embedding-style instructions.This supports attributing gains to output-centric embeddings rather than instruction wording alone.
  • Component analysis: Additional analyses vary response generators and encoder teachers within the Qwen-3 family, with performance comparisons reported in Figure 6.Figure 7 compares LLM2VEC-GEN with unsupervised LLM2Vec and BGE teachers across model sizes.

G Additional analysis on retrieval performance of LLM2VEC-GEN

LLM2VEC-GEN’s decoded retrieval responses remain relevant to query topics, while the larger Qwen-3-8B model more frequently identifies the correct linking target than Qwen-3-4B.

  • Qwen-3-4B generations capture broad query content but less often identify the correct linking target than Qwen-3-8B generations.Examples span universities, sports, and movies, including the University of Kansas, Colorado Buffaloes, and Snatch.

H Ablations on LLM2VEC-GEN components

Ablations show that LLM2VEC-GEN consistently outperforms the LLM2Vec baseline, with same-backbone teacher distillation performing best. The reconstruction objective is necessary for coherent decoded continuations and interpretability.

  • LLM2VEC-GEN outperforms the LLM2Vec baseline across all tested ablations on MTEB-Lite.
  • Same-backbone encoder-teacher distillation delivers the best overall performance among the evaluated teacher configurations.
  • LLM2VEC-GEN consistently surpasses both LLM2Vec and BGE unsupervised teachers across model sizes.This suggests the output-centric paradigm generalizes across unsupervised embedding teachers.
  • The reconstruction objective is necessary for coherent, interpretable decoded outputs compared with alignment-only training.Qualitative decoded samples support the same conclusion.
  • Over 81% of decoded LLM2VEC-GEN outputs form valid continuations of their input queries when trained with reconstruction and alignment objectives.Removing reconstruction collapses validity to near zero across the evaluated datasets.

I LatentLens qualitative analysis

LatentLens analysis finds meaningful contexts associated with LLM2VEC-GEN representations. Additional refusal analyses indicate that malicious queries remain distinguishable rather than collapsing into one generic refusal representation.

  • LatentLens qualitative analysis: LatentLens identifies two meaningful contexts associated with LLM2VEC-GEN special-token representations for Qwen-3-8B.The analysis uses an index built from generations of the same Qwen-3-8B model.
  • Refusal representation diversity: Refusal generations have lower mean pairwise cosine similarity than the original malicious queries, 0.377 versus 0.519.This result argues against collapse into a single generic refusal representation.
  • Refusal representation diversity: Output-centric embeddings retain query diversity, with mean pairwise cosine similarity of 0.668 across 520 AdvBench-IR queries.The comparison uses an output-centric instruction against an input-centric same-encoder baseline.

K Supervised performance of LLM2VEC-GEN

Supervised signals improve LLM2VEC-GEN over its self-supervised configuration, but the paper attributes an asymmetry to how supervised and unsupervised teachers represent responses. The authors therefore position the method mainly for settings where labeled paired data is scarce or unavailable.

  • Supervised variants: The experiments compare supervised teachers, hard-negative training, LoRA adaptation, and curated Echo query-document pairs.These configurations are evaluated against self-supervised training with an unsupervised teacher and unlabeled queries.
  • Supervised variants: Supervised teachers improve LLM2VEC-GEN by a large margin over the self-supervised configuration.The evaluated supervised variants add teacher supervision, hard negatives, LoRA training, or curated paired data.
  • Teacher representations: LLM2VEC-GEN’s alignment objective assumes that the teacher faithfully preserves the semantic content of the LLM response.The paper states that unsupervised encoder objectives encourage preservation of input information without external bias.
  • Scope and limitation: LLM2VEC-GEN is best suited to settings where labeled paired data is scarce or unavailable.When high-quality supervised encoders already exist, using them directly may be more effective than using them as teachers.
Loading 2603.10913v3…