Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval

1 Introduction

• Llama2Vec adapts LLMs for dense retrieval by shifting text embeddings from local next-token semantics toward global sentence-level semantics.Dense retrieval requires global query-document representations, whereas standard LLM pretraining emphasizes predicting the next token.Source: §1 · pp. 1–2
• EBAE prompts an inductive embedding to predict the input sentence, while EBAR prompts a deductive embedding to predict the next sentence.The two tasks jointly support sentence reconstruction and next-sentence prediction using text embeddings.Source: §1 · p. 2, Figure 1 · p. 2
• Llama2Vec supports different semantic relationships by using prompts for correlation relationships such as QA and paraphrasing relationships.The approach is designed to handle diversified semantic relationships through inductive and deductive embeddings.Source: §1 · p. 2
• Llama2Vec is lightweight because its predictions use the LLM text embedding as input and target-sentence vocabularies as multi-classification labels.The method can be realized within an existing auto-regression pipeline.Source: §1 · p. 2
• LLaMA-2-7B adapted on Wikipedia substantially improves downstream retrieval and establishes new state-of-the-art results across MSMARCO and BEIR.The reported gains cover supervised passage and document retrieval on MSMARCO and zero-shot retrieval on BEIR.Source: §1 · p. 2
• The paper presents Llama2Vec as an unsupervised method that improves LLM-based dense retrieval in both supervised and zero-shot settings.The authors identify the method as a first exploration of adapting LLMs for dense retrieval and emphasize its cost-effective pretext tasks.Source: §1 · p. 2

Dense retrieval

• Pretrained language models such as BERT, RoBERTa, and T5 have been widely used to encode queries and documents for dense retrieval.Their large-scale architectures and pretraining provide fine-grained semantic representations.Source: §2 · p. 3
• Dense retrieval quality also depends on contrastive-learning negatives, knowledge distillation, model scale, training scale, and pretraining optimization.These factors affect embedding discriminativeness, accuracy, and generality.Source: §2 · p. 3

Dense retrieval with LLM

• LLMs offer semantic understanding, versatility, instruction following, and extended context lengths that may benefit dense retrieval.These properties motivate their use for complex query-document relationships, multitask retrieval, and long-document retrieval.Source: §2 · p. 3
• Existing LLM-based retrieval methods largely use LLMs directly, despite a mismatch between language modeling and text embedding.The related-work discussion identifies proper LLM adaptation for dense retrieval as an open problem.Source: §2 · p. 3

3.1 Preliminary

• Llama2Vec uses the final ⟨\s⟩ or [EOS] embedding because decoder-only LLMs expose global context primarily at the sequence end.This differs from common BERT-style [CLS] or mean-pooling integration.Source: §3.1 · p. 3

3.2 Unsupervised Adaptation

• The adaptation targets the gap between LLM embeddings focused on next-token prediction and dense retrieval embeddings representing global input semantics.The objective is to transform the LLM text embedding into a global-context representation.Source: §3.2 · p. 4
• The two objectives target induction for the input text and deduction for correlated text, covering correlation and paraphrasing retrieval relationships.The paper frames these as the two embedding capabilities needed for question-answering and natural-language-inference-style retrieval.Source: §3.2 · p. 4
• EBAE inductively represents the input sentence itself, while EBAR deductively represents its next sentence through separate SELF and NEXT prompts.SELF uses “The input sentence is:”, whereas NEXT uses “The next sentence is:”.Source: §3.2 · p. 4
• Llama2Vec jointly computes EBAE and EBAR embeddings with a merged prompt and mutually invisible SELF and NEXT segments, saving almost 50% of computation.The first and second ⟨\s⟩ outputs provide the two embeddings while avoiding repeated processing of the input text.Source: §3.2 · p. 4
• Training projects each text embedding into vocabulary space and predicts all tokens in its target context, after which the adapted LLM is contrastively fine-tuned for retrieval.Prompting can vary by downstream relation: NEXT-SELF for correlation, SELF-SELF for long paraphrases, and NEXT-NEXT for short paraphrases.Source: §3.2 · pp. 4–5

4.2 Supervised Performance

• 43.1 MRR@10 and 99.5 Recall@1000 on MS MARCO passage retrieval establish state-of-the-art results, with nearly +4% MRR@10 over the best hard-negative PLM baselines.Llama2Vec also improves over RepLLaMA, which uses the same backbone without adaptation.Source: §4.2 · p. 6
• +5.7% point in MRR@100 over previous BERT-based methods and +2.3% MRR@100 over RepLLaMA on MS MARCO document retrieval.The authors attribute the document-level advantage partly to LLM context length, which allows full-document encoding instead of chunking.Source: §4.2 · pp. 6–7

4.3 Zero-shot Performance

• 56.4 average NDCG@10 on BEIR establishes a new zero-shot state-of-the-art, with Llama2Vec outperforming RepLLaMA on 12 of 14 datasets.It also outperforms BM25 on 13 of 14 datasets and exceeds BM25’s average by 31% relatively.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Technical Factors

• Adaptation increases lexical similarity between query and answer vocabulary distributions, and this improvement is linked to dense-retrieval performance.The analysis compares adapted and initial LLaMA-2 representations using BM25 over predicted top-N vocabulary tokens.Source: §4.4 · p. 8, Table 6 · p. 8
• Adaptive prompts substantially affect BEIR performance: NEXT-SELF suits correlation, SELF-SELF long paraphrasing, and NEXT-NEXT short paraphrasing.The majority of BEIR tasks concern correlation, where NEXT-SELF works most effectively.Source: §4.4 · p. 8, Table 5 · p. 8
• Reducing embeddings from 4096 to 768 causes notable performance loss with linear projection or distillation, while sparsification preserves retrieval performance more effectively.These findings motivate lightweight processing for LLM-based retrievers.Source: §4.4 · pp. 8–9, Table 7 · p. 8

• Llama2Vec uses EBAE and EBAR to adapt LLM text embeddings into strong global-context representations for dense retrieval.The tasks reconstruct the input sentence and predict the following sentence using text embeddings.Source: §5 · p. 9
• The adapted LLM supports semantic matching for both correlation and paraphrasing relationships through suitable prompts.Source: §5 · p. 9
• Llama2Vec achieves new state-of-the-art performances in both supervised and zero-shot retrieval evaluations.The results indicate substantial improvements in retrieval accuracy and generalization capability.Source: §5 · p. 9

• The method is evaluated only on a 7B model, so its impact on larger LLMs remains unexplored.Source: §6 · p. 9
• The current model targets English-centric scenarios, motivating extensions to other languages.Source: §6 · p. 9
• Efficient running costs remain an open concern for large-scale embedding models.Source: §6 · p. 9

• Llama2Vec inherits potential biases, toxicity, and other problems present in its underlying LLaMA-2 model.Source: §7 · p. 9
• The authors do not recommend using Llama2Vec for retrieval in sensitive contexts.Source: §7 · p. 9
• Training-data influence may produce biased or discriminatory retrieval results.Source: §7 · p. 9