PaperlayerView exact revision on arXiv

Source-linked AI summary

Repetition Improves Language Model Embeddings

Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, Aditi Raghunathan

arXiv:2402.15449v2cs.CLcs.LG

TL;DR

Autoregressive language models have been difficult to use as strong text encoders because causal attention prevents tokens from accessing the full input, while existing solutions often alter the architecture or require extra training. Echo embeddings repeat the input and pool the repeated tokens, yielding around 5% higher zero-shot performance than classical embeddings and nearly matching bidirectionally converted models, while remaining competitive after supervised fine-tuning at matched compute.

  • Problem

    Autoregressive language models are limited as text encoders because strong embedding approaches often require bidirectional architecture changes or additional training.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    Echo embeddings prompt the model to repeat or reconstruct the input, then pool representations from the repeated tokens without changing the architecture.

    Source: §2 · p. 2, §3.2 · p. 4

  • Results

    Around 5% higher zero-shot performance than classical embeddings is reported, with echo embeddings nearly matching bidirectionally converted models trained with additional MLM.

    Source: §1 · p. 2, §4.3 · p. 6

  • Takeaways & Limitations

    Echo embeddings offer a simple route toward using one autoregressive architecture for both generation and embedding tasks.

    Source: §1 · p. 2, §6 · p. 10

    Repeating the input approximately doubles compute, although compute-matched experiments still report strong performance.

    Source: §1 · p. 2, §4.6 · p. 9

Abstract

from arXiv · show

Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have largely had the requirement to modify the LM architecture to be bidirectional. We challenge this premise by introducing "echo embeddings" which converts autoregressive LMs into high quality text embedding models without changing the architecture or requiring fine-tuning. By repeating the input and extracting embeddings from the repeated tokens -- which have access to all original tokens -- echo embeddings improve over classical LM embeddings by over 5% in zero-shot settings. Our zero-shot embeddings nearly match those obtained by bidirectionally-converted LMs that undergo additional masked-language modeling training. Echo embeddings are also compatible with supervised fine-tuning, matching or outperforming bidirectionally-converted LMs in an apples-to-apples comparison, even with an identical compute budget during training and inference. Overall, repetition is a simple and effective strategy to circumvent the need for bidirectional attention in embedding models, paving the way towards a unified architecture for all NLP tasks.

1 INTRODUCTION

The paper asks whether bidirectional attention is necessary for high-quality embeddings from autoregressive language models. It introduces echo embeddings, which repeat the input and extract representations from the repeated tokens without architectural changes or fine-tuning.

Method

  • Echo embeddings repeat the input and pool representations from its second occurrence, whose tokens can access the entire original input despite causal attention.The model is prompted to repeat, rephrase, fix, or reconstruct the input, encouraging the repeated tokens to encode information from the first occurrence.Source: §1 · p. 2

Zero-shot results

  • Around 5% higher zero-shot performance than classical embeddings is reported, nearly matching bidirectionally cast models with additional unsupervised MLM training.Echo embeddings also outperform PromptEOL by a significant margin and are less sensitive to prompt wording.Source: §1 · p. 2

Fine-tuned results

  • Echo embeddings slightly outperform bidirectionally cast classical embeddings after supervised fine-tuning, including when both methods receive identical compute budgets.The authors speculate that avoiding architectural changes better preserves and leverages large-scale pretraining.Source: §1 · p. 2

Scope and cost

  • The approach supports zero-shot use, unified text generation and embedding models, and simple application, but repeating inputs approximately doubles compute.The experiments therefore report compute-matched inference and training results, where gains become slightly smaller but remain strong.Source: §1 · p. 2

2 PRELIMINARIES

The paper defines text embeddings as vectors whose similarity reflects sentence semantics and describes how autoregressive language-model activations are pooled into sentence representations. Classical embeddings pass the sentence once and pool contextualized token representations, commonly by mean or last-token pooling.

Embedding objective

  • Text embeddings map a sentence x to a vector ϕ(x) ∈ R^d so semantic similarity is reflected by similarity between vectors.The paper uses cosine similarity as the practical similarity metric.Source: §2 · p. 2

Language-model embeddings

  • Autoregressive language models use causal attention and a next-token objective, while embeddings are extracted from final-hidden-layer token activations.Each input token x_j has a contextualized representation ϕ_j(x).Source: §2 · p. 2

Pooling

  • Mean pooling averages contextualized token embeddings over selected indices, whereas last-token pooling uses the representation of the sequence’s final token.These are the two pooling strategies emphasized in the paper.Source: §2 · p. 2

Classical embeddings

  • Classical embeddings pass a sentence through the model once and pool token representations, optionally after a task-specific prompt.The paper uses “classical embeddings” for this single-pass extraction procedure.Source: §2 · p. 3

3 ECHO EMBEDDINGS FROM LLMS

The paper first demonstrates why causal attention undermines classical embeddings, then introduces echo embeddings to recover whole-input information without changing the architecture. On the synthetic task, echo embeddings work across both early- and late-discriminatory structures and show that repeated-token representations encode later information.

3.1 FAILURE OF CLASSICAL EMBEDDINGS WITH CAUSAL ATTENTION

  • Classical mean and last-token embeddings perform no better than random on the mixed synthetic dataset containing both early- and late-discriminatory structures.The dataset pairs each query with similar and dissimilar sentences whose informative parts vary between early and later positions.Source: §3.1 · p. 3
  • Last-token pooling misses early discriminatory information, while mean pooling dilutes later discriminatory information because early causal representations cannot attend to future tokens.These complementary failures prevent classical embeddings from handling both structures simultaneously.Source: §3.1 · p. 4

3.2 ECHO EMBEDDINGS: A GENERAL FRAMEWORK

  • Echo embeddings prompt an autoregressive model to repeat or reconstruct the input, feed the sentence twice, and pool contextualized embeddings from the second occurrence.The repeated tokens can attend to the entire first occurrence; prompt wording and task can vary.Source: §3.2 · p. 4
  • Echo embeddings with mean pooling outperform classical embeddings on both synthetic structures, including cases where discriminatory information appears only in later tokens.Pooling only the first A portion still distinguishes similar from dissimilar sentences, showing that those repeated-token representations encode later information.Source: §3.2 · p. 4
  • The synthetic study motivates testing echo embeddings in real-world settings after showing that repetition addresses the causal-attention failure mode without bidirectional attention.Table 1 is identified as the main zero-shot comparison on MTEB across 56 datasets, using Mistral-7B-Instruct-v0.1 for autoregressive embeddings.Source: §3.2 · p. 5, Table 1 · p. 5

4 EXPERIMENTS

Experiments evaluate echo embeddings across zero-shot and fine-tuned settings on MTEB and MTEB-MINI, varying prompts, pooling, models, attention, and compute. Echo embeddings generally outperform classical alternatives, including under matched compute, while their advantage narrows at very small token budgets.

4.2 EVALUATION OF ZERO-SHOT EMBEDDINGS

  • Mean pooling is required for strong zero-shot embeddings, whereas last-token pooling performs better in the fine-tuning setting.The zero-shot pooling choice is supported by both the pooling ablation and the reported experimental setup.Source: §4.2 · p. 6, §4.3 · p. 7, §4.4 · p. 8

4.3 ZERO-SHOT RESULTS

  • Echo embeddings outperform classical embeddings in zero-shot MTEB evaluations, including comparisons with causal and bidirectional attention baselines.The experiments span multiple models and scales, with MTEB-MINI and MTEB used for evaluation.Source: §4.3 · pp. 6–7
  • Echo embeddings outperform PromptEOL by 5% on average across tested MTEB datasets and are less sensitive to prompt wording and formatting.The prompt-sensitivity comparison is reported on MTEB-MINI across different prompts and formats.Source: Figure 3 · p. 6, §4.3 · p. 7
  • Zero-shot echo embeddings perform similarly to LLM2Vec-unsupervised without requiring its additional masked-language-modeling fine-tuning.LLM2Vec-unsupervised is explicitly not zero-shot because it requires extra fine-tuning.Source: §4.2 · p. 6, §4.3 · p. 7

4.5 FINE-TUNING RESULTS

  • Fine-tuned echo embeddings outperform classical embeddings with both causal and bidirectional architectures, including when training and inference compute are matched.They also match approximately the performance of LLM2Vec and perform similarly to a comparable-data GritLM variant without bidirectional attention.Source: Table 5 · p. 7, §4.5 · p. 9

4.6 ADDRESSING THE INFERENCE-TIME COST OF ECHO EMBEDDINGS

  • Echo embeddings require approximately twice the naive compute because the input is repeated, so experiments report compute-matched settings by shortening inputs or reducing training steps.For fine-tuning, compute matching uses half as many training steps and reduced inference input length.Source: §4.2 · p. 6, §4.4 · p. 8, §4.6 · p. 9

4.7 THE ROLE OF COMPUTE

  • Echo embeddings outperform classical bidirectional Mistral-7B embeddings above roughly 64 tokens in zero-shot evaluation, while classical bidirectional embeddings win below that budget.Across all compute budgets, echo embeddings outperform causal classical embeddings; the standard MTEB budget is 512 tokens.Source: §4.7 · pp. 9–10, Figure 4 · p. 9
  • In fine-tuning, echo embeddings outperform classical embeddings above 128 tokens, a threshold below the standard 512-token MTEB budget.At very small computational budgets, the classical approach performs better.Source: Figure 5 · p. 9, §4.7 · p. 10

5 RELATED WORK

Prior sentence-embedding work spans compositional, pretrained, contrastive, prompt-based, and autoregressive approaches. Recent autoregressive methods improve embeddings through additional training, synthetic data, summarization prompts, or architectural changes, motivating echo embeddings as a simpler alternative.

  • Sentence-embedding research progressed from word-vector composition to pretrained models trained with triplet or contrastive objectives.Source: §5 · p. 10
  • To the authors’ knowledge, Jiang et al. (2023b) was the only prior work constructing zero-shot embeddings for autoregressive language models.Source: §5 · p. 10
  • Recent methods add prompts, synthetic data, auxiliary objectives, or bidirectional attention to improve autoregressive language-model embeddings.These approaches include summarization prompts, synthetic training data, generation-preservation objectives, and bidirectional architectural modifications.Source: §5 · p. 10
  • Repetition had also been reported as useful for improving autoregressive reasoning and recurrent-model recall, providing related precedent for the paper’s strategy.Source: §5 · p. 10

6 CONCLUSION

The paper concludes that bidirectional architectures are not essential for high-quality embeddings. Echo embeddings use an autoregressive model as an autoencoder to encode bidirectional information through an inference-time task choice.

  • Echo embeddings show that bidirectional architectures are not crucial for high-quality embedding models.Source: §6 · p. 10
  • Echo embeddings prompt an autoregressive model to act as an autoencoder, enabling it to encode bidirectional information.Source: §6 · p. 10
  • The results link classical autoregressive embedding failures to an inability to encode bidirectional information.Source: §6 · p. 10
  • Choosing the language-model task at inference time is presented as a promising direction for improving embeddings.Source: §6 · p. 10

A GENERAL INFORMATION FOR REPRODUCIBILITY

This reproducibility section states that it contains supplementary information not specific to any particular paper setting.

A.1 MASSIVE TEXT EMBEDDING BENCHMARK

The paper documents its benchmark scope, model identifiers, and societal considerations. MTEB-MINI covers broad embedding tasks with fewer datasets, while the authors acknowledge both potential benefits and misuse risks.

A.1 MASSIVE TEXT EMBEDDING BENCHMARK

  • MTEB contains 56 datasets across classification, clustering, pair classification, reranking, retrieval, sentence similarity, and summarization.Source: §A.1 · p. 14

A.2 BASE MODEL HUGGINGFACE IDS

  • The paper lists Mistral 7B, LLaMA 7B, S-LLaMA 1.3B, and LLaMA 13B instruction-tuned model identifiers.Source: §A.2 · pp. 14–15

B SOCIETAL CONSIDERATIONS

B.1 IMPACT

  • Higher-quality embeddings may support more accurate retrievals and downstream tasks, including retrieval-augmented generation, question answering, and summarization.Source: §B.1 · p. 15
  • The authors acknowledge that the work could also improve fake-news or misinformation generation and that the released aggregated data may contain harmful content.Source: §B.1 · p. 15

C ECHO EMBEDDINGS: ADDITIONAL INFORMATION

This section specifies how embeddings are pooled and compared, and details echo-prompt structures that repeat or reconstruct the input so the second occurrence can be embedded.

C ECHO EMBEDDINGS: ADDITIONAL INFORMATION

  • Cosine similarity measures semantic similarity between sentence embeddings, while Euclidean distance is used for clustering datasets in MTEB.Cosine similarity is also used in the experiments and as the training similarity function.Source: C ECHO EMBEDDINGS: ADDITIONAL… · p. 15
  • Mean pooling averages contextualized token embeddings over selected token positions, and the experiments use the pooled representation for sentence embeddings.For the Section 3 echo setup, the pooled embedding is the mean token embedding associated with the rewritten sentence.Source: C ECHO EMBEDDINGS: ADDITIONAL… · p. 15
  • Echo prompts repeat or reconstruct an input, with embeddings taken over the second occurrence of the text.The paper treats this repeated-input structure as the important component, rather than exact prompt wording.Source: C ECHO EMBEDDINGS: ADDITIONAL… · pp. 15–16
  • The additional examples illustrate echo structures that fill blanks, rewrite sentences, or pair an original sentence S with a reconstructed sentence S′.These structures are generated from synthetic sentence triplets designed to share an initial segment while differing later.Source: C ECHO EMBEDDINGS: ADDITIONAL… · p. 16

D ADDITIONAL ZERO-SHOT RESULTS

Prompt randomization evaluates whether embedding performance depends on exact wording, while toy and STS analyses examine whether echo embeddings address classical embeddings’ early-token bias.

D ADDITIONAL ZERO-SHOT RESULTS

  • Echo prompts use instructions such as repeat, rewrite, rephrase, or fill in the blank, with the input appearing twice and embeddings taken from the second occurrence.Classical prompts contain one input, while PromptEOL prompts ask for a one-word summary.Source: D ADDITIONAL ZERO-SHOT RESULTS · p. 21
  • PromptEOL is highly sensitive to exact prompt wording, whereas classical and echo embeddings are not particularly sensitive; mean-token pooling also outperforms last-token pooling by a large factor.The sensitivity analysis uses randomized wording, punctuation, and capitalization across selected MTEB tasks.Source: D ADDITIONAL ZERO-SHOT RESULTS · pp. 21, 23

E.1 QUALITATIVE COMPARISON OF CLASSICAL AND ECHO EMBEDDINGS.

  • The qualitative STS examples show that echo embeddings improve most when sentence pairs share superficial similarity in their first few tokens.The pair with the greatest improvement has this early-token similarity, whereas the pair with the least improvement does not.Source: E TOY VERIFICATION · p. 23, §E.1 · p. 23

D ADDITIONAL ZERO-SHOT RESULTS

  • The study measures prompting sensitivity for classical, repetition, and PromptEOL across different models and tested datasets, finding PromptEOL highly sensitive while classical and echo were not particularly sensitive.Mean token pooling outperformed last token pooling by a large factor.Source: D ADDITIONAL ZERO-SHOT RESULTS · p. 23

E.2 QUANTITATIVE EVALUATION OF THE FAILURE MODE.

The quantitative evaluation tests whether classical embeddings overestimate similarity for sentence pairs that resemble each other early, and whether echo embeddings avoid this failure mode.

E.2 QUANTITATIVE EVALUATION OF THE FAILURE MODE.

  • The evaluation selects the top 10% of sentence pairs by cosine similarity in either the first or second half, then compares predicted and ground-truth rank differences.Positive rank differences indicate overestimated similarity, while negative differences indicate underestimation.Source: §E.2 · p. 25
  • Classical embeddings generally overestimate similarity more for sentence pairs similar in their first half than for pairs similar in their second half.This pattern appears across the STS datasets and supports the proposed early-token failure mode.Source: §E.2 · p. 25
  • Echo embeddings generally show no difference between estimation-error distributions for first-half-similar and second-half-similar sentence pairs.The result indicates recovery from the particular early-token failure mode examined here.Source: §E.2 · p. 25
  • BIOSSES and STS22 are counterexamples: BIOSSES has few examples and may reflect noise, while STS22 shows identical distributions for both embedding types.The authors nevertheless report that the broader trend holds for every other dataset.Source: §E.2 · p. 25

F ADDITIONAL FINETUNING RESULTS

Additional finetuning analyses compare echo, classical, and bidirectional embeddings across MTEB datasets, pooling strategies, ablations, and training time.

F ADDITIONAL FINETUNING RESULTS

  • The finetuning evaluation uses approximately 1.5M examples drawn from retrieval, question-answering, natural-language-inference, and duplicate-question datasets.Training takes approximately two days on four A100 GPUs, while MTEB evaluation takes approximately two days on eight A100s in parallel.Source: F ADDITIONAL FINETUNING RESULTS · p. 32
  • Last-token pooling with a trainable end-of-sentence token embedding is optimal for finetuning.This pooling comparison reports average MTEB performance across 56 datasets.Source: F ADDITIONAL FINETUNING RESULTS · p. 32, Table 14 · p. 33
  • Echo embeddings still outperform classical embeddings in additional finetuning ablations, including evaluations with instructions removed and at an earlier training step.The ablations use Mistral-7B and compare instruction, pooling, and training-step choices.Source: F ADDITIONAL FINETUNING RESULTS · p. 32, Table 17 · p. 36
  • Performance decreases over training for many tasks in the plotted subset of MTEB datasets.The training-time analysis is shown in Figure 13.Source: F ADDITIONAL FINETUNING RESULTS · p. 33, Figure 13 · p. 34
  • The appendix reports all-dataset MTEB results for echo, classical, and bidirectional embeddings and compares implementations with recent leaderboard models.These results are organized in Tables 18 and 13.Source: F ADDITIONAL FINETUNING RESULTS · pp. 32–33, Table 18 · p. 37

Related papers

Loading 2402.15449v2…