PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Text Embeddings with Large Language Models

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei

arXiv:2401.00368v3cs.CLcs.IR

TL;DR

Existing embedding methods depend on complex pipelines and manually collected data with limited task and language coverage. This paper uses proprietary LLMs to generate diverse synthetic data across many tasks and languages, then fine-tunes open-source decoder-only LLMs with contrastive learning. The resulting models perform competitively without labeled data and achieve state-of-the-art benchmark results when synthetic and labeled data are combined.

  • Problem

    Existing methods rely on complex multi-stage training and manually collected datasets constrained in task diversity and language coverage.

  • Method

    Proprietary LLMs generate diverse synthetic embedding data across hundreds of thousands of tasks in 93 languages for fine-tuning open-source LLMs.

  • Results

    Mistral-7B performs competitively on BEIR and MTEB with synthetic data alone, while synthetic plus labeled data surpasses previous methods by 2%.

  • Takeaways & Limitations

    LLMs and synthetic data can produce strong text embeddings through a more streamlined training process.

  • Takeaways & Limitations

    Using LLMs for embeddings substantially increases inference cost relative to mainstream BERT-style encoders.

Abstract

from arXiv · show

In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.

1 Introduction

Existing text-embedding methods rely on complex multi-stage pipelines and manually collected data with limited task and language coverage. The paper proposes synthetic-data generation with LLMs and reports competitive or state-of-the-art benchmark performance.

  • Existing state-of-the-art methods pre-train on billions of weakly supervised text pairs before fine-tuning on labeled datasets.
  • These approaches require substantial engineering to curate relevance pairs and depend on manually collected datasets constrained in task diversity and language coverage.Instructor uses instructions from 330 English datasets, while BGE focuses on high-resource languages such as English and Chinese.
  • The paper uses proprietary LLMs to generate synthetic data spanning hundreds of thousands of embedding tasks in 93 languages.A two-step strategy first brainstorms candidate tasks and then generates data conditioned on a selected task.
  • Mistral-7B fine-tuned only on synthetic data achieves competitive performance on BEIR and MTEB without labeled training data.
  • A mixture of synthetic and labeled data yields new state-of-the-art results, surpassing previous methods by 2%.The full training process uses fewer than 1k steps.
  • For low-resource languages, performance still has room for improvement because current open-source LLMs are not adequately pretrained on them.

2 Related Work

Text-embedding research spans classical similarity baselines, supervised encoders, synthetic-data methods, and LLM-based approaches. The paper positions its contribution as combining synthetic data with recent LLM advances to train state-of-the-art embeddings.

  • Text embeddings represent text in continuous low-dimensional vectors and support information retrieval, question answering, and retrieval-augmented generation.
  • Supervised embedding methods use natural-language-inference data or labeled query-document pairs, but labeled data often have limited task diversity and language coverage.
  • Synthetic-data approaches generate queries, pseudo-labels, text pairs, or instructions to improve retrieval systems and train models.Examples include Doc2query, InPars, Promptagator, GPL, Query2doc, DINO, Unnatural Instructions, Orca, and Phi.
  • LLM-based embedding studies demonstrate scaling effects but still trail small bidirectional encoders such as E5 and BGE.
  • The paper proposes training state-of-the-art text embeddings by exploiting recent LLM advances and synthetic data.

3 Method

The method generates diverse multilingual synthetic embedding data through task taxonomies and prompt templates, then trains an LLM-based embedding model with instruction-formatted queries and contrastive learning.

  • Synthetic Data Generation: The synthetic-data pipeline categorizes embedding tasks into groups and applies different prompt templates to each group.The design targets diversity across tasks and languages.
  • Synthetic Data Generation: Asymmetric tasks use a two-step prompt that brainstorms task definitions and generates examples conditioned on those definitions.The category includes short-long, long-short, short-short, and long-long query-document matches.
  • Synthetic Data Generation: Symmetric tasks cover monolingual semantic textual similarity and bitext retrieval with separate prompt templates, without the brainstorming step.
  • Synthetic Data Generation: Randomly sampled prompt placeholders vary properties such as query length, while language sampling produces multilingual data with greater weight on high-resource languages.Malformed JSON outputs are discarded and exact duplicates are removed.
  • Training: Queries receive an instruction containing the task definition, while documents remain unmodified so the document index can be prebuilt and task customization occurs on the query side.
  • Training: The model obtains query and document embeddings from the final-layer [EOS] vectors and trains with standard InfoNCE loss using in-batch and hard negatives.Matching uses temperature-scaled cosine similarity, with temperature fixed at 0.02.

4 Experiments

The experiments evaluate synthetic-data training across benchmark performance, multilingual retrieval, and long-context passkey retrieval. The model performs competitively without labeled data, achieves strong English benchmark results with full data, and remains weaker on low-resource languages.

  • Statistics of the Synthetic Data: 500k examples with 150k unique instructions span 93 languages, although the predominant language is English and low-resource languages receive about 1k examples each on average.The synthetic corpus uses about 180M tokens; 25% of examples come from GPT-35-Turbo and the remainder from GPT-4.
  • Statistics of the Synthetic Data: GPT-35-Turbo outputs sometimes violate prompt-template guidelines, although the authors report acceptable overall quality and preliminary benefits from including them.
  • Model Fine-tuning and Evaluation: The pretrained Mistral-7B checkpoint is fine-tuned for one epoch with contrastive loss, using LoRA and memory-saving training techniques.
  • Main Results: The full-data E5mistral-7b model achieves the highest average MTEB score, outperforming the previous state-of-the-art model by 2.4 points.The MTEB table averages results across 56 datasets in its English subset.
  • Main Results: Synthetic-data-only fine-tuning remains competitive without labeled data, while mixed synthetic and labeled training produces new state-of-the-art benchmark results.The reported benchmark evidence concerns BEIR and MTEB performance.
  • Multilingual Retrieval: The model excels in high-resource multilingual retrieval but remains suboptimal for low-resource languages on MIRACL and bitext mining.The authors attribute this limitation to Mistral-7B’s predominantly English pre-training.

5 Analysis

The analysis examines whether contrastive pre-training is necessary for LLM-based embeddings, evaluates configuration choices, and tests long-context retrieval. Results suggest extensive autoregressive pre-training can reduce the need for contrastive pre-training, while context handling and instructions remain important design considerations.

  • Contrastive Pre-training: Weakly-supervised contrastive pre-training underlies many existing text embedding models, using constructed or filtered text pairs as positives.
  • Configuration Analysis: The default MTEB configuration uses last-token pooling, LoRA rank 16, and natural-language instructions, while instruction handling substantially affects performance.Pooling type and LoRA rank have comparatively small overall effects in the reported configurations.
  • Contrastive Pre-training: 8.2 points improve XLM-Rlarge retrieval performance with contrastive pre-training, whereas Mistral-7B models show negligible quality impact.The comparison suggests that contrastive pre-training benefits encoder-style models more than these extensively autoregressively pretrained LLMs.
  • Long-Context Retrieval: Personalized passkey retrieval evaluates whether embeddings retain information from long contexts, with variants changing sliding-window size and RoPE rotation.The task addresses the short-context bias of existing evaluation datasets.
  • Long-Context Retrieval: Long-context evaluation is motivated by the need to assess capabilities beyond the short inputs typical of existing text embedding datasets.

6 Conclusion

The paper uses proprietary LLMs to generate diverse multilingual synthetic data and combines it with Mistral's language understanding to train text embeddings efficiently. This approach produces state-of-the-art MTEB performance while avoiding intermediate pre-training, although multilingual improvement remains future work.

  • Proprietary LLMs generate diverse synthetic embedding data with instructions in many languages, which is combined with Mistral's language understanding.
  • The model achieves new state-of-the-art results for nearly all task categories on the competitive MTEB benchmark.
  • The streamlined training process obviates the need for intermediate pre-training used by existing multi-stage approaches.
  • Future work targets further multilingual improvement and the use of open-source LLMs for synthetic data generation.

Limitations

The method has higher inference and storage costs than mainstream BERT-style encoders, and its synthetic-data pipeline depends on manually engineered prompts. The paper identifies dimensionality reduction and automatic prompt optimization as directions for improvement.

  • Using Mistral-7B for embeddings substantially increases inference cost compared with mainstream BERT-style encoders.The paper suggests that more advanced GPUs and improved kernel implementations could improve inference efficiency.
  • 4096-dimensional embeddings make the model comparatively more expensive to store than lower-dimensional alternatives.Matryoshka representation learning is cited as an approach that can reduce dimensions while maintaining competitive performance.
  • Synthetic-data generation relies on manual prompt engineering to elicit high-quality outputs from proprietary LLMs.Automatic prompt optimization is proposed as a potential way to improve synthetic-data quality.

A Implementation Details

The implementation uses public baseline checkpoints, a mixture of synthetic and established labeled datasets, LoRA-based fine-tuning, and benchmark-specific evaluation protocols. Reported tables cover multilingual retrieval and the effects of contrastive pre-training.

  • Baselines: Baseline experiments use publicly available mE5base and mE5large checkpoints, with weighted mean pooling implemented following SGPT.Task-type prefixes are added for selected asymmetric and long-short matching settings.
  • Training Data: The full-data E5mistral-7b setting combines generated synthetic data with datasets including ELI5, HotpotQA, FEVER, MIRACL, MSMARCO, NQ, NLI, SQuAD, and TriviaQA.
  • Fine-tuning: Mistral-7B fine-tuning uses batch size 2048, learning rate 10^-4, 100 warmup steps, linear decay, one hard negative, and LoRA adapters on all linear layers.The setup has 42M trainable parameters and takes roughly 18 hours on 32 V100 GPUs with maximum sequence length 512.
  • Evaluation Tables: MIRACL results report nDCG@10 and Recall@100 across 16 languages, while a separate table evaluates contrastive pre-training effects.The contrastive-pre-training comparison pre-trains Mistral-7B for 10k steps following the mE5 recipe.
  • Artifacts: The released models and evaluation scripts are intended to support reproducibility and further research.

B Test Set Contamination Analysis

The analysis finds no substantial contamination risk in the evaluated data, while acknowledging that contamination in Mistral-7B and GPT-4 training data cannot be estimated because those data are inaccessible.

  • String matching was used to compare MTEB test sets with the training set, ignoring differences in character case and spacing.
  • Four DBPedia test questions overlap with TriviaQA training data, but their minor proportion makes their overall performance impact insignificant.
  • Shared retrieval corpora, such as Wikipedia passages across DBPedia, NQ, and TriviaQA, are treated as standard evaluation practice rather than contamination.
  • The authors did not detect substantial contamination risks capable of altering the paper’s main findings.
  • Contamination in Mistral-7B and GPT-4 training data remains difficult to estimate because their training data are not publicly accessible.

C Prompts for Synthetic Data Generation

Synthetic-data prompts cover multiple matching configurations and multilingual generation, with sampling settings chosen to encourage data diversity.

  • Prompts cover short-long matching with varied query types, lengths, difficulty levels, clarity, and document word counts.
  • Multilingual generation samples languages from the XLM-R language list, assigning higher probability to high-resource languages.
  • GPT-4/3.5 generation uses temperature 1.0 and top-p 1.0 to encourage greater diversity than default settings.

D Instructions for Training and Evaluation

The paper documents manually written training and evaluation instructions alongside prompt templates, synthetic-data samples, and per-dataset MTEB results.

  • Training and evaluation datasets use manually written instructions listed in Tables 14 and 15.
  • Long-short matching prompts vary document length, difficulty, and clarity through explicitly listed placeholder values.
  • Short-short and long-long matching templates omit negative documents, respectively because matching is difficult and API latency is costly.
  • Monolingual STS and bitext-retrieval templates vary score ranges, units, and difficulty levels through defined placeholders.
  • Table 16 provides random samples for each synthetic-data subgroup.
  • Table 17 reports results for every MTEB dataset, with evaluation metrics and detailed baseline results referenced from the original benchmark paper.
Loading 2401.00368v3…