Gecko: Versatile Text Embeddings Distilled from Large Language Models

• Gecko targets a single embedding model that supports retrieval, semantic similarity, classification, and other downstream tasks.Source: §1 · p. 1
• LLM-generated synthetic data offers an alternative to collecting large, comprehensive training sets for general-purpose embedding models.The motivating question is how directly LLM knowledge can improve text embedding models.Source: §1 · p. 1
• Gecko generates task-query pairs from unlabeled passages, retrieves candidate passages, and uses an LLM to rerank positives and negatives.The reranking step addresses cases where the source passage is not the best answer target for its generated query.Source: §1 · p. 1
• Gecko is trained across document retrieval, semantic similarity, and classification using FRet, whose queries and passage labels are generated or mined by LLMs.Source: Figure 1 · p. 2
• Gecko-1B with 768-dimensional embeddings achieves the best MTEB performance among models with compatible dimensions and sizes, often exceeding larger or higher-dimensional systems.The model combines LLM-generated and LLM-ranked data with human-annotated data.Source: §1 · p. 2

• Prior embedding models often struggle to generalize across tasks and domains, motivating unified training across diverse task types.Source: §2 · p. 2
• Contrastive-learning research emphasizes selecting informative negatives, while Gecko also uses LLMs to identify better positive passages for each query.The selected positives and negatives fit the format of existing human-annotated training data.Source: §2 · p. 2
• Synthetic-query methods typically address unavailable target-domain labels, whereas Gecko distills broader LLM knowledge into a versatile embedding model.Source: §2 · p. 3
• Unlike related instruction-based approaches, Gecko generates task-query pairs from web passages and uses LLMs to choose positive and negative targets.Source: §2 · p. 3

3. Training Recipe for Gecko

• Gecko uses a 1.2B-parameter pretrained transformer followed by pre-finetuning and fine-tuning on a unified mixture of FRet and academic datasets.Each example is formatted with a task description, query, positive passage, and negative passage.Source: §3 · p. 3, §3.3 · p. 6

3.1. Pre-finetuning

• Pre-finetuning exposes the model to diverse unsupervised text pairs from community question-answer data and web title-body pairs, using task features and in-batch negatives.The objective uses cosine similarity with a temperature parameter and omits hard negatives during pre-finetuning.Source: §3.1 · pp. 3–4

3.2. FRet: Two-Step LLM Distillation

• FRet first prompts an LLM to generate diverse retrieval tasks and queries from sampled web passages, including question-answering and fact-checking intents.Web topics and styles provide additional diversity, while task descriptions guide the generated queries.Source: §3.2 · pp. 4–5
• FRet retrieves nearest-neighbor candidates with an embedding model, has the same LLM rank them using query likelihood and relevance classification, and ensembles the rankings with reciprocal rank fusion.The resulting ranking supports mining both positive passages and hard negatives.Source: Figure 2 · p. 5, §3.2 · pp. 5–6
• About 15% of examples relabel the positive passage because the LLM-selected passage differs from the randomly sampled seed, while FRet contains 6.6M task-query-positive-negative examples.This relabeling approximates the LLM’s preference over the corpus rather than assuming the seed passage is optimal.Source: §3.2 · p. 6

3.3. Unified Fine-tuning Mixture

• The unified fine-tuning mixture combines FRet with academic datasets, handles classification through same-label positives and different-label hard negatives, and supports 768- and 256-dimensional embeddings.A unique ID prevents overlapping classification positives from becoming false in-batch negatives.Source: §3.3 · pp. 6–7

4.1. Main Results

• Gecko significantly surpasses similarly sized baselines across every MTEB text embedding task, while Gecko-1b-768 often matches or exceeds larger models using higher-dimensional embeddings.The model particularly balances retrieval and STS performance and sets state-of-the-art results on classification, STS, and summarization.Source: §4.1 · p. 8

4.2. Multilingual Retrieval Results

• English-only LLM-generated training data transfers well to multilingual retrieval tasks when used with multilingual language models and MIRACL data.The multilingual model is evaluated on retrieval across 18 languages.Source: §4.2 · p. 8, Table 2 · p. 8

4.3. Analysis

• LLM-selected positives outperform original seed passages, showing that generated queries can have more relevant targets elsewhere in the corpus.The relabeled positive differs from the seed passage in about 15% of the dataset, and qualitative examples show this occurs often.Source: §4.3 · p. 9, Table 5 · p. 10
• Training on uniformly sampled examples from all four FRet tasks yields superior cross-task performance, while unified formatting helps separate task behaviors.The evaluated tasks include question answering, search result, fact checking, and sentence similarity.Source: §4.3 · p. 9, Table 4 · p. 9
• Adding symmetric formatting, same-tower negatives, NLI data, and classification data improves semantic similarity and classification, with the full FRet mixture reaching 66.31.The STS improvement is 1.6 on average, while classification improves without significant degradation on other tasks.Source: §4.3 · p. 9
• Qualitative examples show that LLM-generated queries are diverse, LLM-mined positives can answer them more directly, and ranked hard negatives create nuanced distinctions.These examples illustrate how the two-step distillation transfers domain knowledge and ranking preferences into the embedding model.Source: §4.3 · p. 10

5. Conclusion

• Gecko combines LLM-ranked positives and negatives in the synthetic FRet dataset to improve targets for synthesized queries.The model uses this data in a unified format alongside multiple task types.Source: §5 · p. 10
• Combining synthetically generated data in a unified format enables strong performance across multiple tasks at the same time.The conclusion attributes this finding to the combined training strategy rather than to a single task.Source: §5 · p. 10
• Ablations identify LLM-based relabeling and dataset diversity as important components while demonstrating strong zero-shot generalizability.These findings summarize the paper's analysis of Gecko and FRet.Source: §5 · p. 10

• RRF combines QL and RC scores using the reciprocal ranks assigned by the two rerankers.R(q, p) = 1/r_QL(q, p) + 1/r_RC(q, p), with both rank positions positive.Source: A. Enhancing Few-shot LLM Ranking with… · p. 16
• The rerankers use identical few-shot prompts across BEIR rather than task-specific prompts for each BEIR task.Source: A. Enhancing Few-shot LLM Ranking with… · p. 16
• RRF significantly improves overall reranking quality and consistently improves the initial retriever across BEIR tasks except FEVER.The comparison uses nDCG@10 on BEIR.Source: A. Enhancing Few-shot LLM Ranking with… · p. 16, Table 6 · p. 16
• Query likelihood and relevance classification reranking each improve performance over the initial retriever.Source: A. Enhancing Few-shot LLM Ranking with… · p. 16

• A unified encoding format standardizes inputs and targets across datasets and tasks in the FRet mixture.Source: B. Formatting in FRet · p. 16, Table 7 · p. 16
• BEIR performance is sensitive to formatting, whereas symmetric-task performance is relatively stable across formats.The comparison covers asymmetric BEIR and symmetric STS tasks.Source: B. Formatting in FRet · p. 16
• Table 7 compares the unified format with alternative formats for BEIR and STS.Source: Table 7 · p. 16

• Full MTEB results are reported separately for each dataset.Source: C. Full MTEB Results and Instructions · p. 16, Table 8 · p. 17
• Most MTEB tasks use consistent instructions, while BEIR uses multiple instructions reflecting its different intents.The task-intent distinction follows Dai et al. (2022).Source: C. Full MTEB Results and Instructions · p. 16
• Table 9 lists the instruction for each MTEB dataset and summarizes each generated task with a simplified task type.Source: Table 9 · p. 18