Source-linked AI summary
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu
TL;DR
Existing embeddings can lose performance across tasks and domains, often requiring task-specific finetuning. INSTRUCTOR combines text with task instructions, trains on 330 instructed datasets, and achieves state-of-the-art results across 70 evaluation datasets, including 66 unseen during training. Its results support instruction finetuning as a broadly applicable approach for text embeddings.
Problem
Existing embeddings often degrade on new tasks or domains, creating a need for annotated downstream data and task-specific finetuning.
Method
INSTRUCTOR is a single multitask embedder that combines text inputs with task and domain instructions and trains with contrastive loss on MEDI's 330 instructed datasets.
Results
3.4% average improvement over prior state-of-the-art embedding models across 70 diverse datasets, including 66 unseen during training.
Takeaways & Limitations
Instruction finetuning supports broadly applicable, task- and domain-aware text embeddings without further task- or domain-specific training.
Takeaways & Limitations
Finetuning used only four negative examples because of computation constraints, and GTR-XXL was not instruction-finetuned.
Abstract
from arXiv · showhide
We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets. Our model, code, and data are available at https://instructor-embedding.github.io.
1 Introduction
Existing text embeddings often specialize in particular tasks or domains, motivating a single model that uses task and domain instructions to adapt embeddings without further finetuning. INSTRUCTOR trains on diverse instructed datasets and performs strongly across broad embedding applications.
- Motivation: Existing embeddings can degrade on new tasks or domains, often requiring substantial annotated data for downstream finetuning.DPR is stronger for retrieval than similarity, while SimCSE shows the opposite pattern; domain shifts such as medicine and finance also pose challenges.
- Approach: INSTRUCTOR generates task- and domain-aware embeddings by combining each text input with instructions describing its end task and domain.The same text can be embedded into different vectors for different downstream goals.
- Approach: INSTRUCTOR is trained with contrastive loss on MEDI, a collection of 330 text embedding datasets annotated with human-written task instructions.Training maximizes similarity between semantically related pairs and minimizes similarity between unrelated pairs.
- Evaluation: Across 70 embedding evaluation datasets, including 66 unseen during training, INSTRUCTOR spans classification, similarity, retrieval, generation evaluation, and prompt retrieval.The evaluation covers diverse domains such as finance, medicine, and news.
- Results: INSTRUCTOR outperforms prior state-of-the-art embedding models by an average of 3.4% across the 70 diverse datasets.It also outperforms a variant trained without task instructions, and the analysis reports robustness to instruction paraphrases.
2 INSTRUCTOR
INSTRUCTOR is a single encoder that produces task- and domain-aware embeddings by pairing each text with instructions, then trains on diverse instruction-annotated data using contrastive learning.
- Architecture: INSTRUCTOR encodes each input together with task instructions to produce task-specific fixed-sized embeddings for downstream language tasks without additional training.Its single encoder architecture uses a GTR backbone and mean-pools representations over the input tokens.
- Instruction-based finetuning: Training represents diverse tasks as distinguishing good and bad candidate outputs for an input, using instructions associated with both sides.Retrieval uses relevant versus irrelevant documents, while similarity and classification construct positive and negative text pairs differently.
- Instruction-based finetuning: The objective maximizes cosine similarity for positive pairs and minimizes it for negative pairs with bidirectional in-batch sampled loss.The loss uses a softmax temperature and combines the objective computed with the input and candidate swapped.
- MEDI data: MEDI combines 300 Super-NaturalInstructions datasets with 30 existing embedding datasets to create 330 instruction-annotated datasets across task categories and domains.For datasets lacking instructions, the authors manually write prompts using a unified template.
- MEDI data: Instructions specify text type, an optional task objective, and an optional domain in a unified format describing how each input is used.For asymmetric tasks such as open-domain question answering, separate instructions describe questions and documents.
- Evaluation: Across 70 evaluation datasets, instruction finetuning improves performance by 5.9% over same-size GTR models and by 3.4% or 4.1% over Sent-T5-XXL variants.The evaluation spans nine downstream applications, including retrieval, classification, semantic textual similarity, and text generation evaluation; 66 tasks are unseen during finetuning.
3 Experiments
INSTRUCTOR is evaluated across 70 downstream embedding tasks and three benchmarks, where it achieves state-of-the-art average performance across diverse task categories. It also outperforms same-size and larger specialized baselines.
- 70 downstream tasks span classification, reranking, information retrieval, prompt retrieval, and text generation evaluation.
- INSTRUCTOR achieves state-of-the-art performance across all three benchmarks on average.
- 5.7%, 18.3%, and 5.7% are the gains over GTR-Large in MTEB, Billboard, and prompt retrieval, respectively.
- 18.3%, 10.1%, and 8.9% are INSTRUCTOR’s improvements over GTR-Large on text evaluation, classification, and clustering tasks.
- 58.4 vs. 56.5 on average is INSTRUCTOR’s performance compared with Sent-T5-XXL, despite 335M versus 4.8B parameters.
- Specialized retrieval- and similarity-based baselines excel on narrower task groups, whereas INSTRUCTOR performs well across diverse categories.
4 Analysis and Ablations
The analyses examine how instructions, training-data diversity, instruction variation, model size, domain shifts, and embedding geometry affect INSTRUCTOR. Instructions are especially useful for combining diverse data and generalizing across tasks and domains.
- Importance of Instructions: Instruction finetuning enables INSTRUCTOR to benefit from combining symmetric and asymmetric training data.
- Instruction Robustness: 300 Super-NaturalInstructions datasets reduce performance gaps across paraphrased instructions, improving robustness to different formats and styles.
- Complexity of Instructions: Performance improves as instructions progress from none and dataset tags to simple and detailed task descriptions.
- Model Sizes and Instruction Finetuning: INSTRUCTOR benefits more than GTR from increasing encoder size, perhaps because instructed embeddings require additional capacity.
- Domain Shifts: Instruction finetuning improves GTR-Large’s performance on the unseen geography, biology, and civil comments domains.
- Qualitative Analysis: With instructions, same-sentiment pairs become closer and different-sentiment pairs become farther apart in the embedding space.
5 Related Work
Prior embedding models often specialize in particular tasks or domains, while instruction finetuning had not been broadly studied for general-purpose embeddings. INSTRUCTOR addresses this gap by conditioning embeddings on task instructions.
- Text Embeddings: Existing models such as SBERT, SimCSE, DPR, and Contriever focus on narrower task families.
- Text Embeddings: MEDI combines symmetric and asymmetric datasets to train INSTRUCTOR across diverse embedding tasks.
- Instruction Finetuning: Instruction finetuning enables language models to perform new tasks from natural-language instructions, but its use for broadly applicable embeddings remained understudied.
- Instruction Finetuning: INSTRUCTOR embeds inputs with task instructions to produce task-specific representations usable across downstream language tasks without additional training.
6 Conclusion
INSTRUCTOR creates broadly applicable text embeddings from natural-language instructions and achieves state-of-the-art performance across embedding benchmarks and prompt retrieval.
- INSTRUCTOR creates broadly applicable text embeddings using natural-language instructions.
- INSTRUCTOR is a single model trained on the diverse MEDI dataset with instructions.
- INSTRUCTOR achieves state-of-the-art performance on text embedding benchmarks and prompt retrieval for few-shot in-context learning.
7 Limitations
The authors identify computational constraints and instruction design as limitations and directions for future improvement.
- Four negative examples were used during finetuning because of computation constraints.The authors suggest scaling the number of negatives and mining hard negatives in future work.
- Instruction design remains a limitation because the unified format may be improved with demonstrations and explanations.The paper identifies other instructional elements as a future research direction.
A Training Setups
The evaluation uses diverse benchmarks and compares INSTRUCTOR with embedding baselines spanning retrieval and semantic textual similarity, including larger models.
- Training: Training samples are drawn by randomly selecting a dataset and constructing each minibatch from that dataset.Large datasets are downsampled, and same-dataset in-batch negatives are used.
- Training: INSTRUCTOR is initialized from GTR-Large and finetuned on MEDI for 20K steps with AdamW.The training uses learning rate 2 × 10^-5, warmup ratio 0.1, and softmax temperature 0.01.
- Baselines: Baselines include retrieval-specialized models such as Contriever-MS, GTR, and coCondenser-MS, alongside semantic-textual-similarity models.The retrieval baselines are trained on open-domain question-answering datasets.
- Baselines: The two 4.8B-parameter models achieve the first and second best average performances.
- Evaluation: 70 evaluation datasets are divided into 9 categories, with 66 tasks unseen during training.
B.1 Massive Text Embedding Benchmark
The benchmark section defines the tasks, metrics, and datasets used to assess embeddings across retrieval, classification, similarity, clustering, generation evaluation, and prompt retrieval.
- MTEB: MTEB provides a holistic evaluation of embedding models across a wide range of tasks and domain-specific datasets.The paper reports average performance over 56 datasets.
- Retrieval: Retrieval embeds queries and corpus documents separately and ranks documents by cosine similarity, measured with NDCG@10.
- Reranking: Reranking orders documents by query-document cosine similarity and uses MAP for evaluation.
- Clustering: Clustering maps documents into embeddings, partitions them with k-means, and evaluates clusters using v-measure.
- Pair Classification: Pair classification uses cosine similarity between separately encoded texts to predict binary labels, evaluated with average precision.
- Classification: Classification uses frozen sentence embeddings as features for a trained classifier and reports test-set accuracy.
- STS: STS measures cosine similarity between separately embedded sentence pairs and compares scores with human judgements using Spearman’s rank correlation.
- Generation Evaluation: Generation evaluation compares reference and machine-generated summaries by cosine similarity between their embeddings and correlates scores with human judgements.The setup is described for automatic summarization evaluation.
D Full Results
This section provides detailed evaluation results and documents the datasets and instruction designs used in training and evaluation.
- Detailed MTEB, Billboard, and prompt retrieval benchmark scores are reported in Tables 9 and 10.
- Table 5 lists the number of training instances for each dataset and marks datasets whose test splits are included in evaluation.
- Example paraphrased instructions for AmazonPolarityClassification and FIQA2018 follow a unified template while varying wording choices.
- For the asymmetric training dataset, MSMARCO examples are clustered into 30 groups and each group receives a domain label.
- For the symmetric training dataset, task definitions from Super-NaturalInstructions specify the task objective.