Source-linked AI summary
Nomic Embed: Training a Reproducible Long Context Text Embedder
Zach Nussbaum, John X. Morris, Brandon Duderstadt, Andriy Mulyar
TL;DR
Existing embedding models provide limited open long-context coverage and often lack transparent training artifacts. The paper trains nomic-embed-text-v1 and releases its weights, code, and data for replication. The model surpasses OpenAI baselines on MTEB and LoCo, while showing a limitation on the JinaAI long-context benchmark and a questionable WikiCities measure.
Problem
Open-source embedding models are often limited to short contexts or lack transparent training data and code, while many strong long-context models are closed-source.
Method
The paper develops a long-context text embedding model through an end-to-end training pipeline and releases its weights, code, and training data.
Results
nomic-embed-text-v1 outperforms OpenAI text-embedding-ada-002 and text-embedding-3-small on MTEB and LoCo and is the only open-source long-context model reported to do so on MTEB.
Takeaways & Limitations
The released artifacts enable end-to-end auditability and replication of a high-performing open-source long-context embedding model.
Takeaways & Limitations
On the JinaAI Long Context Benchmark, nomic-embed-text-v1 underperforms the cited OpenAI models, and WikiCities may not reliably measure long-context embedding performance.
Abstract
from arXiv · showhide
This technical report describes the training of nomic-embed-text-v1, the first fully reproducible, open-source, open-weights, open-data, 8192 context length English text embedding model that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on the short-context MTEB benchmark and the long context LoCo benchmark. We release the training code and model weights under an Apache 2.0 license. In contrast with other open-source models, we release the full curated training data and code that allows for full replication of nomic-embed-text-v1. You can find code and data to replicate the model at https://github.com/nomic-ai/contrastors.
1 Introduction
Text embeddings support retrieval and other NLP applications, but existing open-source models often have short context limits while strong long-context models are frequently closed-source. The paper presents a 137-million-parameter model that outperforms OpenAI baselines on short- and long-context benchmarks and releases artifacts for replication.
- Motivation: Text embeddings encode sentence semantics as low-dimensional vectors for retrieval, clustering, classification, and visualization.They are used in applications including retrieval-augmented generation and semantic search.
- Motivation: Most top open-source MTEB models support only 512-token contexts, limiting utility when document semantics extend beyond individual sentences or paragraphs.Examples include E5, GTE, and BGE.
- Motivation: Many strong embedding models with contexts longer than 2048 tokens are closed-source, including Voyage-lite-01-instruct and text-embedding-ada-002.This creates a gap between long-context capability and open availability.
- Motivation: Existing open long-context alternatives have limitations: Jina Base does not surpass Ada-002, while 7-billion-parameter E5-Mistral is resource-intensive and performs poorly beyond 4096 tokens.These constraints affect both comparative performance and engineering feasibility.
- Contribution: 137 million parameters: nomic-embed-text-v1 outperforms OpenAI text-embedding-ada and text-embedding-3-small on MTEB and LoCo.The comparison covers both short-context and long-context benchmarks.
- Contribution: The authors release model weights, training code, and training data to enable end-to-end auditability and replication.The released artifacts cover the full training pipeline.
2 Related Work
Prior work includes both open and closed long-context embedding models, but high-performing open-weight systems often trail closed-source baselines and provide incomplete training transparency. The paper positions its contribution against gaps in performance, efficiency, and reproducibility.
- Open long-context models: Jina-embedding-v2-base underperforms text-embedding-ada-002 on MTEB and the Jina Long Context Benchmark.It also underperforms several open-weight short-context models, including E5, GTE, and BGE.
- Training transparency: High-performing open-weight embedding models often omit training data, curation details, mining procedures, or training code.The cited examples include E5, GTE, and Jina embeddings.
- Training transparency: Closed-source models such as Voyage-lite-01-instruct, text-embedding-ada-002, and text-embedding-3 also provide few training details.This limits visibility into how those systems were trained.
3 Background
Text embedding models commonly use three training stages: masked language modeling, weakly supervised contrastive pretraining, and contrastive finetuning. The background also introduces RoPE context-length extrapolation methods, including position interpolation and frequency-based scaling, while Table 1 frames the model's benchmark comparison.
- Training stages: State-of-the-art text embedding models generally combine masked language modeling, weakly supervised contrastive pretraining, and contrastive finetuning.These stages respectively support representation learning, discrimination between relevant and irrelevant documents, and performance improvements using human-labeled datasets.
- Weakly-supervised Contrastive Pretraining: InfoNCE contrastive pretraining teaches the model to distinguish similar documents from irrelevant documents using cosine similarity scores.The paper uses a unidirectional query-to-document loss rather than adding a document-to-query loss.
- Contrastive Finetuning: Contrastive finetuning adds hard negative documents to the loss to improve downstream performance, especially for question answering and web-search retrieval.Hard negatives are documents chosen to be close to the positive document but not true positives for the query.
- RoPE Context Length Extrapolation: RoPE encodes relative positional information through rotations within attention layers but has limitations when sequence lengths exceed those seen during training.The paper discusses position interpolation and frequency-based scaling as extension methods.
- Position Interpolation: Position interpolation rescales position indices into the original training range, but stable longer-context performance requires fine-tuning on a smaller dataset.The target extended context length is denoted L′ > L.
- Frequency-based Scaling: Dynamic NTK scaling introduces α, preserves original position embeddings within the pretrained context, and gradually scales embeddings for longer sequences.This gradual scaling is described as preventing abrupt performance degradation and can be used without finetuning.
4.1 Masked Language Modeling
The masked-language-modeling stage trains a modified BERT model on packed 2048-token BooksCorpus and Wikipedia sequences. Architectural and training changes produce a 137 million parameter encoder that is trained at 2048 tokens and extended to 8192 tokens at inference.
- Data and sequence construction: BooksCorpus and a 2023 Wikipedia dump are tokenized and packed into 2048-token chunks for masked language modeling.Short documents are concatenated, while longer documents are split across chunks.
- Architecture: The model replaces absolute positional embeddings with rotary positional embeddings, uses SwiGLU activations and Flash Attention, and removes dropout.The architecture also uses a vocabulary size that is a multiple of 64.
- Architecture: 137 million parameters result from the adapted BERT architecture.
- Context extension: 2048-token training is extended to 8192-token inference using Dynamic NTK interpolation.SwiGLU is selected over GeGLU because runtime is roughly 25% faster with the Flash Attention repository.
- Training modifications: The training recipe uses a 30% masking rate instead of 15% and removes Next Sentence Prediction.The model is trained with AdamW, linear warmup and decay, large batches, DeepSpeed stage 2, mixed precision, and no gradient clipping.
4.2 Weakly-Supervised Contrastive Pretraining
The model is trained with large, filtered contrastive datasets, long-context text pairs, task-specific prefixes, and supervised fine-tuning across multiple objectives and domains.
- Contrastive data curation: 470 million contrastive pairs across 29 datasets span web retrieval, scientific-article clustering, and other objectives and domains.The curated data is filtered before training to remove potential false positives.
- Contrastive data curation: Consistency filtering uses pretrained embedding models and top-2 cosine-similarity neighbors to discard examples whose documents are not among the nearest neighbors.The authors chose gte-base over all-MiniLM-L6-v2 because the latter discarded true positives with low lexical overlap; filtering reduced the dataset to approximately 235 million pairs.
- Long-context pair construction: Long-context pairs combine Wikipedia titles with article bodies and S2ORC abstracts with full paper text to expose the model to sequences beyond the shorter source datasets.These pairs are curated specifically to support learning long-range dependencies.
- Task conditioning: Task-specific prefixes distinguish retrieval, semantic similarity, and clustering behaviors in the biencoder.The prefixes address conflicting reward signals between tasks such as semantic similarity and question answering.
- Supervised fine-tuning: Supervised fine-tuning uses datasets including MSMarco, NQ, NLI, HotpotQA, FEVER, MEDI, WikiAnswers, and Reddit, with hard negatives mined for retrieval datasets.The training set contains 1.6 million datapoints, and batches are filled from one sampled dataset at a time.
5 Results
The paper evaluates its adapted encoder and text embedding model across GLUE, MTEB, Jina Long Context, and LoCo benchmarks. Results show strong short-context performance and competitive long-context behavior, with limitations against some closed-source models at 8k context.
- 5.1 nomic-bert-2048 GLUE Results: nomic-bert-2048 scores similarly to MosaicBERT across GLUE tasks except CoLA.The authors suggest corpus composition and positional encoding differences may explain the CoLA gap.
- 5.1 nomic-bert-2048 GLUE Results: JinaBERT performs slightly worse on average than nomic-bert-2048 and MosaicBERT, although it outperforms nomic-bert-2048 on CoLA.JinaBERT uses a 512-token maximum sequence length, whereas nomic-bert-2048 uses a longer sequence length.
- 5.2.1 MTEB Results: nomic-embed-text-v1 outperforms all similarly sized open-source models on MTEB except BGE-Base and also exceeds several larger open-source models.The larger models named are E5 Large v2, GTR XXL, and Sentence T5 XXL.
- 5.2.1 MTEB Results: nomic-embed-text-v1 outperforms text-embedding-ada-002 and text-embedding-3-small on average and notably on MTEB Retrieval.It is reported as the only open-source long-context embedding model to outperform both closed-source models on MTEB.
- 5.2.3 JinaAI Long Context Benchmark: Across all context lengths, nomic-embed-text-v1 outperforms jina-embeddings-v2-base, but at 8k context it is outperformed by text-embedding-ada-002.At shorter sequence lengths, nomic-embed-text-v1 performs similarly to text-embedding-ada-002.
- 5.2.4 LoCo Benchmark: At 4096 tokens, both nomic-embed-text-v1 variants perform similarly to E5 Mistral, a model approximately 70x larger, except on Tau Scrolls.At 8192 tokens, both variants outperform text-embedding-ada-002 and text-embedding-3-small while performing similarly to jina-embeddings-v2-base.
6 Conclusion
The paper presents nomic-embed as a fully open-source long-context text embedding model that surpasses named OpenAI baselines on short- and long-context benchmarks. It releases the model, code, and reproduction recipe.
- nomic-embed is presented as the first fully open-source long-context text embedding model to surpass OpenAI text-embedding-Ada-002 and text-embedding-003-small on short- and long-context benchmarks.
- The released reproduction recipe includes the model weights, training code, and data.
- The model had over 14 million downloads on the Hugging Face model hub as of the report's writing.
A Training Resources
Full training can be completed in roughly one week on one 8xH100 node, with masked language modeling and contrastive pretraining taking most of the time.
- Full training of nomic-embed-text-v1 can be conducted in a single week on one 8xH100 node.
- Masked language modeling takes roughly 4 days, while contrastive pretraining lasts 3 and a half days.
- Contrastive finetuning takes one hour.
B Pretraining Dataset Distribution
Table 7 presents the weakly unsupervised pretraining dataset distribution, reporting datapoint counts per source after consistency filtering.
- Table 7 details the datasets used for weakly unsupervised contrastive pretraining.
- The table reports the distribution of weakly unsupervised pretraining data across sources.
- Datapoint counts are reported after consistency filtering.