NV-Retriever: Improving text embedding models with effective hard-negative mining

1 Introduction

• Dense embeddings represent queries and passages for efficient retrieval, including in RAG systems that let LLMs access external context without modifying model parameters.The introduction situates text retrieval across search, question answering, semantic textual similarity, and item recommendation.Source: §1 · p. 1
• Hard-negative mining methods remain underexplored or poorly detailed in papers introducing models that top the MTEB leaderboard.Those studies primarily focus on architectures, fine-tuning approaches, and training-data blends.Source: §1 · p. 1
• NV-Retriever-v1 achieved 1st place on the MTEB Retrieval leaderboard when published on July 11, 2024.The authors describe its leading retrieval accuracy as relying particularly on their positive-aware hard-negative mining methods.Source: §1 · pp. 1–2
• The paper introduces positive-aware hard-negative mining methods intended to remove potential false negatives and improve embedding-model accuracy.It also investigates mining configurations, teacher models, and base models, reporting that retrieval models are sensitive to mining choices.Source: §1 · p. 1

2.1 Text embedding models

• Text embedding models map variable-length text to fixed-dimensional vectors, and benchmarks such as BEIR and MTEB provide retrieval evaluation, with MTEB Retrieval comprising 15 BEIR datasets.The background also surveys Sentence-BERT, DPR, and E5-family approaches to embedding and training.Source: §2.1 · p. 2

2.2 Hard-negative mining for fine-tuning embedding models

• Hard negatives retrieved for a query can be more informative for contrastive learning than random in-batch negatives, which are easy to discriminate and contribute little to convergence.The background notes that hard negatives may be mined from a corpus using sparse or dense retrievers, while incremental mining can be complex and costly for large corpora.Source: §2.2 · p. 2

2.2.1 False negatives.

• Mining can select relevant but unlabeled passages as negatives; one cited MS-MARCO experiment found about 70% of the most query-similar passages should actually be labeled positive.Prior approaches filter potential false negatives with cross-encoders or decoder LLMs, which may require costly inference for every query-negative pair.Source: §2.2.1 · p. 2
• Prior work offers limited detail on mining choices, though reported ablations compare absolute-score thresholds, top-k sampling ranges, and teacher models.The background notes reports that stronger mining models can yield more effective hard negatives.Source: §2.2.1 · pp. 2–3

3.1 Positive-aware hard-negative mining methods

• Top-K shifted-by-N can discard useful hard negatives or retain false negatives, while TopK-Abs applies a maximum negative-score threshold regardless of the positive passage's relevance.The paper presents these limitations as motivation for positive-aware filters.Source: §3.1 · p. 3
• Positive-aware mining filters use the positive relevance score to filter potential false negatives from retrieved top-k candidates.TopK-MarginPos sets the maximum negative-score threshold by subtracting an absolute margin from the positive score; TopK-PercPos sets it as a percentage of that score.Source: §3.1 · p. 3

3.2 Research Questions

• The experiments ask how teacher-model choice and ensembling mined negatives affect downstream accuracy, and how mining methods compare.The planned comparisons include threshold ablations for filtering false negatives.Source: §3.2 · p. 4

3.3.1 Training.

• The general setup fine-tunes e5-large-unsupervised or Mistral-7B-v0.1 using negatives mined with selected methods and teacher models of different sizes.The training set combines Natural Questions, a 2023 Stack Exchange dump, and SQuAD, totaling 287k examples.Source: §3.3.1 · p. 4

3.3.2 Evaluation.

• Evaluation uses NQ, HotpotQA, and FiQA2018 from MTEB Retrieval/BEIR, with a scaled RQ3 experiment evaluating on the full MTEB Retrieval benchmark.The selected datasets are described as more relevant to question-answering RAG systems.Source: §3.3.2 · p. 4

4.1 RQ1. Using different teacher models for mining

• BM25-mined negatives produced the weakest results, followed by random negatives; larger supervised teachers, NV-Embed-v1 and e5-mistral-7b-instruct, produced the strongest fine-tuned models.The comparison fine-tuned e5-large-unsupervised models using negatives mined by different teachers.Source: Table 1 · p. 4, §4.1 · p. 4

4.2 RQ2. Ensembling hard-negatives from different teacher models

• Cross-sample ensembling did not beat the best single teacher, while intra-sample ensembling was more effective and retaining duplicate negatives worked better than deduplicating them.The four teachers' top-4 negatives had less than 30% Jaccard similarity.Source: §4.2 · pp. 4–5, Table 2 · p. 5

4.3.1 Ablation study on mining methods configurations.

• On three BEIR Q&A datasets, Top-k shifted by N favored discarding the top 10; TopK-Abs favored 0.7, while positive-aware settings favored a 0.05 margin or 95% of the positive score.Larger TopK-MarginPos margins and TopK-PercPos thresholds above 95% reduced accuracy.Source: Figure 1 · p. 5, §4.3.1 · pp. 5–6

4.3.2 Sampling from mined negatives.

• Sampling four negatives from top-10 was best; increasing k reduced accuracy for Sampled Top-k, but had less effect when the top-ranked negative was always included.The sampling ablation used TopK-PercPos with a 95% threshold.Source: Figure 2 · p. 6, §4.3.2 · p. 6

4.3.3 Comparing mining methods at their best configuration from the ablation.

• For e5-large-unsupervised, positive-aware methods performed best, with TopK-PercPos leading at 95% of the positive score; added top-k sampling did not further improve results.The table compares each mining method at its best ablation configuration using top-4 negatives.Source: Table 3 · p. 6, §4.3.3 · p. 6

4.3.4 Replicating mining methods experiments with Mistral-7B-v0.1.

• With Mistral-7B-v0.1, positive-aware methods again performed best, and sampling among top-10 was slightly better than always taking the top-ranked negative.Only one negative per example was used for this base model because of memory limits.Source: §4.3.4 · pp. 6–7, Table 4 · p. 7

4.5 The effects of positive-aware neg. mining

• Across NQ and StackExchange train sets, the authors’ methods mined 57% and 50% fewer false negatives, respectively, than Naive Top-k, whose false-negative rates were 38.8% and 47%.Separately, TopK-PercPos separates positive and negative score distributions and limits maximum cross-entropy loss, making training more stable.Source: §4.5.1 · p. 7, §4.5.2 · p. 7

4.4 RQ3.b Comparing mining methods at scale with NV-Retriever-v1

• 60.55 Avg. NDCG@10 for TopK-PercPos would place it first on the MTEB Retrieval / BEIR leaderboard when NV-Retriever-v1 was published; it was best on 13 of 15 datasets.The scaled comparison used the NV-Retriever-v1 setup and 15 retrieval datasets.Source: §4.4 · p. 7, Table 5 · p. 8

• Positive-aware mining uses positive relevance scores to select better negatives for embedding-model fine-tuning.Source: §5 · p. 8
• The ablation compares mining configurations, teacher models, and ensembles, and examines false-negative removal, loss stabilization, and accuracy.Source: §5 · p. 8
• Preliminary multimodal fine-tuning found the methods effective for retrieval accuracy, and the authors encourage disclosure of mining methodology for reproducibility.Source: §5 · p. 8

• Teacher models’ top-four hard negatives have Jaccard similarity below 30% in general across NQ, SQUAD, and StackExchange.Source: A Similarity of hard-negatives mined… · p. 9, Table 6 · p. 9
• The appendix compares teacher pairs using Jaccard-similarity matrices for their top-four mined negatives across the three ablation train sets.Source: A Similarity of hard-negatives mined… · p. 9, Table 6 · p. 9

• NV-Retriever-v1 was placed first on the MTEB Retrieval leaderboard when published, with the proposed positive-aware mining methods described as crucial to its high retrieval accuracy.Source: B NV-Retriever-v1 · p. 9

B.1 Model architecture

• The model converts Mistral 7B to an encoder with bidirectional attention and uses mean pooling across the final Transformer layer for sentence embeddings.Source: §B.1 · p. 10

B.2 Train sets

• Training sets are based on those used for E5-Mistral and NV-Embed-v1, with the same dataset-specific instruction prompts as E5-Mistral.Source: §B.2 · p. 10

B.3 Hard-negative mining

• Hard-negative mining uses E5-Mistral-7B with a 4096-token maximum sequence length, and TopK-PercPos caps negative relevance at 95% of the positive score.Source: §B.3 · p. 10
• Instruction prefixes are added to queries but not passages, and instruction tokens are masked during average pooling in training and evaluation.Source: §B.3 · p. 10

B.4 Training setup

• Stage one uses supervised retrieval datasets with in-batch negatives and mined hard negatives; stage two blends that retrieval data with classification, regression, and semantic-similarity datasets.Source: §B.4 · p. 10
• The ArguAna dataset is the sole exception to the mining setup: an internal Mistral embedding model improved its test-set NDCG@10 versus the E5-Mistral-7B causal-attention model.Source: §B.4 · p. 10
• The retrieval training datasets for NV-Retriever-v1 are documented in Table 7.Source: Table 7 · p. 10
• Training uses Hugging Face Transformers and PEFT, with model and training hyperparameters reported in Tables 8 and 9.Source: §B.4 · p. 10, Table 8 · p. 10

B.5 Computational cost

• 12 epochs per stage and about 90 hours on 8x A100 GPUs are required for full NV-Retriever-v1 training.Source: §B.5 · p. 10
• Embedding 1 million documents with the e5-mistral-instruct teacher takes about 1.5 hours on 8x A100 GPUs, with max token length 4096.Hard-negative mining time depends on corpus size and teacher-model complexity.Source: §B.5 · p. 10
• Evaluating an embedding model on the 15 MTEB BEIR retrieval datasets takes about 120 hours on 8x A100 GPUs.Source: §B.5 · p. 10
• Training hyperparameters for the two stages are presented in Tables 8 and 9.Source: §B.5 · p. 11, Table 9 · p. 11

B.6 NV-Retriever-v1 results on MTEB Retrieval

• 60.9 average NDCG@10 earned NV-Retriever-v1 first place on the MTEB Retrieval leaderboard on July 11, 2024.Source: §B.6 · p. 11, Table 10 · p. 11
• Mistral 7B was the foundation model for four of the leaderboard’s top five models, which used similar training sets.Source: §B.6 · p. 11, Table 10 · p. 11
• The authors identify positive-aware mining as the main difference in NV-Retriever-v1’s training approach compared with those top models.Source: §B.6 · p. 11