Source-linked AI summary
zELO: ELO-inspired Training Method for Rerankers and Embedding Models
Nicholas Pipitone, Ghita Houir Alami, Advaith Avadhanam, Anton Kaminskyi, Ashley Khoo
TL;DR
Pointwise reranking depends on in-batch negatives and unsupervised negative sampling, leaving hard-negative false negatives intractable. The paper trains rerankers with Elo-inspired pairwise scoring, and reports that zerank-1 consistently outperforms other state-of-the-art rerankers across specialized domains, with larger margins on private datasets.
Problem
Pointwise reranking with InfoNCE requires in-batch negatives and unsupervised negative sampling, while false negatives in hard-negative mining remain intractable.
Source: §3.2 · p. 2Method
The paper models query-document relevance with Elo-inspired scoring based on pairwise comparisons rather than pointwise human annotations.
Source: §7 · p. 10Results
zerank-1 consistently outperforms other state-of-the-art rerankers across specialized domains, with larger improvement margins on private datasets.
Source: §6 · p. 9Takeaways & Limitations
zerank-1-small retains much of zerank-1’s superior performance despite being less than half its size and is available under an Apache 2.0 license.
Source: §6 · p. 9The pairwise reranker ideally induces a total order over documents for each query, but the paper says this is not strictly required.
Source: §4.1 · p. 3
Abstract
from arXiv · showhide
We introduce a novel training methodology named zELO, which optimizes retrieval performance via the analysis that ranking tasks are statically equivalent to a Thurstone model. Based on the zELO method, we use unsupervised data in order train a suite of state-of-the-art open-weight reranker models: zerank-1 and zerank-1-small. These models achieve the highest retrieval scores in multiple domains, including finance, legal, code, and STEM, outperforming closed-source proprietary rerankers on both NDCG@10 and Recall. These models also demonstrate great versatility, maintaining their 0-shot performance on out-of-domain and private customer datasets. The training data included 112,000 queries and 100 documents per query, and was trained end-to-end from unannotated queries and documents in less than 10,000 H100-hours.
2. Summary of Contributions
The section presents zELO’s multi-stage training pipeline, released rerankers and benchmark gains, and applications of automated annotations to private-data tuning and live evaluation.
2.1. zELO: A novel Elo-based multi-stage training pipeline
- zELO generates candidate documents, gathers sparse pairwise preferences from LLM ensembles or a distilled SLM, converts them to absolute scores with Thurstone, and fine-tunes pointwise rerankers.Source: §2.1 · p. 1
2.2. Open-source reranker models released
- The released cross-encoder rerankers zerank-1 and zerank-1-small were trained on 112,000 query–corpus pairs, comprising over 5 million query-zELO pairs.Their weights are available on Huggingface; zerank-1-small uses Apache 2.0, while zerank-1 is available for license from ZeroEntropy.Source: §2.2 · p. 1
2.3. State-of-the-art reranking accuracy across domains, languages, and retrieval methods
- zerank-1 improves NDCG by up to 5 points across finance, medicine, legal, code, and math benchmarks, outperforming commercial rerankers twice its size and larger LLM rerankers.It also delivers strong gains with BM25, embedding-based, and hybrid retrieval.Source: §2.3 · p. 1
2.4. Unsupervised training and fine-tuning
- Experiments found that zELO ensembles produced higher-quality data on average than an equivalent number of human annotators, with scores used for private-document benchmarking and domain-specific fine-tuning.Ensemble inference can scale until the target document’s zELO score converges, indicating query-document relevance.Source: §2.4 · p. 1
- Randomly sampled production query logs can be automatically annotated with zELO to uncover retrieval issues, fine-tune rerankers live, or support personalized recommendations.Source: §2.4 · pp. 1–2
3. Motivation
Rerankers provide query–document relevance scores, but corpus-scale retrieval relies on imperfect candidate generators and current training methods depend on human positives and mined negatives. The paper argues that increasingly capable hard-negative miners can produce false negatives that cap or reduce student performance, motivating pairwise ensemble-generated annotations and ELO scores.
3.1. Existing SOTA
- Rerankers score query–document pairs, but scanning an entire corpus is computationally infeasible, so retrieval systems use sparse keyword search or dense embeddings to select candidates.The passage notes keyword search can fail when users do not recall exact terms, while dense embeddings face limitations in representing true relevance.Source: §3.1 · p. 2
- Rerankers can also be trained through SFT distillation, using annotations from a teacher reranker to train a student model.The passage describes human-annotated binary relevance labels as a traditional teacher signal.Source: §3.1 · p. 2
- Current reranker training commonly uses InfoNCE triplets with human-annotated positives, while negative selection ranges from weak random in-batch sampling to hard-negative mining.Random negatives are usually obviously irrelevant and provide weak signal, requiring batches larger than 100k; hard-negative mining seeks more relevant negatives, often using ensembles of embedding models and rerankers.Source: §3.1 · p. 2
3.2. Laffer Curve: The Fundamental Constraint on Existing SOTA Hard Negative Mining
- Hard-negative mining can reduce student performance when mined negatives are more relevant than human-labeled positives, creating a Laffer curve as negative quality rises.The authors report that this occurred with highly capable miners; Figure 1 depicts the proposed curve, but they note they did not collect sufficient data with miner intelligence as the only independent variable.Source: §3.2 · p. 2, Figure 1 · p. 2
- The authors motivate pairwise annotations from frontier-model ensembles and ELO scores as ground truth, arguing their training method avoids hard-negative mining’s theoretical performance cap beyond teacher accuracy.They identify false negatives as an inherent challenge for pointwise InfoNCE, which requires in-batch negatives and an unsupervised negative-sampling strategy.Source: §3.2 · p. 2
4. The zELO Method
zELO turns pairwise document preferences into latent Elo relevance scores, then uses those estimated scores as targets to fine-tune a pointwise reranker. Its method combines pairwise-comparison models with sparse graph sampling to estimate scores efficiently.
4.1. Definitions:
- A pointwise reranker orders documents by relevance scores, while a pairwise reranker estimates the preference probability and strength of one document over another for a query.The ideal pairwise function induces a total order for each query, though the method does not require this property.Source: §4.1 · p. 3
4.2. Using Rpair to generate SFT
- Pairwise reranker scores on retrieved document pairs form a preference matrix that is transformed into latent relevance scores for pointwise reranker training.For a query’s retrieved subset, the method infers pairwise scores, extends them to a sparse matrix by setting non-inferred pairs to zero, and uses the Bradley-Terry framework to estimate scores.Source: §4.2 · p. 3
4.2.1. CONNECTION TO BRADLEY-TERRY MODEL
- The Bradley-Terry formulation parameterizes document abilities as exponentiated Elo ratings and fits ratings by maximum-likelihood gradient descent; Zermelo 1929 establishes a unique local minimum under a final constraint.A normalization constraint is also applied across queries.Source: §4.2.1 · pp. 3–4
4.2.2. THURSTONE MODEL
- The final training run uses Thurstone’s erf() model, which assumes normally distributed document-comparison noise and is reported to fit observed data better than Bradley-Terry’s σ().The authors justify the normal-noise assumption by multiple sources of comparison noise and the central limit theorem.Source: §4.2.2 · p. 4
4.2.3. EXTENSION TO PLACKETT-LUCE FOR COMPLETE RANKINGS
- Plackett-Luce extends pairwise comparisons to complete rankings, but zELO primarily uses Bradley-Terry because its comparisons are sparse for computational tractability.The method describes complete-ranking probabilities in terms of Elo scores.Source: §4.2.3 · p. 4
4.2.4. SPARSE MATRIX SUBSAMPLING FOR ELOS
- Sparse subsampling infers O(n) pairwise comparisons, using unions of random cycles to construct connected, near-regular graphs efficiently for Elo estimation.The cycle-union graph can be generated in O(n) time and is k-connected; it is k-regular if no edges are duplicated, supporting stable estimates through connectivity and degree.Source: §4.2.4 · pp. 4–5
4.3. Training Rpoint from SFT
- The pointwise reranker is fine-tuned with mean squared error using estimated Elo relevance scores, rather than human-annotated binary scores, as targets.The training examples are query-document pairs with relevance scores in [0, 1].Source: §4.3 · p. 6
5. Training the Pairwise Reranker
The section describes a pipeline that turns 112,000 queries and web-scale retrieved documents into pairwise LLM judgments, sparse Elo targets, and pointwise rerankers, then uses human-guided failure mining for a second training pass.
5.1. Dataset
- 112,000 public queries and more than 100M public web-scale documents supplied the training data, with top-100 candidates retrieved using Qwen3-Embedding-4B cosine similarity fused with BM25 by RRF.The lexical pipeline used language detection followed by language-specific stemming.Source: §5.1 · p. 6
5.2. Ensemble Annotation as Source of Truth
- Human labels are costly, noisy, and weaker than the LLM ensemble, so ensemble judgments serve as primary truth while top human-annotated documents provide signal in a second training run.For an internal dataset across several domains, two documents were sampled from a query’s top-k retrieval results, with k inversely sampled from 10 to 100; annotators were sampled until the standard error approached 0.1.Source: §5.2 · p. 6
5.3. Ensemble of LLMs as Synthetic Data Generators
- 3 LLMs generate pairwise preferences for sampled document pairs, and their ensemble judgments train the pairwise reranker; an early BioASQ run reported 93% vs 75% to 81% for SOTA rerankers.The early run found margins of 20% and 10% for 8B and 70B models, respectively; the models also performed similarly on private data, while competing rerankers dropped off.Source: §5.3 · pp. 6–7
5.4. Elo Score Calculation
- Using the method of cycles, the study sampled Elos with N = 400 inferences (0.4% of the full matrix) and k = 8 (4 random cycles), producing ELO scores as pointwise estimates for training R′.With random sampling, the cross-entropy loss between subsampled Elo estimates and estimates from the full 100 × 100 matrix went to zero at about a 1% sampling ratio.Source: §5.4 · p. 8
5.5. Training the Pointwise Reranker
- Qwen-4B and Qwen-1.7B models were fine-tuned with MSE to fit the Elo-derived pointwise relevance scores, producing zerank-1 and zerank-1-small.The models use the same relevance scores generated by the preceding steps.Source: §5.5 · p. 9
5.6. Improving the Pairwise Reranker (RLHF)
- The final training pass adds ensemble-labeled comparisons between a human top-scored document and the document ranked immediately above it when the pointwise reranker misses an acceptable rank threshold.The pairwise reranker is retrained on the expanded data, then Elo targets are regenerated for a second pointwise-reranker iteration.Source: §5.6 · p. 9
6. Results
The zerank rerankers outperform state-of-the-art models across public and private evaluations, with larger margins on private datasets. zerank-1-small retains much of zerank-1’s performance at less than half its size.
6. Results
- Evaluations compare zerank-1 and zerank-1-small with Cohere rerank-v3.5, Voyage AI rerank-2, and Salesforce Llama-rank-v1; neither zerank model was trained on evaluated datasets or human annotations.Source: §6 · p. 9
- zerank-1-small maintains much of zerank-1’s superior performance despite being less than half its size, and is available under an Apache 2.0 license.Source: §6 · p. 9
- The rerankers significantly improve NDCG@10 over OpenAI text-embedding-3-small, BM25, and hybrid initial retrieval with k = 100.Source: §6 · p. 9
- Both zerank models offer large margins over existing state-of-the-art methods, with margins increasing on private datasets.The public comparison uses NDCG@10; Table 1 also reports zerank-1 improvements in NDCG@10 and recall.Source: Table 1 · p. 9, §6 · p. 9
- zerank-1 is as fast or faster than competing models while delivering its performance.The comparison is reported in Table 1.Source: Table 1 · p. 9, §6 · p. 9
- The trained reranker dramatically outperforms gemini-reranker, which ranks documents using Gemini-v2.5-flash pairwise comparisons and Elo scores.Source: §6 · p. 9
7. Conclusion
The conclusion presents zerank-1 and zerank-1-small as state-of-the-art rerankers trained with Elo-inspired pairwise relevance modeling rather than relying primarily on pointwise human annotations. It emphasizes their cross-domain performance and offers both models’ weights for non-commercial use.
7. Conclusion
- The report introduces zerank-1 and zerank-1-small as state-of-the-art models for reranking and information retrieval, trained with Elo-inspired pairwise relevance modeling instead of pointwise human annotations.The pipeline mathematically models query-document relevance scores using pairwise comparisons.Source: §7 · p. 10
- The models exhibit cross-domain versatility, with strong performance across fields including code, finance, and medicine.Source: §7 · p. 10
- The report offers the weights of both models for non-commercial use.Source: §7 · p. 10
9. Appendix
The appendix specifies a pairwise relevance-scoring prompt that compares two documents for a query and returns a signed score. Document order is randomized to mitigate position preference, with the score negated when the order is reversed.
9.1.1. SINGLE-QUERY PAIRWISE SCORES
- The prompt asks the model to weigh each document’s pros and cons before deciding which is more relevant to the query.It instructs the model to consider Document A first, then Document B, and make its comparison only at the end.Source: §9.1.1 · p. 13
- Scores range from -1.0 to 1.0, with negative values favoring Document A and positive values favoring Document B.Source: §9.1.1 · p. 13
- Document A/B assignment is randomized to mitigate position preference, and the score is negated when the order is reversed.Source: §9.1.1 · p. 13