Source-linked AI summary
Data Curation and Model Souping: Unlocking Qwen3-0.6B for MS MARCO Passage Ranking
Shuguang Han
TL;DR
This report asks how lightweight Qwen3-0.6B models can rank MS MARCO passages competitively across re-ranking and retrieval settings. It uses curated training data and weight merging, and reports competitive performance across three settings while retaining single-model inference latency.
Problem
The report examines how to achieve competitive MS MARCO passage ranking with lightweight Qwen3-0.6B models across retrieval and reranking settings.
Source: §4 · p. 9Method
The approach combines target-corpus continued pre-training, curated training data and hard negatives, training-inference alignment, and Model Soup weight merging.
Source: §3.1 · p. 4, §3.2 · p. 6, §3.3 · p. 8Results
Across BM25 reranking, corpus-wide dense retrieval, and dense-retrieval reranking, Qwen3-0.6B models achieve competitive ranking without architectural changes, customized losses, or cross-encoder distillation.
Source: §4 · p. 9Takeaways & Limitations
Model Soup boosts ranking quality while keeping inference latency identical to that of a single model.
Source: §4 · p. 10Claude Code often defaulted to generic solutions, so high-level experiment intuition and direction still relied on the researcher.
Source: §1 · p. 2
Abstract
from arXiv · showhide
This report studies passage retrieval and re-ranking on the MS MARCO benchmark using lightweight Qwen3-0.6B models. Across three experimental settings --- BM25-then-reranking task, corpus-wide dense retrieval task, and dense retrieval then re-ranking task --- the results demonstrate that competitive ranking performance can be achieved without complex architectural modifications, custom loss functions, or cross-encoder distillation. Instead, substantial gains stem from data-centric strategies: corpus-targeted continued pre-training, rigorous data curation, hard-negative mining, training--inference distribution alignment, and weight merging via Model Soup. In particular, Model Soup delivers marked performance gains while preserving the inference efficiency of a single model. These findings highlight how data curation and weight merging can push sub-billion-parameter models to state-of-the-art performance. My models can be downloaded from Hugging Face: https://huggingface.co/shuguangh/models
1 Introduction
MS MARCO passage ranking spans re-ranking retrieved candidates and ranking across the full corpus. This report investigates Qwen3-0.6B ranking, emphasizing data curation and model merging while noting title-preprocessing differences that complicate comparisons.
1 Introduction
- MS MARCO contains 1 million real-world queries and 8.8 million candidate passages, with human judgments identifying zero or more relevant passages per query.The benchmark includes passage re-ranking and full-ranking tasks.Source: §1 · p. 1
- Re-ranking sorts the top 1,000 BM25 passages, whereas full ranking retrieves relevant passages from the entire corpus and typically requires retrieval and re-ranking stages.Full-ranking performance is not bounded by BM25's top-1,000 recall.Source: §1 · p. 1
- Prior MS MARCO work has centered on BERT-family models and specialized pre-training objectives, with newer studies exploring generative models such as T5 and LLaMA.The passage cites BERT [2] [10] [33], RoBERTa [3], ELECTRA [4], ERNIE [12] [27], SimLM [21], CoT-MAE [26], and T5 and LLaMA [5] [6] [7].Source: §1 · p. 1
- Qwen3-0.6B rankers and retrievers achieve state-of-the-art performance through deliberate training-data curation and Model Soup, without specialized objectives, distillation, loss changes, or architectural changes.Model Soup averages weights across fine-tuned checkpoints.Source: §1 · p. 2
- Passage-title use makes published MS MARCO results not directly comparable; appending titles improves retrieval MRR@10 by 2–3 percentage points and re-ranking MRR@10 by 1–2 percentage points.The report reproduces prior baselines with open-source checkpoints for comparison.Source: §1 · p. 2
- Claude Code helped diagnose title-use discrepancies, but its generic solutions meant high-level experiment intuition and direction still relied on the researcher.The author also says the volume of generated code made comprehensive manual correctness checks practically impossible.Source: §1 · p. 2
2 Methodology
The methodology fine-tunes Qwen3-0.6B embedding and reranker models across BM25 and dense-retrieval pipelines. It constructs task-specific training data and evaluates ranking with recall and top-rank metrics.
2 Methodology
- The experiments cover BM25-then-reranking, corpus-wide dense retrieval, and dense-retrieval-then-reranking using the top 1,000 initial candidates for each reranking pipeline.The report focuses on sub-1B models because of GPU resource constraints; larger variants are left for future exploration.Source: §2 · p. 2, §2.1 · p. 2
2.1 Backbone Models
- Qwen3-0.6B-Embedding uses a dual encoder, while Qwen3-0.6B-Reranker uses a cross encoder to process query-passage pairs.Both are initialized from the pre-trained Qwen3-0.6B model; the embedding model supports Matryoshka Representation Learning.Source: §2.1 · p. 2
2.2 Data Preparation
- Training data retains labeled relevant passages and samples negatives: BM25 reranking uses BM25 candidates, while retrieval training mines hard negatives ranked 20–200 by a retrieval model or reranker.Dense-retrieval reranking samples negatives from its trained retriever's top 1,000 outputs.Source: §2.2 · p. 3
2.3 Experiment Setup
- Experiments use Recall@K, MRR@10, and nDCG@10; Recall measures corpus-wide retrieval coverage, while MRR and nDCG quantify top-rank precision.Hardware constraints set reranking sequences to 384 tokens and retrieval query and document limits to 64 and 144 tokens.Source: §2.3 · p. 4
3 Experimental Results
Across BM25 reranking, corpus-wide dense retrieval, and dense-retrieval reranking, Qwen3-0.6B systems improve through continued pre-training, data selection, training–inference alignment, and model merging. The final full-ranking model reaches an MRR@10 of 0.4296.
3.1 Reranking the official BM25 retrieval result (BM25-then-Reranking)
- Target Corpus Pre-training on MS MARCO passages yields incremental reranking gains across training-data scales and supplies the backbone for downstream fine-tuned rerankers.The model uses a causal language-modeling next-token prediction objective on fixed-length 512-token chunks.Source: §3.1 · pp. 4–5, Table 2 · p. 5
- 0.3952 MRR@10 from 500K-query fine-tuning already beats the best non-ensemble baseline; performance trends upward with more queries, while K=14 negatives are optimal.Increasing the negative count beyond 14 degrades metrics.Source: §3.1 · p. 5, Table 3 · p. 5
- Model Soup BR1 surpasses the strongest single-model baseline by 2.74%, while a six-run prediction ensemble performs on par with the TFR-BERT ensemble.The report's combined single-model approach reaches 0.4007 vs. 0.3900 MRR@10 for the previous best single-model baseline.Source: §3.1 · p. 5
- 0.4007 vs. 0.3900 MRR@10: combining target corpus pre-training, data scaling, and model merging beats the previous best single-model baseline.The result also slightly exceeds RankT5's 0.3997 MRR@10.Source: §3.1 · p. 5, Table 4 · p. 6
3.2 Retrieving the whole data corpus (the Dense Retrieval task)
- Hard negatives mined from stronger retrieval models consistently improve MRR@10 and nDCG@10 while Recall@1000 remains relatively stable; FR1-mined negatives score highest across all three metrics.The tested sources include GTR-Large, SPLADE-coCondenser, SimLM, ColBERTv2, and the FR1 cross-encoder reranker.Source: §3.2 · pp. 6–7, Table 6 · p. 7
- 0.9861 Recall@1000 and 0.3952 MRR@10 place dense-retrieval soup DR1 near SOTA; DR1 leads evaluated dual encoders, while late-interaction DR2 scores highest overall.DR1 remains the default full-ranking retriever to preserve simplicity and minimize inference latency.Source: §3.2 · pp. 7–8, Table 7 · p. 8
3.3 Reranking the dense retrieval results (the DS-then-Reranking task)
- Training on dense-retriever candidates improves alignment with inference and raises MRR@10 from 0.4087 to 0.4178 over the BM25-trained counterpart.The training set retains labeled relevant passages and samples 14 non-relevant passages from the dense retriever's top 1,000 outputs.Source: §3.3 · p. 8
- 0.4296 MRR@10 is the final full-ranking model's score, outperforming fully reproduced Table 8 baselines and RankT5-3B's 0.4208.The final model combines a model soup, retriever–reranker score fusion, and an HLATR reranker.Source: §3.3 · p. 9, Table 8 · p. 9
4 Conclusions
Qwen3-0.6B models achieve competitive performance across three MS MARCO ranking settings through data-centric strategies and Model Soup, without complex architectural changes. Model Soup improves ranking quality while preserving single-model inference latency, and title handling remains important for fair evaluation.
4 Conclusions
- Across BM25 reranking, dense retrieval, and dense-retrieval-then-reranking, Qwen3-0.6B achieves competitive ranking without architectural changes, custom losses, or cross-encoder distillation.The report attributes primary gains to continued pre-training, data curation, hard-negative mining, and training-inference alignment.Source: §4 · p. 9
- Model Soup boosts ranking quality cost-efficiently while keeping inference latency identical to that of a single model.The conclusion also identifies scaling to larger Qwen3 models as future exploration.Source: §4 · p. 10
- 2–3 pp in retrieval MRR@10 and 1–2 pp in re-ranking MRR@10 accompany passage-title inclusion; major baselines were re-evaluated under a unified protocol.The report connects these differences to inconsistent title use across prior evaluations.Source: §4 · p. 10
- Claude Code supported engineering and analysis, while key breakthroughs came from human intuition rather than the assistant’s suggestions.The report describes the assistant as effective on engineering-intensive tasks but limited in research taste and high-level ideation.Source: §4 · p. 10