PaperlayerView exact revision on arXiv

Source-linked AI summary

SPLADE-v3: New baselines for SPLADE

Carlos Lassance, Hervé Déjean, Thibault Formal, Stéphane Clinchant

arXiv:2403.06789v1cs.IRcs.CL

TL;DR

The report addresses the need to document updated SPLADE baselines after training-structure modifications. It presents SPLADE-v3 models and reports broad effectiveness gains over earlier iterations and BM25, with results that can rival some re-rankers.

  • Problem

    The report addresses the need to document updated SPLADE baselines after simple training-structure modifications, despite limited publication novelty.

    Source: §1 · p. 1

  • Method

    The authors release SPLADE-v3 models using training changes that include cross-encoder ensemble distillation, combined KL-Div and MarginMSE losses, and SPLADE++SelfDistil initialization.

    Source: §1 · p. 1, §2.2 · p. 1, §2.3 · p. 2

  • Results

    SPLADE-v3 is statistically significantly more effective than previous iterations, outperforming BM25 in most query sets, including zero-shot settings, and rivaling some re-rankers.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Variant results expose tradeoffs: SPLADE-v3-Lexical struggles on BEIR, while the query-free SPLADE-v3-Doc remains competitive with dense bi-encoders.

    Source: §4 · p. 3

Abstract

from arXiv · show

A companion to the release of the latest version of the SPLADE library. We describe changes to the training structure and present our latest series of models -- SPLADE-v3. We compare this new version to BM25, SPLADE++, as well as re-rankers, and showcase its effectiveness via a meta-analysis over more than 40 query sets. SPLADE-v3 further pushes the limit of SPLADE models: it is statistically significantly more effective than both BM25 and SPLADE++, while comparing well to cross-encoder re-rankers. Specifically, it gets more than 40 MRR@10 on the MS MARCO dev set, and improves by 2% the out-of-domain results on the BEIR benchmark.

1 Introduction

The report introduces SPLADE-v3 as a new series of models built on simple modifications to SPLADE training. It aims to document the models and provide stronger SPLADE baselines.

1 Introduction

  • The technical report accompanies the release of the latest SPLADE library version.The authors say they have used this version of the code in most of their recent work.Source: §1 · p. 1
  • Simple modifications to the training structure motivate releasing a new series of SPLADE models despite limited novelty for a conventional publication.The report's stated purpose is to document SPLADE-v3 and provide the community with better SPLADE baselines.Source: §1 · p. 1
  • The authors report using the updated code in most of their recent works.Source: §1 · p. 1

2 Better Training

The section describes training changes for SPLADE-v3, including more hard negatives, cross-encoder distillation scores, combined losses, and a different initialization choice.

2.1 Multiple Negatives Per Batch

  • More hard negatives improve results especially in-domain, but add little out-of-domain generalization; the described setup uses 100 negatives per query.They use 50 from the top 50 and 50 sampled at random from the top 1k.Source: §2.1 · p. 1

2.2 Better Distillation Scores

  • Cross-encoder ensembles score each of 500k MS MARCO training queries against 100 negatives; rescored scores affine-match the previous scores’ mean and standard deviation.The authors report that changing the score distribution helps with distillation, especially MarginMSE, but they did not investigate why.Source: §2.2 · pp. 1–2, §2.1 · p. 1

2.3 Two Distillation Losses

  • MarginMSE focused more on Recall and KL-Div more on Precision with extra negatives, so the authors combined them using cross-validated weights for better results.They used λKL = 1 and λMSE = 0.05.Source: §2.3 · p. 2

2.4 Further Fine-Tuning SPLADE

  • Starting from SPLADE++SelfDistil4 and applying the training changes led to better effectiveness than starting from CoCondenser or DistilBERT checkpoints.The authors suggest curriculum learning could explain the result but say it needs further investigation.Source: §2.4 · p. 2

3 A New Baseline, SPLADE-v3

SPLADE-v3 is evaluated across multiple query sets against BM25, SPLADE++SelfDistil, and cross-encoder re-rankers. The comparisons show broad gains over BM25 and the prior SPLADE model, while results against re-rankers vary by model.

3 A New Baseline, SPLADE-v3

  • The base SPLADE-v3 model starts from SPLADE++SelfDistil and uses a mix of KL-Div and MarginMSE, with eight negatives per query sampled from SPLADE++SelfDistil.Experiments use the original MS MARCO collection without titles.Source: §3 · p. 2
  • Evaluation uses a meta-analysis over up to 44 query sets from multiple datasets, measuring nDCG*@10 when both positive and negative judgments exist and otherwise standard nDCG@10.Source: §3 · p. 2
  • Across 44 query sets, SPLADE-v3 significantly improves over BM25 in most, with decreases on Touché-2020 and both TREC-MQ sets; the aggregate effect favors v3 despite lower efficiency.Source: §3 · pp. 2–3
  • Against SPLADE++SelfDistil, only Quora has a significant decrease; most datasets gain, and the overall summary effect favors SPLADE-v3.Source: §3 · p. 3
  • MiniLM reranking has a summary effect close to zero at a 95% confidence interval, while DeBERTaV3 outperforms SPLADE-v3 on most query sets.Both re-rankers rerank the top 50 documents returned by SPLADE-v3; ArguAna is an exception for DeBERTaV3.Source: §3 · p. 3

4 SPLADE-v3-DistilBERT, SPLADE-v3-Lexical and SPLADE-v3-Doc

The paper releases three SPLADE-v3 variants with different query-side computation and efficiency profiles. Their results vary by benchmark: Lexical struggles on BEIR, while Doc is less effective overall but remains competitive with dense bi-encoders given its efficiency.

4 SPLADE-v3-DistilBERT, SPLADE-v3-Lexical and SPLADE-v3-Doc

  • The paper releases three additional SPLADE-v3 variants: DistilBERT, Lexical, and Doc.Their defining choices are, respectively, a DistilBERT starting checkpoint, no query expansion, and no query-side computation.Source: §4 · p. 3
  • SPLADE-v3-Lexical is very effective on MS MARCO and LoTTE but struggles on out-of-domain BEIR.The paper attributes its reduced retrieval FLOPS to removing query expansion.Source: §4 · p. 3
  • SPLADE-v3-DistilBERT is a clear downgrade from BERT, yet remains more effective than the Lexical version on BEIR.It starts training from DistilBERT and has a smaller inference footprint.Source: §4 · p. 3
  • SPLADE-v3-Doc is least effective overall, especially zero-shot, but remains competitive with dense bi-encoders given its efficiency.It starts from CoCondenser, performs no query computation, and requires no query encoding with fewer posting lists to traverse.Source: §4 · p. 3

5 Conclusion

The report releases SPLADE-v3 models and evaluates them against earlier SPLADE iterations, BM25, and re-rankers. It reports significant improvements over previous iterations and says SPLADE-v3 often outperforms BM25 and can rival some re-rankers.

5 Conclusion

  • The report describes the release of SPLADE-v3 models.The authors report that this series is statistically significantly more effective than previous iterations.Source: §5 · p. 4
  • SPLADE-v3 is statistically significantly more effective than previous SPLADE iterations.Source: §5 · p. 4
  • In most query sets, including zero-shot settings, SPLADE-v3 outperforms BM25 and can rival some re-rankers.Source: §5 · p. 4

Related papers

Loading 2403.06789v1…