PaperlayerView exact revision on arXiv

Source-linked AI summary

Dealing with Typos for BERT-based Passage Retrieval and Ranking

Shengyao Zhuang, Guido Zuccon

arXiv:2108.12139v2cs.IR

TL;DR

The paper asks whether BERT-based passage retrievers and re-rankers handle query typos, a specific form of keyword mismatch. It tests these models and introduces typos-aware training; on MS MARCO, this training reduces typo-related effectiveness losses while preserving effectiveness on original queries.

  • Problem

    Whether BERT-based passage ranking handles query typos was unquantified, despite expectations that WordPiece tokenization could represent typo tokens.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate typo-modified queries and train Dense Retriever and BERT re-ranker models with a 50% chance of injecting a synthetic typo into each training query.

    Source: §1 · p. 2, §2.1 · p. 2, §2.2 · p. 3

  • Results

    Typos-aware training reduced MRR@10 losses from 52.3% to 24.3% for Dense Retriever and from 34% to 22.7% for BERT re-ranker, while preserving effectiveness on original queries.

    Source: §1 · p. 2, §4 · p. 4

  • Takeaways & Limitations

    The results support using typos-aware training to improve typo tolerance in these BERT-based methods on MS MARCO passage ranking.

    Source: §1 · p. 2, §6 · p. 5

    BERT re-rankers require multiple query-time inferences, resulting in high latency and confining their use to second-stage re-ranking.

    Source: §1 · p. 2

Abstract

from arXiv · show

Passage retrieval and ranking is a key task in open-domain question answering and information retrieval. Current effective approaches mostly rely on pre-trained deep language model-based retrievers and rankers. These methods have been shown to effectively model the semantic matching between queries and passages, also in presence of keyword mismatch, i.e. passages that are relevant to a query but do not contain important query keywords. In this paper we consider the Dense Retriever (DR), a passage retrieval method, and the BERT re-ranker, a popular passage re-ranking method. In this context, we formally investigate how these models respond and adapt to a specific type of keyword mismatch -- that caused by keyword typos occurring in queries. Through empirical investigation, we find that typos can lead to a significant drop in retrieval and ranking effectiveness. We then propose a simple typos-aware training framework for DR and BERT re-ranker to address this issue. Our experimental results on the MS MARCO passage ranking dataset show that, with our proposed typos-aware training, DR and BERT re-ranker can become robust to typos in queries, resulting in significantly improved effectiveness compared to models trained without appropriately accounting for typos.

1 Introduction

The paper examines whether BERT-based Dense Retrievers and re-rankers withstand query typos, finding that typos reduce effectiveness and that typos-aware training improves robustness. It frames this as a test of whether BERT-based passage ranking handles a specific form of keyword mismatch.

1 Introduction

  • Passage ranking often precedes conversational search and question answering, so ranking quality affects the effectiveness of downstream tasks.Source: §1 · p. 1
  • Traditional TF-IDF and BM25 rely on exact keyword matches, limiting retrieval of relevant passages that use different keywords; BERT-based models use latent embeddings for semantic matching.BERT's WordPiece tokenization also offers a rationale for handling out-of-vocabulary keywords.Source: §1 · pp. 1–2
  • Whether BERT-based passage ranking handles typos remained untested, despite the expectation that WordPiece can represent out-of-vocabulary typo tokens as character-level sub-tokens.Source: §1 · p. 2
  • Typos-aware training makes Dense Retriever and BERT re-ranker models more tolerant to query typos without reducing effectiveness on typo-free queries.The study evaluates these methods on MS MARCO passage ranking.Source: §1 · p. 2
  • The study investigates how typos affect BERT-based Dense Retrievers and re-rankers and how these models adapt to them.Source: §1 · p. 2

2 Methodology

The methodology tests the effects of query typos and typos-aware training on BERT-based retrieval and re-ranking. It creates synthetic typos and exposes models to typo-bearing queries during training.

2 Methodology

  • The research questions cover typo impacts, differences across typo types, and whether typos-aware training improves typo-query effectiveness without hurting typo-free queries.Source: §2 · p. 2

2.1 Synthetic Typo Generation

  • The authors synthetically augment MS MARCO queries because the dataset lacks labels identifying typo queries, enabling control over typo number and type.Source: §2.1 · p. 2
  • Five generators insert, delete, substitute, or swap characters within a query word; only words longer than three characters are modified, with one word changed per query.The swap operations use neighboring characters or adjacent keyboard keys.Source: §2.1 · pp. 2–3

2.2 Typos-aware Training

  • For each training query, an unbiased coin keeps it unchanged or injects one typo from a uniformly sampled generator, exposing models to both forms.The procedure uses a 50% chance of injecting a typo and is intended to encourage invariance to typo types.Source: §2.2 · p. 3

3 Experimental Settings

The experiments use MS MARCO passage ranking and evaluate Dense Retriever and BERT re-ranker effectiveness on original and typo-modified queries. The re-ranker operates on initial rankings from the Dense Retriever.

3.1 Dataset and Evaluation Measures

  • MS MARCO provides 8.8 million passages, about 503,000 training queries, and 6,980 development queries; evaluation uses original and typo-modified queries.MRR@10 measures ranking effectiveness; Dense Retriever Recall@1000 is also reported because retrieval recall affects re-ranking.Source: §3.1 · p. 3

3.2 DR Training Details

  • The Dense Retriever uses a shared-parameter BERT-Siamese encoder, pairwise hinge loss, and MS MARCO training triples.It fine-tunes bert-base-uncased with Adam at 3e-6 for 210K steps on one V100 GPU.Source: §3.2 · p. 3

3.3 BERT re-ranker Training Details

  • The BERT re-ranker fine-tunes bert-large-uncased as a binary classifier on query-passage pairs, with negative pairs sampled from a standard-trained Dense Retriever's top 1,000 passages.Positive-to-negative pairs are sampled at a 1:4 ratio; typos-aware training otherwise uses the same settings as standard training.Source: §3.3 · p. 3

4 Results

Typos significantly reduce retrieval and ranking effectiveness under standard training, while typos-aware training makes both methods more tolerant. The improvement does not significantly affect effectiveness on original queries, and rank-loss patterns vary across queries.

  • Under standard training, typos reduce DR MRR@10 by 52.3% and Recall by 24.3%, and BERT re-ranker MRR@10 by 34%.Source: §4 · p. 4, Table 1 · p. 4
  • Different typo types hurt effectiveness similarly, though DR’s MRR@10 loss is about 5% smaller for RandDelete and BERT re-ranker losses are generally uniform.Source: §4 · p. 4
  • MRR@10 losses halve for DR (52.3% to 24.3%) and fall for BERT re-ranker from 34% to 22.7% with typos-aware training.Source: §4 · p. 4, Table 1 · p. 4
  • On original queries, typos-aware training does not produce statistically significant effectiveness differences from standard training.Source: §4 · p. 4
  • Typos-aware training consistently reduces DR rank loss across queries; roughly 300 queries instead gain when typos affect non-essential keywords.Source: §4 · p. 4, Figure 1 · p. 4

5 A Case Study

The case study examines a genuine typo query in MS MARCO, where typos-aware training moves the relevant passage into the top 127 results. The authors note that broader testing on real-world typo queries remains future work.

  • MS MARCO contains very few genuine user typo queries, so this example is a limited real-world test.Source: §5 · pp. 4–5
  • The authors suggest synthetic typo training may generalize to real-world errors, and propose testing this with more real typo queries and relevance annotations.Source: §5 · p. 5

6 Conclusion

The paper finds that typos substantially hurt BERT-based retrieval and re-ranking, and that exposing both methods to typos during training improves their tolerance. The authors propose this lightweight augmentation as a training step that can improve typo-query effectiveness without hurting original-query effectiveness.

  • BERT-based DR and re-ranker are not tolerant to query typos when relying solely on the BERT encoder.Source: §6 · p. 5
  • Typos-aware training exposes DR and BERT re-ranker to typo queries and makes both methods more tolerant.Source: §6 · p. 5
  • The authors propose typos-aware training as a standard augmentation step, reporting light typo-generation computations and no effectiveness loss on queries without typos.Source: §6 · p. 5

Related papers

Loading 2108.12139v2…