Relevance-guided Supervision for OpenQA with ColBERT

1 Introduction

• OpenQA retrieval must surface passages relevant to answering questions, since a reader may not see the best evidence when retrieval relevance is misaligned with question answering.Prior supervision approaches can require hand-labeled positives, produce weak positives or unchallenging negatives, or incur frequent re-indexing or a frozen document encoder.Source: §1 · pp. 1–2
• ColBERT models fine-grained question–passage interactions by comparing question-term and passage-term embeddings while scaling to millions of documents with low query latency.This contrasts with single-vector retrievers, which estimate relevance using one dot product.Source: §1 · pp. 1–2
• Relevance-guided supervision (RGS) uses a weak retriever and heuristic to label top-k passages, then iteratively trains a more effective retriever without frequent re-indexing or freezing the document encoder.The process is applied two to three times and requires only one or two corpus re-indexings.Source: §1 · p. 2
• ColBERT-QA achieves state-of-the-art retrieval and extractive OpenQA results on Natural Questions, SQuAD, and TriviaQA; RGS improves ColBERT by up to 2.3 and a single-vector retriever by up to 3.2 Success@20 points.The resulting system establishes state-of-the-art retrieval and downstream performance.Source: §1 · p. 2
• The paper systematically compares ColBERT’s fine-grained interactions with single-vector retrievers and reports strong transfer to new OpenQA datasets and gains over existing retrievers.The introduction identifies DPR as an example of a recent single-vector retriever.Source: §1 · p. 2

2.1 Machine Reading Comprehension

• MRC commonly asks a model to answer questions about passages, with answers selected from options or appearing as literal passage substrings.The paper focuses on the extractive version, where the answer is a literal substring of the passage.Source: §2.1 · pp. 2–3

2.3 Retrieval Models for OpenQA

• Single-vector retrievers score independently encoded questions and passages with one dot product, while concatenated-sequence attention models allow richer interactions but are too expensive beyond reranking small candidate sets.ColBERT occupies a middle ground: it separately encodes query and document tokens, then creates late, fine-grained interactions between them.Source: Figure 2 · p. 3, §2.3 · p. 4

2.2 OpenQA

• OpenQA requires retrieving a passage from a large corpus before answering, without a guarantee that the answer appears in retrieved passages.Source: §2.2 · p. 3

2.3 Retrieval Models for OpenQA

• DPR uses labeled evidence when available, BM25-based weak supervision otherwise, and in-batch negatives; this strategy established a new state of the art for extractive OpenQA.Source: §2.3 · p. 3
• The paper distinguishes its work by testing late interaction against concurrent state-of-the-art single-vector OpenQA models and reporting larger gains with RGS than with traditional supervision.It also reports strong results in an out-of-domain transfer setting.Source: §2.3 · p. 4

2.4 Supervision Paradigms in OpenQA

• OpenQA supervision can use gold evidence, but some datasets provide only question–answer pairs, and manual annotations may miss other answer-bearing passages.Source: §2.4 · p. 4
• Weak supervision uses the short answer string to identify candidate positives, often filtering passages retrieved by an existing model or retrieving them within the training loop.Retrieve-and-filter approaches can rely on BM25 ranking, while inner-loop approaches include ORQA, REALM, and RAG.Source: §2.4 · p. 4
• RGS is presented as an alternative to BM25-biased retrieve-and-filter methods and inner-loop strategies that restrict document-encoder training or require frequent re-indexing.It lets the retriever collect training examples while fine-tuning the document encoder and requires only 1–2 additional re-indexings.Source: §2.4 · pp. 4–5

3 ColBERT-QA

• ColBERT-QA uses three fine-tuning stages, producing ColBERT-QA1, ColBERT-QA2, and ColBERT-QA3.RGS is presented as a scalable strategy using the retriever being trained and a weak heuristic to gather examples in discrete rounds.Source: §3 · p. 5

3.1 The ColBERT Model

• ColBERT encodes question and passage tokens separately, then sums each question token’s maximum similarity to any passage token.This fine-grained scoring scales to billions of tokens across millions of passages, allowing retrieval directly from a large corpus.Source: §3.1 · p. 5

3.2 Relevance-Guided Supervision

• RGS retrieves ranked passages in discrete rounds, uses a weak answer-string heuristic to select positives and negatives, and trains the next retriever on those examples.The procedure uses the highest-ranked answer-containing passages among the top k+ as positives and non-answer-containing passages among the top k− as negatives.Source: §3.2 · p. 6
• RGS collects examples outside the training loop, allowing large-depth sampling and full document-encoder fine-tuning without frequent corpus re-indexing.To reduce overfitting, each model is trained on one half of the questions and does not retrieve examples for questions it trained on.Source: §3.2 · p. 6
• Two further RGS rounds produce ColBERT-QA2 and ColBERT-QA3, whose retrieval consistently exceeds ColBERT-QA1 and ultimately surpasses existing retrievers.The procedure bootstraps with BM25 to train ColBERT-QA1, then uses each successive model to supervise the next.Source: §3.2 · p. 6

3.3 Reader Supervision for OpenQA

• For reader training, the retriever supplies question, positive-passage, and negative-passage triples; the reader optimizes the marginal likelihood of answer-matching spans in the positive passage.The procedure can be extended to all training questions by training two independent supervisors on separate halves and combining their rankings; this is left for future work.Source: §3.3 · p. 6

4.1 Methods

• Success@k measures the share of questions with an answer-containing passage retrieved by depth k; reader EM evaluates retriever–reader combinations on the validation set.Success@k is an upper bound for an extractive reader that reads k passages and always finds an answer when present.Source: §4.1 · p. 7

4.2 Baselines

• ColBERT’s fine-grained contextual matching consistently outperforms single-vector representations and the retrieval baselines across Natural Questions, TriviaQA, and SQuAD.The comparison includes a single-vector ColBERT ablation trained in the same manner as ColBERT-QA1 to better account for differences from DPR.Source: §4.2 · p. 8

4.3 Evaluating ColBERT out of domain

• Out-of-domain ColBERT-QA outperforms BM25 and is competitive with DPR across the three datasets.On Natural Questions, its development-set Success@5 exceeds 65%, while zero-shot ORQA and REALM are below 40% on a different random train/dev split.Source: §4.3 · p. 8

4.4 Evaluating Relevance-Guided Supervision

• ColBERT-QA1, trained with BM25-collected examples, already outperforms the baselines considered, including DPR.Its gains over the out-of-domain model are consistent.Source: §4.4 · pp. 8–9
• ColBERT-QA2 and ColBERT-QA3 raise test-set Success@20 over ColBERT-QA1 by 0.9–2.4 points across the three datasets, with gains up to 5.7 points in S@1.These models require one or two additional rounds of indexing and retrieval; ColBERT-QA2 and ColBERT-QA3 are consistently the most effective.Source: §4 · p. 7, §4.4 · p. 9
• On the development set, ColBERT-QA3’s Success@20 falls from 84.3% to 80.7% when reranking BM25’s top 1,000 instead of retrieving from the full corpus.The authors describe full-corpus retrieval as leveraging ColBERT’s scalability to improve recall.Source: §4.4 · p. 9
• RGS improves development-set reader EM by 0.7–2.4 points over ColBERT-QA1, but higher Success@k does not always yield higher EM with a particular reader.The authors hypothesize that a stronger retriever can find more positive passages while also retrieving challenging negatives that may confuse an imperfect reader.Source: §4.4 · p. 9

5.1 Datasets

• The test sets use open versions that discard validation and test evidence passages, with original development data used for testing after a random training-set split.The main evaluation uses the December 2018 Wikipedia dump, with an additional 2016-dump comparison for SQuAD.Source: §5.1 · p. 9

5.1 Datasets; 5.2 Baselines; 5.3 Our Models

• The evaluation reports Exact Match (EM) of gold answers on test sets for the OpenQA versions of NQ, TQ, and SQuAD, comparing heuristic-retrieval approaches with learned vector-based retrieval systems.SQuAD additionally has end-to-end results on the 2016 Wikipedia dump alongside the primary December 2018 dump. ColBERT-QA uses either a BERT-base reader (base) or a BERT-large whole-word-masked reader (large).Source: §5.2 · p. 9, §5.1 · p. 9, §5.3 · p. 10

5.3 Our Models

• ColBERT-QA with a BERT-base reader outperforms existing BERT-base-reader models on all three datasets and surpasses RAG on TriviaQA despite using fewer parameters and extractive reading.The base configuration already reaches state-of-the-art extractive OpenQA performance on TriviaQA and SQuAD.Source: §5.3 · p. 10
• ColBERT-QA with a BERT-large reader establishes state-of-the-art extractive OpenQA performance on Natural Questions, TriviaQA, and SQuAD, outperforming all included baselines, including RAG.Every baseline other than REALM, which lacks results on SQuAD and TriviaQA, trails ColBERT-QA by several EM points on at least one dataset.Source: §5.3 · p. 10

6 Conclusion

• ColBERT-QA is an end-to-end OpenQA system that employs ColBERT to improve both retriever modeling and supervision.Source: §6 · p. 10
• ColBERT-QA improves retrieval quality over existing methods by over five Success@20 points and achieves state-of-the-art extractive OpenQA results across three datasets.The system employs ColBERT to improve retriever modeling and supervision in OpenQA.Source: §6 · pp. 10–11
• Its training strategy progressively improves OpenQA training data without hand-labeled evidence passages.Source: §6 · p. 11

A Datasets

• Natural Questions contains real Google-submitted questions filtered so a Wikipedia article appears among the top five results, and is the main experimental dataset.Its answers are short spans in Wikipedia articles.Source: A Datasets · p. 11
• SQuAD v1.1 consists of crowdworker-written questions about Wikipedia articles.Source: A Datasets · p. 11
• TriviaQA consists of trivia questions originally created by enthusiasts.Source: A Datasets · p. 11
• Each dataset has approximately 79k training and 9k validation questions; NQ has 4k test questions, while SQuAD and TriviaQA have 11k each.The study uses train/validation splits released by Karpukhin et al. (2020), and test sets are the same for all methods.Source: A Datasets · p. 11

B Implementation Details

• The corpus is full English Wikipedia, excluding tables, lists, and disambiguation pages, using the 20 December 2018 preprocessed passages unless stated otherwise.Experiments requiring Wikipedia 2016 use that dump processed in the same manner.Source: B Implementation Details · p. 11
• Passage titles are prepended, and retrieval relevance is based on whether the passage contains the short answer string.Including title matches changes ColBERT-QA3’s Success@20 advantage over DPR by only 0.1–0.6 points, from a reported 5.7–12.2-point gap.Source: B Implementation Details · p. 11
• ColBERT training uses batches of 64 triples, a 3 × 10^-6 learning rate, and 128-dimensional embeddings; checkpoints are selected by Success@20.RGS samples from the top 1000 passages and selects positives from the top 50 when available.Source: B Implementation Details · p. 11
• Table 4 reports NQ validation-set slice sizes, ColBERT-QA3 Success@1, and improvements over ColBERT-QA1, DPR, and BM25.Source: Table 4 · p. 12
• Table 5 reports results from applying RGS to a single-vector ablation of ColBERT.Source: Table 5 · p. 12

C Retrieval Analysis

• ColBERT-QA3 gains 2–7 Success@1 points over ColBERT-QA1, 7–20 over DPR, and 21–35 over BM25 across NQ validation slices.The gains are stable and consistent across the models and slices.Source: C Retrieval Analysis · p. 12
• The largest gains against all three baselines occur on the superlative slice, which contains over 600 queries with comparative markers such as “best” or “largest”.Source: C Retrieval Analysis · p. 12

• The experiment uses end-to-end retrieval rather than BM25 reranking, with in-batch negatives collected per GPU in a four-GPU setting.Source: D RGS for Single-Vector Retrieval · p. 12
• RGS yields +3.2, +1.3, and +2.3 point gains between rounds one and three for the single-vector ablation.Source: D RGS for Single-Vector Retrieval · p. 12
• The single-vector ablation’s absolute results are considerably weaker than ColBERT-QA’s and average 1.0 point below DPR’s core setting.Source: D RGS for Single-Vector Retrieval · p. 12

• Each retriever training and validation round takes 7–8 hours, while passage precomputation and FAISS indexing take approximately 6 hours.Source: E Computational Cost & Latency · p. 12
• Batch retrieval for all Natural Questions train, development, and test questions takes 2–3 hours; preprocessing and postprocessing scripts add a few hours.The scripts leave room for optimization.Source: E Computational Cost & Latency · p. 12
• On NQ-dev, retrieval latency is 71ms per query for ColBERT and 36ms for the single-vector ablation; the research implementations are not production-optimized.Source: E Computational Cost & Latency · p. 12