Source-linked AI summary
Simple Applications of BERT for Ad Hoc Document Retrieval
Wei Yang, Haotian Zhang, Jimmy Lin
TL;DR
Ad hoc retrieval must apply BERT to documents longer than its input limit, while available relevance judgments are generally document-level. The paper uses sentence-level BERT inference with score aggregation and reports the highest known neural average precision on TREC microblog and Robust04 collections, while acknowledging that sentence-level supervision was sidestepped.
Problem
Long documents exceed BERT’s input length, and existing relevance judgments generally do not identify which individual sentences are relevant.
Method
The approach applies BERT inference to document sentences and aggregates sentence scores with the original document score.
Results
The method reports the highest known average precision on TREC Microblog and Robust04 datasets among neural approaches.
Takeaways & Limitations
Sentence-level inference and simple score aggregation provide an effective initial adaptation of BERT for both short social media posts and longer newswire documents.
Takeaways & Limitations
The study sidesteps the lack of sentence-level relevance judgments by fine-tuning on existing sentence-level datasets rather than projecting document labels to sentences.
Abstract
from arXiv · showhide
Following recent successes in applying BERT to question answering, we explore simple applications to ad hoc document retrieval. This required confronting the challenge posed by documents that are typically longer than the length of input BERT was designed to handle. We address this issue by applying inference on sentences individually, and then aggregating sentence scores to produce document scores. Experiments on TREC microblog and newswire test collections show that our approach is simple yet effective, as we report the highest average precision on these datasets by neural approaches that we are aware of.
1 Introduction
BERT’s success in question answering motivates its application to ad hoc document retrieval, where long documents require a sentence-level inference strategy. The paper reports strong neural retrieval results on microblog and newswire collections.
- Neural retrieval commonly reranks candidate documents initially retrieved with bag-of-words term matching.
- BERT’s pretraining has produced impressive gains across NLP tasks and motivated its use for retrieval.
- The paper applies BERT inference to individual document sentences and aggregates sentence scores into document scores.
- The approach achieves the highest reported average precision among known neural approaches on the evaluated TREC datasets.
2 Background and Related Work
Ad hoc document retrieval emphasizes relevance matching, which differs from semantic matching and question answering. Long documents also exceed BERT’s input length, while relevance judgments usually exist only at the document level.
- Ad hoc retrieval ranks corpus documents for a short query using metrics such as average precision.
- Relevance matching depends more heavily on exact-match signals than semantic matching, which requires stronger semantic representations.
- Question answering shares characteristics with retrieval but remains a different task, leading to distinct neural ranking architectures.
- Because relevant passages are usually sparse and labels are document-level, fine-tuning BERT with existing judgments is unclear for long documents.
- The paper avoids this training challenge by aggregating sentence-level scores during inference.
3 Searching Social Media Posts
For microblog retrieval, the method combines an initial Anserini ranking with BERT classification scores. Experiments compare it with neural and bag-of-words systems on TREC Microblog collections and report consistent improvements.
- Microblog retrieval avoids BERT’s long-document limitation because social media posts are short.
- The system uses Anserini with QL and RM3 to retrieve 1,000 candidates before BERT reranking.
- BERT classifies concatenated query-document sequences for binary relevance prediction.
- Training uses cross-year TREC Microblog data, with 10% held out for validation and a learning rate of 3×10−6 for 10 epochs.
- Table 1 reports average precision and precision at rank 30 while comparing BERT with selected neural ranking models.
- The comparison excludes non-neural learning-to-rank methods using manually engineered and non-textual features.
- The BERT-based technique produces substantial and consistent improvements over previous results.
4 Searching Newswire Articles
For longer newswire documents, the paper applies BERT to individual sentences and aggregates the highest-scoring sentences with the original document score. Fine-tuning uses either microblog retrieval data or QA datasets, with Robust04 experiments showing improvements and no benefit from adding a fourth sentence.
- Method: Sentence-level BERT inference adapts retrieval to documents exceeding BERT’s input length by scoring sentences and combining their scores with the original document score.The aggregation weights can be tuned via cross-validation.
- Fine-tuning: Fine-tuning compares microblog retrieval data with the union of TrecQA and WikiQA, contrasting task match against domain similarity.Microblog data matches document retrieval, while QA data uses corpora closer to newswire.
- Experimental setup: Robust04 experiments use the same experimental procedure as Lin (2018), including sentence segmentation and fixed-size chunks when inputs exceed 512 tokens.Across 250 topics, documents average 43 sentences with 27 tokens per sentence.
- Results: The learned w4 value is zero, indicating that additional sentences do not help beyond the top three under the tuning procedure.Consequently, 4S results are omitted from the table.
- Results: Fine-tuning BERT on microblog data is more effective than QA data, suggesting task matters more than document genre; the difference is significant at p < 10−7.The comparison uses a paired t-test in one five-fold setting against BM25+RM3.
- Results: A well-tuned BM25+RM3 baseline already outperforms neural ranking approaches, while the simple BERT reranker yields further significant improvements.This comparison preserves the paper’s reported baseline and conclusion.
5 Conclusions
The paper presents a simple sentence-level adaptation of BERT for document retrieval and reports substantial improvements on social-media and newswire ranking. It acknowledges that the approach sidesteps sentence-level relevance judgments and leaves several extensions for future work.
- Conclusion: Sentence-level inference and simple score aggregation adapt BERT for document retrieval in this preliminary study.The approach is described as the most obvious adaptation explored by the paper.
- Conclusion: The method reports the highest AP scores known to the authors for neural approaches on TREC Microblog and Robust04.The paper notes that non-neural approaches perform better on both tasks.
- Limitations: The approach sidesteps the absence of sentence-level relevance judgments, leaving distant-supervision methods for projecting document labels to sentences as an unexplored next step.The authors characterize the techniques as simple and identify other obvious directions for future work.