Source-linked AI summary
PARADE: Passage Representation Aggregation for Document Reranking
Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, Yingfei Sun
TL;DR
Long documents exceed pretrained transformers’ sequence limits, and passage-score aggregation may miss relevance signals distributed across a document. PARADE aggregates passage representations and compares these strategies; representation aggregation generally outperforms score aggregation, with the preferred approach varying across collections.
Problem
Long documents can contain relevance signals at different positions and in different passages, while common maximum-passage scoring may not capture them all.
Source: §1 · p. 1Method
PARADE studies aggregating passage representations into document scores and evaluates CNN- and Transformer-based aggregators alongside strategies for improving efficiency.
Source: §1 · pp. 1–2Results
Representation aggregation with CNNs or Transformers outperforms passage-score aggregation overall, while PARADE–Max outperforms PARADE–Transformer on TREC DL and Genomics.
Source: §1 · p. 1, §5.4 · p. 9Takeaways & Limitations
The authors’ results indicate that incorporating diverse relevance signals from full text matters for ad-hoc ranking rather than relying on a single passage.
Source: §6 · p. 9Pretrained language models have fixed sequence-length limits, so long documents cannot be processed directly as a single sequence.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Pretrained transformer models, such as BERT and T5, have shown to be highly effective at ad-hoc passage and document ranking. Due to inherent sequence length limits of these models, they need to be run over a document's passages, rather than processing the entire document sequence at once. Although several approaches for aggregating passage-level signals have been proposed, there has yet to be an extensive comparison of these techniques. In this work, we explore strategies for aggregating relevance signals from a document's passages into a final ranking score. We find that passage representation aggregation techniques can significantly improve over techniques proposed in prior work, such as taking the maximum passage score. We call this new approach PARADE. In particular, PARADE can significantly improve results on collections with broad information needs where relevance signals can be spread throughout the document (such as TREC Robust04 and GOV2). Meanwhile, less complex aggregation techniques may work better on collections with an information need that can often be pinpointed to a single passage (such as TREC DL and TREC Genomics). We also conduct efficiency analyses, and highlight several strategies for improving transformer-based aggregation.
1 INTRODUCTION
Transformer models rank passages effectively but cannot process long documents in one sequence, so passage signals must be combined into document scores. PARADE studies representation aggregation as a way to preserve broader document-level evidence and compares its effectiveness and efficiency.
1 INTRODUCTION
- Relevant excerpts may occur at different document positions, and passage order may also affect document relevance, so top-passage scores may miss full-document signals [36].Prior empirical studies found that more relevant documents contain more relevant passages and that passage-score aggregation can outperform using the maximum passage score [20].Source: §1 · p. 1
- Non-relevant content can itself signal relevance because adding it should lower a document’s score under LNC1.Applying document-level labels to individual passages can also introduce unnecessary training noise.Source: §1 · p. 1
- Aggregating passage representations with CNNs or Transformers outperforms passage-score aggregation, and knowledge distillation can make representation aggregators smaller and more efficient.The introduction also identifies analyses of passage count and dataset characteristics as contributions.Source: §1 · pp. 1–2
- The paper formalizes passage-score and passage-representation aggregation strategies and shows how they can be trained end-to-end.Source: §1 · p. 1
- The study compares aggregation strategies across benchmarks and examines model size, passage count, and dataset characteristics as factors in effectiveness and efficiency.Source: §1 · p. 2
2 RELATED WORK
Prior work applies contextualized language models to ranking, passage scoring, and representation aggregation, while this paper explores more sophisticated passage-representation aggregators for document ranking. Related efficiency research includes compact models, distillation, and longer-context attention.
Contextualized Language Models for IR
- BERT cross-encoders have been used for passage reranking [56], while BERT-MaxP and Birch aggregate passage-level scores [17].CEDR also uses representation averaging, and this work considers an improved CEDR variant as a baseline [53].Source: §2 · p. 2
- Efficiency research reduces model size, precomputes representations, builds sparse representations, or reduces Transformer layers [16] [58] [42] [51] [52] [32] [54].Source: §2 · p. 2
- Long-context attention models include Sparse Transformer and Longformer [11] [4], and QDS-Transformer adapts Longformer for ranking [38]; representation aggregation is reported as more effective than increasing input text size and can be combined with these models.Source: §2 · p. 2
Passage-based Document Retrieval
- Earlier work showed passage-relevance representation aggregation can perform well with pre-BERT models [20], and other approaches aggregate evidence or model passage importance.The cited cumulative-gain model uses passage-level labels and an LSTM to predict passage gains, with the final passage’s gain serving as document gain.Source: §2 · p. 2
Representation Aggregation Approaches for NLP
3 METHOD
PARADE estimates document relevance by aggregating passage-level relevance representations into a document representation and then producing a score. The method contrasts score pooling with representation aggregation and describes several aggregation designs and their computational costs.
3 METHOD
- PARADE estimates a document’s relevance by aggregating passage-level relevance representations into a document representation used to produce a score.Source: §3 · p. 3
3.1 Creating Passage Relevance Representations
- Because BERT has a fixed sequence-length limit, the document is split into overlapping 225-token passages with a stride of 200 and each passage is processed by BERT [17].The paper notes that BERT is used as the representative PLM and that the same length limitation applies to most PLMs.Source: §3.1 · p. 3
3.2 Score vs Representation Aggregation
- Score aggregation first predicts passage scores and pools them, whereas PARADE directly aggregates passage representations into an overall document representation.The cited prior score-aggregation methods include BERT-MaxP and Birch [17].Source: §3.2 · p. 3
3.3 Aggregating Passage Representations
- PARADE’s aggregators summarize passage [CLS] representations into one dense document representation; variants include elementwise max, learned attention weights, equal-weight sum or length-normalized average, hierarchical CNNs, and Transformers.PARADE–Max pools representations rather than passage relevance scores; the Transformer models passage ordering and dependencies, while CNN layers combine non-overlapping pairs.Source: §3.3 · pp. 3–4
3.4 Generating the Relevance Score
- Most PARADE variants apply a single-layer feed-forward network to the final document representation to generate a relevance score.PARADE–CNN instead scores each CNN representation with a one-hidden-layer feed-forward network and sums those scores.Source: §3.4 · p. 4
3.5 Aggregation Complexity
- Passage processing dominates computational complexity; Max, Attn, and Sum aggregation are inexpensive, while CNN and Transformer aggregation networks are shallower than the passage-modeling Transformer in practice.Source: §3.5 · p. 4
4 EXPERIMENTS
The experiments compare passage-aggregation approaches across several ranking collections and evaluate their training and reranking setup. PARADE’s strongest variants perform well on Robust04 and GOV2, while simpler aggregation is more effective on Genomics and TREC DL.
4.1 Datasets
- The experiments cover Robust04, GOV2, Genomics, MSMARCO document ranking, and ClueWeb12-B13; GOV2 and Genomics documents are much longer than Robust04 documents.Robust04 and GOV2 include keyword and description queries; MSMARCO document labels are transferred from passage labels, so a relevant passage makes its document relevant.Source: §4.1 · p. 4
4.5 Main Results
- PARADE–CNN and PARADE–Transformer consistently outperform the four other PARADE variants on Robust04 and GOV2, although comparisons among simpler variants are mixed on GOV2.PARADE–Transformer also significantly outperforms Birch and ELECTRA-MaxP on most metrics on both collections; it matches zero-shot T5-3B on Robust04 using 4% of its parameters.Source: Table 2 · p. 5, §4.5 · p. 6
4.2 Baselines
- Birch-Passage is trained end-to-end on passages and the target corpus, while ELECTRA-MaxP uses the maximum passage score as its document score.The baseline comparison also includes kernel-pooling and CEDR-KNRM variants, plus zero-shot T5-3B results.Source: §4.2 · p. 5
4.3 Training
- Documents are split into at most 16 overlapping passages, and ranking is trained on top-1,000 retrieved documents using a pairwise hinge loss.Longer documents retain the first and last passages and uniformly sample the remainder; reported test results average five cross-validation folds, with reranking limited to 1,000.Source: §4.3 · pp. 5–6, §4.4 · p. 6
4.5 Main Results
- On Genomics, neural methods generally underperform BM25 and the best TREC submission; PARADE–Max is the strongest neural method, and significance is difficult to establish across 64 queries.This contrasts with Robust04 and GOV2, where hierarchical aggregation performs strongly.Source: Table 3 · p. 6, §4.5 · p. 6
4.4 Evaluation
- Effectiveness is measured using MAP, Precision, ERR, and nDCG at different cutoffs; NTCIR WWW-3 uses NTCIREVAL.Source: §4.4 · p. 6
4.6 Results on the TREC DL Track and NTCIR WWW-3 Track
- In TREC DL’19, all PARADE variants beat ucas_runid1 and TUW19-d3-re on nDCG@10 but not idst_bert_r1; in TREC DL’20, PARADE–Max beats PARADE–Transformer.A fair comparison with idst_bert_r1 was not possible because its pre-trained structBERT model was unavailable.Source: Table 4 · p. 6, §4.6 · pp. 6–7
- On NTCIR WWW-3, PARADE–Transformer is comparable to KASYS-E-CO-NEW-1 and substantially outperforms PARADE–Max.KASYS-E-CO-NEW-1 is a Birch-based method using BERT-Large.Source: Table 5 · p. 7, §4.6 · p. 7
5 ANALYSIS
The analysis compares PARADE with long-text transformers, studies efficiency and passage-count choices, and examines when representation aggregation is preferable. Across collections, the advantage of more complex aggregation varies with how many relevant passages documents contain.
5.1 Comparison with Long-Text Transformers (RQ1)
- Long-text transformer approaches achieve similar effectiveness but underperform PARADE–Transformer; the comparison is qualified because their CLS features feed a downstream model rather than directly predicting relevance.The authors say a larger study using similar configurations is needed to draw conclusions.Source: §5.1 · p. 7
5.2 Reranking Effectiveness vs. Efficiency (RQ2)
- Reducing BERT model size monotonically reduces effectiveness, and hidden-layer size matters more for performance than the number of layers.The analysis varies model sizes and reports parameter counts and inference time to examine complexity–effectiveness trade-offs.Source: §5.2 · p. 7, Table 7 · p. 8
- 11% fewer parameters let 10-layer PARADE outperform its teacher, while the 8-layer model is comparable and distillation raises nDCG@20 across distilled models.BERT-Small PARADE achieves nDCG@20 above 0.5, outperforming BERT-MaxP using BERT-Base, with 1.14 ms inference per document.Source: §5.2 · p. 8, Table 7 · p. 8
5.3 Number of Passages Considered (RQ3)
- More passages generally improve PARADE–Transformer nDCG@20, while PARADE–Max and PARADE–Attn degrade slightly at 64 passages.Increasing passages also increases computation; training on more passages than used at inference can generally yield a small nDCG increase.Source: §5.3 · pp. 8–9, Figure 3 · p. 8, Table 8 · p. 8
5.4 When is the representation aggregation approach preferable to score aggregation? (RQ4)
- PARADE–Max performs better on TREC DL and Genomics, where most documents contain one relevant passage and the vast majority contain one or two.The authors relate this pattern to the reduced importance of aggregating relevance across passages when relevant-passage counts are low; they note that DL passage judgments may be incomplete.Source: §5.4 · p. 9
6 CONCLUSION
The conclusion presents PARADE as an end-to-end document reranker and emphasizes incorporating relevance signals from full text. It also reports that distillation boosts smaller models and that dataset characteristics help identify when representation aggregation is more effective.
6 CONCLUSION
- PARADE is an end-to-end document reranking model evaluated on ad-hoc benchmark collections.The paper frames its results as indicating the importance of incorporating diverse relevance signals from full text rather than relying on a single passage.Source: §6 · p. 9
- Knowledge distillation boosts smaller PARADE models while substantially reducing their parameters.Source: §6 · p. 10
- The paper analyzes dataset characteristics to explore when representation aggregation strategies are more effective.Source: §6 · p. 10
A.1 Results on the TREC-COVID Challenge
The TREC-COVID results vary across rounds: PARADE achieves the highest nDCG in Round 2, is less competitive in Round 3, and improves in Round 4 but is surpassed by other approaches.
A.1 Results on the TREC-COVID Challenge
- PARADE used question topics because they worked much better than the other topic types; all rounds used PARADE–Transformer.Source: §A.1 · p. 12
- In Round 2, PARADE achieves the highest nDCG; its nDCG improves in Round 4 but is outperformed by other approaches.The Round 4 comparison includes an ensemble run comprising more than 20 runs, whereas PARADE runs were single models except for the Round 3 fusion run.Source: §A.1 · p. 12, Table 10 · p. 12, Table 12 · p. 12
- Round 3 runs are less competitive than in Round 2; one possible reason is that the collection doubled between rounds, increasing potential training–testing inconsistencies.A PARADE run performing poorly by nDCG was comparable to other variants on bpref, which considers judged documents only.Source: §A.1 · p. 12, Table 11 · p. 12