Efficient Document Re-Ranking for Transformers by Precomputing Term Representations

1 INTRODUCTION

• PreTTR targets the high cost of running large pretrained transformers and can adjust precomputation and compression settings to application needs.The paper notes that transformer models grew from about 110 million to over 8.3 billion parameters in roughly a year.Source: §1 · pp. 1–2
• PreTTR processes documents through the first l transformer layers at index time, then combines their stored term representations with query representations to finish scoring.Training masks query-document attention in those first l layers; the method is called Precomputing Transformer Term Representations (PreTTR).Source: §1 · p. 2
• The compression layer is trained to make attention scores with compressed representations match those of the uncompressed network.The compression function is placed between transformer layers to reduce the storage burden of precomputed term representations.Source: §1 · p. 2
• Up to 42× acceleration on TREC WebTrack 2012 with comparable P@20; compression reduces PreTTR storage by up to 97.5% without substantial ranking degradation.The evaluation covers multiple pretrained transformers on TREC WebTrack 2012 and TREC Robust 2004.Source: §1 · p. 2

2.1 Neural Ranking

• Neural ranking models include representation-focused approaches that compare learned query and document vectors and interaction-focused approaches that learn relevance patterns from their matches.Neural rankers commonly serve as a final-stage function after a less expensive initial ranking stage.Source: §2.1 · p. 2

2.2 Pretrained Transformers for Ranking

• BERT-based methods have been applied to passage reranking, question answering, and document ranking, while Doc2Query uses a transformer at index time to add terms to documents.BERT has also been combined with existing neural ranking architectures and used for sentence-level score aggregation.Source: §2.2 · pp. 2–3

2.3 Neural Network Efficiency

• Neural-network efficiency methods include pruning, weight quantization, and knowledge distillation; unlike pruning, PreTTR leaves the dense network structure intact and precomputes document term representations.PreTTR can also be applied to transformers produced by distillation, while the paper leaves quantization of stored embeddings to future work.Source: §2.3 · p. 3

2.4 Neural Ranking Efficiency

• Prior neural ranking efficiency work includes sparse representations and hashing for interaction models, but those techniques do not directly fit transformers requiring further term-embedding processing.Earlier transformer-ranking speedups either limit network depth or trade away ranking performance; PreTTR instead splits computation between indexing and query time.Source: §2.4 · p. 3

3 MOTIVATION

• The paper treats the final [CLS] representation as the transformer's output used to generate a ranking score, ignoring other token outputs in this setup.The transformer layers operate on token-representation matrices, with self-attention as a common component.Source: §3 · p. 4
• A Vanilla BERT ranker concatenates query and document tokens, passes them through the transformer, then linearly combines the output into a ranking score.The sequence includes [CLS] and [SEP] tokens, and the output uses a tuned combination weight matrix.Source: §3 · p. 4
• State-of-the-art transformer neural rankers process about 50 documents per second on a modern GPU, which the paper describes as impractical for most ad-hoc retrieval tasks.Source: §3 · p. 4
• About 50% of total processing time is spent on attention-related tasks and about 48% on feed-forward computation, motivating reduced traversal of transformer layers.The feed-forward cost is largely attributed to the large intermediate hidden representation for each token.Source: §3 · p. 4

4.1 PreTTR: Precomputing Transformer Term Representations

• PreTTR precomputes document term representations through layer l and combines them with query representations at query time to finish the ranking computation.Training masks query-document attention in the first l layers; documents and queries are separately encoded through those layers before the representations are joined.Source: §4.1 · pp. 4–5
• Because queries are shorter than documents and query representations can be reused across candidate documents, increasing l can reduce query-time layer computation, but l=n yields identical document scores.The possible reduction is bounded: when l=n, document information never contributes to the ranking score.Source: §4.1 · p. 5

4.2 Token Representation Compression

• The compressor reduces each stored token representation to e values, then expands it back to dimension d for the remaining transformer layers.The compressed representation is produced by a learned feed-forward transformation with GELU; the expanded representation uses a second linear transformation and batch normalization.Source: §4.2 · p. 5
• Compression parameters are pretrained to match attention scores between the compressed and original models, then fine-tuned with the ranking network for relevance.The attention-score mean squared error spans layers l+1 through n, and pretraining can use unlabeled text.Source: §4.2 · p. 5

5.1 Datasets

• Experiments use TREC WebTrack 2012 and TREC Robust 2004, with 2012 WebTrack judgments held out for testing and five-fold evaluation for Robust.WebTrack uses 2011 for validation and judgments from 2009, 2010, 2013, and 2014 for training; Robust has 249 queries, with three folds for training, one for validation, and one for testing per iteration.Source: §5.1 · pp. 5–6

5.2 Transformer Networks

• The primary model is pretrained bert-base-uncased Vanilla BERT; generality tests also include RoBERTa-base and DistilBERT-base-uncased.The authors omit large BERT variants, citing marginal ranking gains alongside greater run cost.Source: §5.2 · p. 6

5.3 Training

• Transformer rankers use pairwise softmax loss and Adam at a 2×10−5 learning rate, with random training pairs and validation-based checkpoint selection.The compressor is trained on Wikipedia text from TREC CAR using heading-paragraph pairs and attention-score matching; its training converged by 2M samples.Source: §5.3 · p. 6

5.4 Evaluation

• Ranking is evaluated primarily with P@20, alongside ERR@20 for WebTrack and nDCG@20 for Robust; latency tests use one GeForce GTX 1080 Ti GPU.Latency measurements assume the model and term representations are in main memory but include transfers to and from GPU memory.Source: §5.4 · p. 6

5.5 Baselines

• The unmodified Vanilla transformer is the primary baseline, with tuned BM25, CEDR-KNRM, and Birch also reported in a single-domain setting.CEDR-KNRM's use of representations from every transformer layer entails considerably more storage, so optimizing it is left for future work.Source: §5.5 · p. 6

6.1 Precomputing Transformer Term Representations

• At l=11, WebTrack retains comparable P@20 but loses ERR@20, while Robust falls below BM25; on Robust, nDCG@20 drops already at l=8.On Robust, precomputing the first 10 layers yields comparable P@20 to the base model; reported differences in highest-performing models are not statistically significant.Source: §6.1 · p. 7

6.2 Term Representation Compression

• At e=256 (67% of the original 768 dimensions), compression usually avoids substantial effectiveness loss; Robust drops sharply at e=128 for layers 7–10.The reported differences are generally not statistically significant, and WebTrack has no clear best compression size.Source: §6.2 · p. 7, Table 4 · p. 8
• 5.7TB stores filtered ClueWeb09-B representations at e=128, a 95% space reduction; 16-bit representations further reduce storage to about 2.8TB.The 5.7TB figure follows spam filtering that reduces the collection's storage estimate from about 112TB to 34TB.Source: §6.2 · p. 7

6.3 Re-ranking Efficiency

• 42.0× speedup over Vanilla BERT: PreTTR reranks 100 WebTrack documents in 46 milliseconds at l=11, with comparable P@20 performance.At l=10, speedups are 8.9× on WebTrack and 9.2× on Robust; l=11 performs poorly on Robust.Source: Table 5 · p. 8, §6.3 · p. 8

6.4 Single Layer Ranking (l = 11)

• Single-layer l=11 effectiveness varies by dataset; on web collections, this setting may remain practical where latency matters while preserving high precision among top-ranked documents.The authors attribute the difference between WebTrack and Robust performance to dataset characteristics.Source: §6.4 · p. 9

6.5 PreTTR for Other Transformers

• 56.3× speedup: PreTTR reaches this at l=11 on RoBERTa; DistilBERT reaches 24.1× at l=5, with BERT-like quality and efficiency trends.The reported times are 0.041 seconds per 100 documents for RoBERTa, down from 1.89 seconds, and 0.035 seconds for DistilBERT.Source: §6.5 · p. 9, Table 6 · p. 9

• Up to 42× faster web query execution was achieved with minimal impact on P@20.Source: §7 · p. 9
• 97.5% less storage was required using a pretrained compression/decompression function and 16-bit floating-point arithmetic.Source: §7 · p. 9
• Effectiveness was demonstrated across transformer architectures and on both web and news search.Source: §7 · p. 9
• Future work could test LSH-hashing for advanced networks such as CEDR, which require additional term representations, and investigate the feed-forward step's importance.Source: §7 · p. 9