DoPR: Reusable Compressed Document Prefixes for Efficient LLM Reranking

1 Introduction

• Pointwise reranking supports parallelization, batching, integration with candidate-generation systems, and fine-grained query-document interactions, but repeatedly re-encodes documents retrieved for different queries.This redundant computation can bottleneck LLM rerankers under strict latency and memory constraints.Source: §1 · p. 1
• Prior efficiency methods reduce per-instance inference costs or precompute representations for other matching functions, leaving document-side reuse inside pointwise LLM reranking unresolved.The cited approaches include prompt compression, token pruning, alternative reranking paradigms, succinct document representations, precomputed term representations, and late-interaction retrieval (Mu et al., 2023; Chevalier et al., 2023; Kim et al., 2022; Cohen et al., 2022; MacAvaney et al., 2020; Khattab and Zaharia, 2020).Source: §1 · p. 1
• DoPR selects salient document representations, converts them into compressed prefix states offline, and reuses them while online scoring processes only the query and scoring token.This combines compact document prefixes with cross-query document reuse.Source: §1 · pp. 1–2
• Across TREC DL, BEIR, and BRIGHT, DoPR retains competitive effectiveness while substantially reducing online memory footprint and inference latency.It shows stronger effectiveness retention on larger backbones in standard reranking, with efficiency benefits most pronounced for long or repeatedly retrieved documents.Source: §1 · p. 2

2.1 Compression for Efficient Reranking

• Compression methods shorten online inputs, but their representations are usually tied to a query, prompt, or reranking context rather than reusable across queries.Examples include passage-embedding rerankers PE-Rank (Liu et al., 2025a) and E2Rank (Liu et al., 2025b), prompt and context compression, and token pruning or sparse attention.Source: §2.1 · p. 2

2.2 Reusable Document-side Representations

• Prior document-side precomputation reduces repeated processing, but DoPR applies query-independent prefix-state reuse inside a pointwise LLM reranker.Earlier approaches include succinct document representations (Cohen et al., 2022), precomputed term representations (MacAvaney et al., 2020), and ColBERT’s contextualized document tokens (Khattab and Zaharia, 2020).Source: §2.2 · p. 2

3.1 Problem Formulation

• DoPR replaces repeated online processing of full documents with selected document representations and reusable prefix states, while conventional pointwise reranking jointly encodes each full query-document pair.At inference, the online input contains only the query and score token; the stored prefix supplies document information.Source: §3.1 · pp. 2–3, §3 · p. 2

3.2 Training Phase

• DoPR trains document representation selection and relevance scoring end to end, using a strict document-to-query bottleneck and pairwise RankNet loss.The framework selects salient document token states, then optimizes the resulting representations for ranking.Source: §3.2 · p. 3

3.2.1 Attention-based Key Token Selection

• Attention concentration provides a lightweight, non-trainable salience signal for selecting top-Ktrain document token states as the sole document-side information path during training.The score uses top-p attention values in each token's attention column; selected states form Cd.Source: §3.2.1 · pp. 3–4

3.2.2 Structured Attention Mask

• A structured attention mask makes Cd the only route through which query and score tokens can access document information during training.Raw document tokens are encoded independently, aligning training with deployment where they are unavailable to the query and score tokens.Source: §3.2.2 · p. 4, Figure 2 · p. 4

3.2.3 Training Objective

• The score-token hidden state feeds a lightweight MLP, and pairwise RankNet training backpropagates through query interactions and selected document states to optimize them for ranking.The objective uses positive and negative candidate documents for a query.Source: §3.2.3 · p. 4

3.3 Offline Compression

• Offline, DoPR selects top-Kinfer salient document positions, converts their representations into multi-layer key-value prefix states, and stores them for deployment.Because the stored states are query-independent, Kinfer can be chosen independently of Ktrain to trade ranking quality against online cost.Source: §3.3 · p. 4

3.4 Online Reranking

• Online, DoPR retrieves each document's stored prefix states and processes only the query and score token, reducing the document-side budget from n tokens to Kinfer ≪n states.Efficiency gains come from both document-side compression and reuse of prefix states across queries.Source: §3.4 · p. 4

4.4.1 Results on TREC DL and BEIR

• On TREC DL and BEIR, DoPR retains 97.1%, 99.3%, and 99.5% average NDCG@10 of matched Qwen3-Rerank at 0.6B, 4B, and 8B, respectively.At 0.6B, drops are larger on Touche, Signal, and Robust; at 4B and 8B, the effectiveness gap narrows and DoPR slightly outperforms the baseline on several datasets.Source: Table 1 · p. 5, §4.3 · p. 5, §4.4.1 · p. 6

4.4.2 Results on BRIGHT

• On BRIGHT, DoPR retains 98.6%, 99.1%, and 98.8% of average NDCG@10 at 0.6B, 4B, and 8B, respectively.Retention is not monotonic with model size; DoPR matches or exceeds the full-document baseline on StackOverflow and Pony but drops on Biology and LC at some scales.Source: Table 2 · p. 6, §4.4.2 · p. 6

4.5.1 Online Reranking Efficiency

• DoPR yields 1.21×–1.84× speedups on DL19 and 2.85×–8.04× on Covid, with larger gains on the longer documents.Online document-side memory falls in proportion to the prefix compression rate; latency gains vary with document length and model scale.Source: §4.5.1 · p. 6, Table 3 · p. 7

4.5.2 Controlled Cross-query Reuse Analysis

• Under controlled reuse, amortized speedup increases with reuse count and grows more quickly on Covid than DL19.The one-time offline prefix-construction cost is shared across queries, while longer Covid documents make repeated full-document reranking more expensive.Source: §4.5.2 · pp. 6–7, Figure 3 · p. 7

4.6.1 Effect of the Training Prefix Ktrain

• Larger training prefix budgets generally improve reranking effectiveness, while very small budgets substantially decrease performance; the default is Ktrain = 32.Models are evaluated with inference budgets matched to their training prefix setting.Source: Table 4 · p. 7, §4.6.1 · pp. 7–8

4.5.3 Effectiveness and Efficiency Trade-off

• Increasing Kinfer generally improves ranking performance but increases memory footprint and latency; DoPR-8B approaches Qwen3-Rerank-8B effectiveness at substantially lower per-query latency.On DL19, DoPR outperforms the remaining compared methods in nDCG@10; listwise rerankers are excluded because their latency is not directly comparable to pointwise reranking.Source: §4.5.3 · p. 7, Figure 4 · p. 8, Figure 5 · p. 8

4.6.2 Effect of Compression-free Warmup

• Compression-free warmup improves performance over activating compressed prefixes from the start; a moderate period works best, so the default is 200 steps.Warmup first trains the model as a full-document reranker before enabling the prefix bottleneck; longer warmup does not consistently add gains.Source: Table 5 · p. 8, §4.6.2 · p. 8

4.6.3 Effect of Prefix Construction Strategy

• Attention-based Top-K achieves the best overall performance among the tested query-independent selection strategies without adding trainable parameters.The comparison uses the same representation budget and reranking framework across First-K, Uniform-K, Random-K, and attention-based Top-K; Top-K is adopted by default.Source: Table 6 · p. 8, §4.6.3 · p. 8

• Across TREC DL, BEIR, and BRIGHT, DoPR retains 97.1%–99.5% of matched full-document rerankers’ average NDCG@10, reduces online document-side memory by 8.0×, and achieves up to 8.04× latency speedup.Source: §5 · p. 8
• DoPR selects query-independent document representations and converts them into compressed document prefix states, moving reusable document-side computation offline for reuse during online reranking.Source: §5 · p. 8
• These findings suggest that reusable compressed prefix states can amortize document-side computation in pointwise LLM reranking.Source: §5 · p. 8

• When documents change rapidly or are rarely retrieved repeatedly, DoPR’s cross-query reuse benefit is limited, though compression still reduces the online document budget.Source: Limitations · p. 9
• DoPR reduces online reranking latency but requires collection preprocessing and compressed-prefix storage, adding potential indexing and storage overhead at scale.Source: Limitations · p. 9

Ethical considerations

• DoPR introduces no new data collection procedures and requires no additional user information beyond standard retrieval inputs.Source: Ethical considerations · p. 9
• DoPR may inherit backbone-LLM and training-data biases that affect document visibility across domains or user groups.Source: Ethical considerations · p. 9
• Compressed document prefixes should receive the original collection’s access controls and privacy protections, with careful auditing before high-stakes deployment.This is especially important for sensitive or proprietary documents.Source: Ethical considerations · p. 9

A.1 Training Dynamics with Different Compression Budgets

• Training loss decreases consistently across different Ktrain values, suggesting stable optimization across a broad range of compression budgets.Figure 6 shows the loss curves under the different budgets; all runs use the same training setup.Source: §A.1 · p. 11, Figure 6 · p. 11
• The analysis varies Ktrain, the number of compressed document prefixes, and omits evaluation metrics to focus on optimization dynamics.The plotted training-loss curves are for the 0.6B model and share the same training setup.Source: §A.1 · p. 11

A.2 Additional Experimental Settings

• DoPR uses a default training prefix budget of Ktrain = 32, while Kinfer follows the dataset-specific document-length budget in Table 8.These settings are shared by DoPR and matched Qwen3-Rerank baselines unless otherwise specified.Source: §A.2 · p. 11
• Selected document representations are inserted as prefix inputs to the reranker.Source: §A.2 · p. 11
• The resulting prefix states are stored as multi-layer cached key-value states.Source: §A.2 · p. 11

A.3 Dataset-specific Compression Settings

• DoPR uses a fixed 8.0× compression ratio, retaining 12.5% of original document tokens while adapting absolute prefix length to each dataset’s document budget.The proportional setting keeps compression strength consistent across tasks despite heterogeneous input lengths.Source: §A.3 · p. 11, Table 8 · p. 12

A.4 Efficiency Across Datasets and Model Scales

• DoPR consistently reduces online cache memory and latency across datasets and Qwen3 scales, with larger relative gains on longer-document datasets.The configured document budget is distinct from realized average online memory and latency; those metrics report document-side footprint and per-document inference time, respectively.Source: §A.4 · p. 11, Table 9 · p. 13

A.5 Generalization Across Model Families

• DoPR retains 97.48% of the full Llama-3.2-1B reranker’s average NDCG@10 across TREC DL and BEIR benchmarks (Grattafiori et al., 2024).The full-document reranker and DoPR use the same training data, candidate sets, and evaluation protocol.Source: §A.5 · p. 11, Table 10 · p. 13

A.6 Expanded Out-of-Domain Evaluation

• DoPR retains 96.46% of the matched full-document reranker’s average NDCG@10 across 12 BEIR subsets.TREC DL is in-domain, while BEIR and BRIGHT test out-of-domain transfer without target-domain adaptation; prefixes use no target-collection queries or statistics, though compression parameters are learned from source-domain query–document pairs.Source: §A.6 · pp. 11, 14, Table 11 · p. 14

A.7 KV-State Reuse and Last-K Selection

• Attention-guided Top-K achieves the best results on DL19 and DL20, while Last-K selection consistently outperforms Raw Last-K KV reuse.The variants are independently retrained with identical training data and the same K-state storage budget; Last-K selection retains DoPR’s prefix construction but uses the final K document states.Source: §A.7 · p. 14, Table 12 · p. 14

A.8 Efficiency on BRIGHT

• 98.6% of average NDCG@10 is retained across 12 BRIGHT subsets, with an 8.0× smaller average document-state footprint and 4.89× average latency speedup.On the 1024-token subsets, speedups range from 5.02× to 5.99×.Source: §A.8 · p. 14, Table 13 · p. 15

A.9 Larger Rerankers Under Comparable Online Budgets

• DoPR-8B achieves slightly higher effectiveness than Qwen3-Rerank-0.6B on both datasets, with comparable or lower latency and a smaller document-state footprint.This comparison concerns online document processing; DoPR-8B still requires substantially more memory for model parameters.Source: §A.9 · p. 14, Table 14 · p. 15