PaperlayerView exact revision on arXiv

Source-linked AI summary

RidgeRank: Efficient Visual Document Reranking via Score Fusion and a Shallow Linear Readout

Shubing Yang, Dongfang Zhao

arXiv:2609.34192v1cs.IR

TL;DR

Visual document rerankers can improve retrieval but are costly at full depth, and some compression methods rely on relevance labels to recover accuracy. RidgeRank fuses retriever and reranker scores using a correlation-derived weight and calibrates a shallow readout to the model’s own full-depth scores. Across ViDoRe 2 and ViDoRe 3, it comes within 1.2 pp NDCG@5 of a full cross encoder at up to 48 times the speed.

  • Problem

    Full-cost multimodal reranking is slow, while some compressed rerankers require relevance labels to recover accuracy and use reranker scores alone.

    Source: §1 · p. 1

  • Method

    RidgeRank derives a closed-form fusion weight from score correlations and fits a shallow linear readout to the frozen model’s own uncompressed full-depth scores.

    Source: §1 · p. 1

  • Results

    Within 1.2 pp NDCG@5 of a full cross encoder at up to 48 times the speed across ViDoRe 2 and ViDoRe 3.

    Source: §5 · p. 10

  • Takeaways & Limitations

    RidgeRank combines retriever and reranker scores and approaches full cross-encoder NDCG@5 while substantially reducing latency on the evaluated benchmarks.

    Source: §5 · p. 10

    The linear approximation is empirical and need not describe every possible shallow state.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Multimodal language models rerank visual document retrieval results accurately, but scoring every candidate page at full cost makes them slow. Some methods that compress these rerankers need relevance labels to regain accuracy, and they rank by the reranker score alone. RidgeRank measures how much relevance signal the reranker score lacks and recovers it from the retriever score through a closed-form fusion rule. Maximizing a correlation objective gives the optimal fusion weight, along with the exact condition under which the reranker score by itself cannot reach that optimum. The reranker is further corrected by a single vector applied to an intermediate hidden state, obtained through one centered ridge regression onto the same model's full-depth scores on uncompressed pages. On 12 datasets drawn from ViDoRe 2 and ViDoRe 3, evaluated with two retrievers and two language model backbones, RidgeRank brings NDCG@5 to within 1.2 pp of a full cross encoder with speedups of up to 48 times, advancing the accuracy and latency Pareto frontier for visual document reranking.

1 Introduction

Visual document rerankers can improve ranking but full-image scoring is costly, while some compression methods need relevance labels to recover accuracy and rely on reranker scores alone. RidgeRank combines a closed-form score-fusion rule with a shallow readout calibrated to the model’s own full-depth scores, and approaches full cross-encoder accuracy across 12 datasets at substantially lower latency.

1 Introduction

  • Visual document rerankers improve retrieval quality, but full-image scoring of every candidate is costly despite low-latency needs in document question answering and generation.Applications include biomedical lectures, economics reports, and ESG disclosures.Source: §1 · p. 1
  • Some acceleration methods need relevance labels to recover compression-related accuracy and rank by reranker score alone, motivating calibration and retriever-score fusion.The cited approaches cache prefixes, stop at intermediate layers, or subsample or quantize visual positions.Source: §1 · p. 1
  • A closed-form correlation-based fusion rule measures relevance signal missing from the reranker and supplies it from the retriever, with an exact condition for when the reranker alone cannot attain the optimum.The operating point depends on each score’s correlation with relevance and the correlation between the scores.Source: §1 · pp. 1–2
  • A single centered ridge solve fits a shallow-state vector to reproduce the model’s uncompressed full-depth scores, using the model’s own scores without relevance judgments or gradient updates.Centering lets the fit capture within-query candidate differences, while the backbone remains frozen.Source: §1 · pp. 1–2
  • Within 1.2 pp NDCG@5 of a full cross encoder at up to 48 times lower latency across 12 ViDoRe 2 and ViDoRe 3 datasets, two retrievers, and two backbones.The evaluated backbones are Qwen2.5-VL-7B and GLM-4.1V-9B.Source: §1 · p. 2

2 Related Work

Prior work accelerates multimodal reranking through caching, early stopping, and compressed visual inputs, while other approaches combine first-stage and neural scores or fit layerwise readouts. RidgeRank instead analyzes the retriever’s remaining relevance signal and fits one closed-form readout to the frozen reranker’s own uncompressed score.

2 Related Work

  • RidgeRank characterizes relevance information discarded by the reranker but retained by the retriever for a fixed candidate set.Prior multimodal systems commonly replace retriever scores with pointwise reranker scores; other methods combine retrievers or feed reranker scores back into query representations.Source: §2 · p. 2
  • Existing efficiency methods cache query-independent inputs, stop forward passes early, compress stored visual states, or score candidate lists together.RidgeRank adopts these techniques unchanged and claims none of them as a contribution.Source: §2 · p. 2
  • Unlike collection-specific, label-tuned fusion and gradient-trained layerwise readouts, RidgeRank derives an optimal fusion point and fits one vector to the frozen reranker’s own uncompressed score.The readout compensates for reduced depth and compressed input at once.Source: §2 · pp. 2–3

3 Method

RidgeRank reranks a fixed retrieved candidate set by fusing standardized retriever and reranker scores, while a shallow linear readout approximates the frozen model’s uncompressed score. Its fusion weight is derived from score correlations, and its readout is fitted by centered ridge regression on unlabeled query-candidate lists.

3 Method

  • The retrieved candidate set remains fixed; RidgeRank changes only its ordering, using a shallow readout and retriever scores.The method stores query-independent page and instruction prefixes in int8 and retains a fraction of visual positions; these are fixed serving settings.Source: §3 · p. 3

3.1 Fusion as Operating Point

  • The fusion weight maximizes correlation with relevance for standardized scores; the retriever receives positive weight when its relevance correlation exceeds its correlation with the reranker multiplied by the scores’ intercorrelation.The reverse condition ensures the reranker also receives positive weight; when both hold, either score alone has a lower correlation objective.Source: §3.1 · pp. 3–4

3.2 Self-Supervised Target

  • A shallow-state linear readout is trained against the same backbone’s full-depth answer-token margin on an uncompressed page, computed offline.The target margin is the difference between the yes- and no-token logits; the readout can approximate the relevant direction without reconstructing the entire final state.Source: §3.2 · p. 5
  • The linear approximation is empirical and need not hold for every shallow state; one fitted vector can compensate for depth truncation and cache compression when it holds on calibration data.The fit optimizes agreement with the teacher over centered calibration data, but does not guarantee improved relevance correlation for every candidate set; ridge strength is selected on held-out queries using centered error.Source: §3.2 · p. 5, §3.3 · p. 6

3.3 Centered Ridge in Closed Form

  • Centering removes each query’s mean from candidate states and targets, aligning the ridge fit with within-list ranking differences and eliminating an intercept.The closed-form solution is obtained by one linear solve, with no gradient step and the backbone unchanged.Source: §3.3 · p. 5

4 Evaluation

Across ViDoRe benchmarks, RidgeRank brings NDCG@5 close to the full cross-encoder reference at substantially lower measured latency. The evaluation also tests transfer across candidate pools, validates the theoretical objectives, and examines component and parameter sensitivity.

4 Evaluation

  • Evaluation spans ViDoRe 2 and ViDoRe 3, with 1,038 and 4,730 queries across four and eight corpora, and uses corpus-held-out readout fitting.Four readouts are fitted using 154 calibration queries from ViDoRe 2; fusion weights are estimated separately within each benchmark.Source: §4 · pp. 6–7

4.1 Ranking Quality

  • 1.2 and 1.5 points below the full cross encoder on ViDoRe 2's ColNomic and ColPali pools, RidgeRank also improves on the retriever in all four settings on NDCG@5.The same coefficients close 85% and 91% of the ColPali distance on ViDoRe 2 and 82% on ViDoRe 3 with either backbone.Source: §4.1 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.2 Accuracy and Latency Tradeoff

  • 48 times cheaper than the full cross encoder on ViDoRe 2, the GLM configuration costs 141 ms versus 6,765 ms; on ViDoRe 3, RidgeRank retains the same ordering.On ViDoRe 3, RidgeRank takes 213 ms and 233 ms versus 7,534 ms for the cross encoder, while its margin over GQR narrows to 1.6 points.Source: §4.2 · p. 7, Figure 2 · p. 9

4.3 Theoretical Validation

  • 0.894–0.994 correlation, with median 0.975, links Equation 9 agreement r with c_s across eight backbone–corpus pairings; predicted and measured weights correlate at 0.898.The predicted-weight figure reports a 95% interval for measured weights; the weaker retriever leaves more weight to the reranker.Source: §4.3 · p. 9, Figure 5 · p. 9, Figure 4 · p. 9

4.4 Component Ablation

  • 4.1 points over the logit lens come from the ridge solve; removing the model score costs 2.7 points, while candidate-list centering adds 0.6 points.The ablation removes one component at a time with remaining settings fixed.Source: §4.4 · p. 9, Table 3 · p. 10

4.5 Parameter Sensitivity

  • Both backbones peak before the final layer; quality rises through 20 candidates and stays flat to 100, varying by less than 1.6 points across two decades of λ.Fusion weights nearest the best change ranking quality by less than half a point.Source: Figure 6 · p. 9, Figure 7 · p. 9, §4.5 · p. 10
  • 16 calibration queries reach 99.4% of the 154-query quality on GLM and 98.3% on Qwen.The fit is obtained by drawing calibration queries at random and refitting.Source: §4.5 · p. 10

5 Conclusion

RidgeRank addresses the cost of multimodal reranking and the need for relevance judgments in some compression methods. It fuses retriever and reranker scores and uses a shallow readout trained on the model’s uncompressed scores; across 12 datasets, it approaches full-cross-encoder NDCG@5 at up to 48× speed.

5 Conclusion

  • On ViDoRe 2 and ViDoRe 3, RidgeRank’s NDCG@5 lies within 1.2 pp of a full cross encoder at up to 48 times the speed.Source: §5 · p. 10
  • RidgeRank fuses normalized retriever and reranker scores with a weight derived in closed form from their correlations with relevance and with each other.The rule quantifies relevance information absent from the reranker score and recovers it from the retriever score.Source: §5 · p. 10
  • A single centered ridge solve fits a shallow linear readout to the uncompressed model’s answer-token margin without gradient updates or relevance annotations.The fit uses differences among candidates within each list, while the backbone remains frozen.Source: §5 · p. 10

A Proofs

The proofs express correlations as inner products of standardized candidate-list scores, derive the optimal fusion direction and its endpoint conditions, and bound relevance agreement from agreement with a target score.

A Proofs

  • For standardized vectors, inner products equal correlations, allowing the fusion objective to be written with candidate-list inner products.The proof uses centered, unit-norm vectors for the retriever score, reranker score, and relevance.Source: A Proofs · p. 13

A.1 Proposition 1

  • The correlation-maximizing fusion direction is proportional to (cb − ρcs, cs − ρcb), then normalized to weights summing to one.The derivation uses the score correlation matrix and its positive definiteness when |ρ| < 1.Source: §A.1 · pp. 13–14
  • The optimum is interior exactly when cb > ρcs and cs > ρcb; otherwise at least one score receives zero weight.Positive weighted positions preserve the proof, and the deployed discount is fixed by the retriever’s ranking order.Source: §A.1 · p. 14

A.2 Proposition 2

  • The positive-semidefinite score correlation matrix constrains target-score correlations, and the lower bound on relevance agreement rises with agreement to the target.The proof obtains the bound from the allowable interval for cs; its lower-root function is strictly increasing for r, ct ∈ (0, 1).Source: §A.2 · pp. 14–15

B Additional Results

The fusion rule applies to the full cross encoder as well as to the readout. On every evaluated subset, the full cross encoder’s best fusion weight is below one, and relying on its score alone loses ranking quality.

B Additional Results

  • The additional results report ranking quality under four further metrics alongside the measured cost of each arm.The supplied passage does not state the values or winners for those metrics.Source: B Additional Results · p. 15

B.1 Fusion at Full Depth

  • Across all 24 subsets, the full cross encoder's fusion optimum is never w = 1, and the median optimum is w = 0.80; cross-encoder-only ranking loses 1.5 points on average, up to 4.3.Source: §B.1 · p. 15
  • Ranking by the full cross encoder alone loses 1.5 points on average and up to 4.3 compared with its fused optimum.Proposition 1 applies to any pair of scores, including the uncompressed cross encoder and retriever scores.Source: §B.1 · p. 15

B.2 Number of Calibration Queries

The calibration-query analysis measures quality as the number of sampled queries increases. Sixteen unjudged queries already approach the full-sample quality on both backbones, and both remain above GQR on the same queries at every size.

B.2 Number of Calibration Queries

  • Sixteen calibration queries reach 99.4% of full-sample quality on GLM and 98.3% on Qwen.The queries are drawn randomly, and none has a relevance judgment.Source: §B.2 · pp. 15–16
  • The analysis varies the number of calibration queries while scoring a held-out corpus; the reported queries carry no relevance judgments.Source: §B.2 · p. 15
  • At every calibration size, both backbones score above GQR’s 67.2 on the same queries.The reported comparison is against GQR on the held-out corpus of the 154-query calibration sample.Source: §B.2 · pp. 15–16

B.3 Label Free Regularization

The section compares relevance-judgement and teacher-based choices of ridge strength, then reports their deployed choices and calibration performance.

B.3 Label Free Regularization

  • Within each outer fold, two corpora fit the readout and the third validates ridge strength; Table 7 reports the modal choice across four folds and NDCG@5 at that value.Table 7 compares relevance judgements with centered teacher error for selecting λ.Source: §B.3 · p. 16, Table 7 · p. 17
  • Figure 8 plots NDCG@5 against the number of unlabelled calibration queries, reporting means and standard deviations over five draws.Source: Figure 8 · p. 17
  • On GLM both rules deploy 103 with bit-identical refitted vectors; on Qwen the teacher rule selects 103 in every fold and scores 72.3 against 71.3 for the labelled rule.Reported results use the teacher rule.Source: §B.3 · p. 17

B.4 Pool Depth for Every Method

Across candidate-pool depths, both RidgeRank backbones outperform the three external baselines, with a reported margin from ten candidates onward.

B.4 Pool Depth for Every Method

  • At every depth from five to a hundred candidates, both RidgeRank backbones rank above all three external baselines, by at least 5.0 points from ten candidates onward.Table 8 reports NDCG@5 on ViDoRe 2 with ColNomic candidates.Source: Table 8 · p. 17, §B.4 · p. 17
  • Table 8 sets candidate-pool depth as the comparison axis and includes the retriever-alone and GQR baselines at every depth.Source: Table 8 · p. 17
  • The retriever alone scores 60.9 NDCG@5, while GQR scores 63.8 at every depth.Source: Table 8 · p. 17

B.5 Operating Point Without Relevance Judgements

The section estimates RidgeRank’s fusion weight from score correlations without relevance judgements and compares that estimate with label-based alternatives.

B.5 Operating Point Without Relevance Judgements

  • Under a one-factor model with mutually uncorrelated noise, the retriever score, readout, and uncompressed model’s full-depth margin each carry relevance plus noise.Proposition 1’s needed score–relevance correlations can be recovered from scores alone.Source: §B.5 · p. 18
  • 59.9 average NDCG@5 for the label-free estimate on unlabelled queries, against 60.1 for the deployed rule; sixteen queries per corpus already give 59.8.The estimate uses correlations between scores and needs no relevance judgements.Source: §B.5 · p. 18, Table 9 · p. 18
  • Table 9 compares sweep, Proposition 1, and label-free fusion-weight choices; the label-free setting uses either all unlabelled queries or sixteen sampled queries.The third score is the full-depth margin of Qwen2.5-VL-7B.Source: Table 9 · p. 18

Related papers

Loading 2609.34192v1…