PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Adaptation with Query-Dependent Residuals for Visual Document Retrieval

Zeliang Li, Xiaofen Xing, Kailing Guo, Xiangmin Xu

arXiv:2609.27688v1cs.IR

TL;DR

Visual document retrieval is hard to adapt when encoders and page indexes are fixed, and conventional reranking uses feedback only on selected candidates. Q-REACT reuses reranker preferences through query-dependent adaptation and document-aware full-index scoring. Across eight ViDoRe V3 tasks and five backbones, it improves average retrieval over evaluated baselines at sparse and full-coverage budgets and transfers to held-out queries and tasks.

  • Problem

    When encoder parameters or corpus re-encoding are unavailable, it is unclear how finite reranker feedback can improve retrieval beyond the selected queries and candidate pages.

    Source: §1 · p. 1

  • Method

    Q-REACT learns query-dependent low-rank residuals, combines adapted-query relevance with document context, and distills reranker preferences over the complete task-specific page index.

    Source: §1 · p. 2

  • Results

    Across eight ViDoRe V3 tasks and five open-weight and proprietary backbones, Q-REACT improves average retrieval over evaluated baselines at sparse and full-coverage budgets and transfers to held-out queries and tasks.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Finite reranker feedback can be reused across a query collection without retraining or rebuilding the retriever, with only a small increase in inference cost.

    Source: §6 · p. 10

    Document-context scoring assumes that a strong query match within a source document can provide relevance evidence for its other pages.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Visual document retrieval (VDR) systems depend on page embeddings computed before deployment, which makes adaptation difficult when encoder parameters or corpus re-encoding are unavailable. Rerankers provide useful relevance signals, but conventional reranking applies them only to selected queries and candidate pages. We introduce Q-REACT, a query-side test-time adaptation method that converts limited reranker feedback into reusable retrieval improvements. Q-REACT learns a shared low-rank transformation that produces query-dependent residuals, combines adapted query scores with document-level context, and distills reranker preferences with a student distribution normalized over the complete task-specific page index. This design lets unscored pages compete through cached embeddings while keeping the encoders and page index fixed. Across eight ViDoRe V3 tasks and five open-weight and proprietary backbones, Q-REACT improves average retrieval over evaluated baselines at sparse and full-coverage budgets, transfers to held-out queries and tasks, and adds little inference overhead. The results show that finite reranker feedback can be amortized across a query collection without retraining or rebuilding the retriever.

1 INTRODUCTION

Visual document retrieval is difficult to adapt when encoder parameters or corpus re-encoding are unavailable, while conventional reranking uses feedback only to reorder selected candidates. Q-REACT turns that feedback into reusable, query-dependent retrieval improvements for frozen VDR systems.

1 INTRODUCTION

  • Conventional reranking improves the ordering of selected candidates but leaves relevance feedback unused beyond those rankings, limiting benefits to reranked queries.The introduction notes that sparse query coverage yields negligible average retrieval improvements.Source: §1 · p. 1
  • A fixed residual shared across queries may not accommodate VDR pages that serve diverse queries seeking different textual and visual evidence.The cited evaluation reports that TTT-Embed can degrade retrieval under sparse feedback and offers limited gains at full coverage.Source: §1 · p. 1
  • Across eight ViDoRe V3 tasks and five backbones, Q-REACT improves average retrieval over evaluated baselines at sparse and full-coverage budgets and transfers to held-out queries and tasks.Its low-rank query transformation, document-context scoring, and full-index distillation reuse reranker feedback while keeping the encoders and page index fixed.Source: §1 · p. 2
  • Q-REACT learns query-dependent residuals with a shared low-rank transformation and distills reranker preferences over the complete task-specific page index.Document-structure scoring provides a retrieval prior, while full-index normalization brings unscored pages into competition through cached embeddings.Source: §1 · p. 2

2 RELATED WORK

Related work includes visual document retrievers and multimodal rerankers, as well as methods that adapt queries at test time. Q-REACT uses reranker feedback to adapt a document-aware retrieval function while retaining encoders and cached page embeddings.

Visual document retrieval and multimodal reranking

  • Figure 1 presents reward-budget scaling averaged across five frozen VDR backbones.The introduction cites it for negligible average improvements from sparse conventional reranking and for TTT-Embed’s sparse-feedback degradation and limited full-coverage gains.Source: Figure 1 · p. 2
  • Visual document retrieval preserves page text, tables, images, and layout in offline indexes; multimodal rerankers refine candidate rankings without modifying the initial retriever.The cited systems include single-vector and multi-vector retrievers, with the latter enabling finer-grained matching at higher storage and computation costs.Source: §2 · p. 2
  • Figure 2 depicts Q-REACT’s local reranker preferences supervising a low-rank query adapter through full-index distillation and a residual penalty.The adapted query supplies direct page relevance and document-level evidence under the same scoring rule during adaptation and inference.Source: Figure 2 · p. 3

Test-time and query-side optimization

  • Many test-time query-optimization methods operate independently per query, while TTT-Embed distills reranker rewards into residuals shared globally, per task, or per query.Q-REACT instead learns a low-rank transformation that produces query-dependent corrections.Source: §2 · pp. 2–3

3 METHOD

Q-REACT adapts frozen visual document retrieval through a shared low-rank map that produces query-dependent residuals, document-context scoring, and full-index distillation of reranker preferences. After adaptation, it applies the learned transformation to new queries and retrieves from the complete task-specific index.

3 METHOD

  • Q-REACT learns query-dependent low-rank residuals through full-index distillation and document-context scoring while keeping encoders and page embeddings frozen.Global shares one transformation across tasks, whereas Task learns one per task; both use the same residual form, scoring rule, and objective.Source: §3 · p. 3

3.1 PROBLEM FORMULATION

  • The problem formulation uses frozen query and page encoders, cached page vectors, and a complete task-specific page index for retrieval.Feedback consists of reranker rewards on Base’s ordered top-K candidates for selected queries; the budget counts scored query–page pairs.Source: §3.1 · p. 3

3.2 QUERY-DEPENDENT LOW-RANK RESIDUALS

  • A shared low-rank linear map produces a residual from each query embedding, varying correction directions and magnitudes within a subspace of dimension at most r.Each matrix pair has 2Dr trainable parameters, and zero initialization of B yields the Context-only starting point when document-context scoring is active.Source: §3.2 · p. 4

3.3 DOCUMENT-CONTEXT-AWARE RETRIEVAL SCORING

  • Document-context scoring adds each page’s direct relevance to the strongest query match among pages in the same source document.The method assumes a strong query match within a document can provide relevance evidence for its other pages; context can also raise scores of irrelevant pages in a matching document.Source: §3.3 · p. 4

3.4 FULL-INDEX DISTILLATION FROM LOCAL REWARDS

  • Reranker preferences on fixed top-K candidates supervise an adapted-query student distribution normalized over the complete task-specific index.Cross-entropy fits local preferences, full-index normalization brings unscored pages into competition, and residual regularization constrains query displacement.Source: §3.4 · pp. 4–5

3.5 INFERENCE

  • At inference, the fixed transformation scores the complete task-specific index, allowing retrieval beyond rewarded candidates without per-query optimization or additional reranker calls.The encoders and cached page embeddings remain unchanged.Source: §3.5 · p. 5

4 EXPERIMENTS

Experiments evaluate Q-REACT across ViDoRe V3 tasks, frozen backbones, and feedback budgets, then test generalization without feedback on evaluation queries. Q-REACT exceeds direct reranking and TTT-Embed at full coverage, improves retrieval with sparse feedback, and transfers to held-out queries and tasks.

4.1 SETUP

  • Evaluation covers 14,514 queries, 19,256 pages, and eight ViDoRe V3 tasks, using five frozen single-vector backbones and task-averaged nDCG@10 (×100).Aggregate results average across five backbones.Source: §4.1 · p. 5

4.3 COMPARISON WITH PER-QUERY OPTIMIZATION

  • At b = 10, Q-REACT scores 61.14 and 62.61 in Global and Task modes, exceeding direct reranking by 0.90 and 2.36 points and TTT-Embed by 4.34 and 3.99 points.These are average scores with ten reranker judgments per query.Source: Table 1 · p. 6, §4.3 · p. 7
  • At b = 0.1, Q-REACT scores 56.60 and 56.61 in Global and Task modes, improving over Base by 1.87 and 1.88 points and exceeding GQR by 2.25 and 2.26 points.The feedback-based methods use ten reranker judgments for each of 145 queries.Source: Table 1 · p. 6, §4.3 · p. 7

4.2 RETRIEVAL PERFORMANCE ACROSS FEEDBACK BUDGETS

  • With feedback from 145 of 14,514 queries, Q-REACT approaches 2 nDCG@10 points of gain at b ∈[0.01, 0.1], while direct reranking yields negligible average gains.At b = 1, Q-REACT gains roughly 3 points, compared with about half a point for direct reranking.Source: Figure 3 · p. 6, §4.2 · p. 7
  • As feedback expands, Task overtakes Global; at b = 10, Q-REACT gains 6.41 and 7.88 points, versus TTT-Embed's 2.08 and 3.89.At the sparsest budgets below b = 1, Global initially leads; the adapter rank is held fixed.Source: Table 1 · p. 6, §4.2 · p. 7

4.4 GENERALIZATION TO HELD-OUT QUERIES AND TASKS

  • Without evaluation-query feedback, Q-REACT exceeds TTT-Embed in held-out-query and leave-one-task-out comparisons; Global improves all five backbones over Base in both settings.Held-out queries use the remaining 40% of each task; leave-one-task-out averages eight folds.Source: §4.4 · pp. 7–8, Table 2 · p. 7

5 ANALYSIS

The ablations show that regularization is most consequential with sparse feedback and that document context complements adaptation across feedback levels. Additional analyses compare reward models and report retrieval quality and latency for Q-REACT.

5.1 ABLATION STUDIES

  • At b = 0.01, removing regularization lowers Global and Task scores by 29.11 and 53.17 points, respectively.At b = 0.1, the drops are 5.69 and 29.38 points; at b = 10, changes are only +0.32 and −0.87 points.Source: §5.1 · p. 8, Figure 4 · p. 8
  • At b = 0.1, full Q-REACT gains 1.87 and 1.88 points, versus 0.69 in either mode without context and 0.73 for Context-only.At b = 10, retaining context raises adaptation gains from 4.26 and 6.45 to 6.41 and 7.88 points, respectively.Source: §5.1 · p. 8, Figure 4 · p. 8
  • Across eight matched comparisons, low-rank residuals beat fixed vectors by 1.14–3.89 points; full-index normalization benefits low-rank Task adaptation but not Global.Document context improves every matched configuration by 1.34–2.24 points.Source: §5.1 · pp. 8–9, Table 3 · p. 8

5.2 ADDITIONAL ANALYSES

  • Across four reward models, Global and Task improve over direct reranking by 0.90–4.32 and 2.36–3.51 points, respectively.Qwen3-VL-Reranker-2B yields the highest Global score (61.60), while Qwen3-VL-Reranker-8B yields the highest Task score (62.61).Source: §5.2 · p. 9, Table 4 · p. 9
  • Q-REACT inference takes 12.3 seconds, 2.5% above Base, with per-query latency of 0.85 ms versus 0.83 ms for Base.Training time rises from 2.0 to 38.5 seconds as b increases from 0.1 to 10, without changing inference time; Q-REACT trains once per adaptation run.Source: §5.2 · p. 9

6 CONCLUSION

Q-REACT adapts query-side retrieval while keeping the encoder and page index fixed. Across ViDoRe V3 tasks and diverse backbones, it improves average retrieval over evaluated baselines and transfers to held-out queries and tasks, with only a small inference-cost increase.

6 CONCLUSION

  • Q-REACT is a reusable query-side adaptation method for visual document retrieval with a frozen encoder and page index.Its components are a document-context prior, a query-dependent low-rank map, and full-index distillation.Source: §6 · p. 9
  • Across eight ViDoRe V3 tasks and five open-weight and proprietary backbones, Q-REACT improves average retrieval at sparse and full-coverage budgets and transfers to held-out queries and tasks.It outperforms the evaluated fixed-residual and direct-reranking baselines.Source: §6 · p. 10
  • Because its adaptation state is learned once and reused, Q-REACT achieves its gains with only a small increase in inference cost.Table 5 reports retrieval quality and latency using Qwen3-VL-Embedding-8B in Global mode.Source: Table 5 · p. 9, §6 · p. 10

A IMPLEMENTATION DETAILS

The implementation keeps encoders and page embeddings fixed, adapting only query-side parameters and retrieving from cached vectors. Feedback, budgets, optimization, and adapter settings define the evaluation procedure.

A IMPLEMENTATION DETAILS

  • Normalized query and page embeddings are cached in FAISS, where inner products give cosine similarity; encoders, page vectors, and document membership remain fixed.Document context adds each page’s maximum query–page score within its source document to its direct relevance score.Source: A IMPLEMENTATION DETAILS · p. 13
  • Held-out queries are separated by semantic group, preventing equivalent queries from appearing in both training and evaluation sets.TTT-Embed normalizes its student over rewarded candidates, while Q-REACT uses the complete task-specific page index; their teacher probabilities are matched.Source: A IMPLEMENTATION DETAILS · p. 13
  • Both methods are optimized with Adam for 150 epochs at learning rate 0.01, using FP32 adaptation and a fixed random seed of 42.Experiments use an NVIDIA A800 GPU; optimization uses no learning-rate scheduler.Source: A IMPLEMENTATION DETAILS · p. 13
  • Feedback sets are nested and fixed during optimization; at b = 0.01, 16 of 14,514 queries receive feedback, while b = 10 covers all queries.Each selected query receives ten page-level rewards for 0 < b ≤ 10.Source: A IMPLEMENTATION DETAILS · p. 13
  • The default rank-8 adapter uses Δ(qi) = BAxi/√r, with 65,536 trainable parameters at embedding dimension 4,096.Global shares one matrix pair across tasks and Task uses a separate pair per task; after adaptation, the matrices are frozen.Source: A IMPLEMENTATION DETAILS · p. 13

B BASELINE DETAILS

The evaluation compares five multimodal embedding models, four reranker feedback models, and two test-time adaptation baselines. Their setups differ in feedback scoring, residual parameterization, and whether optimization is performed per query.

B BASELINE DETAILS

  • Five embedding backbones are evaluated, including four locally run models and proprietary Kinfra-VL-Embedding-8B accessed through a remote API.Each independently encodes a text query and complete page image into one vector; embeddings are precomputed and cached.Source: B BASELINE DETAILS · p. 14
  • Four feedback models score query–page pairs; dedicated Qwen rerankers use yes/no logit probabilities, while Gemma and Qwen-Instruct use the raw Yes probability.Direct reranking sorts candidate pages by score, and adaptation converts scores into a teacher distribution.Source: B BASELINE DETAILS · p. 14
  • TTT-Embed learns one additive residual per Global or Task scope by matching teacher and student distributions over reranker candidates.Its inference scales the residual by α = n/(n + 1), where n is the number of feedback queries used to learn it.Source: B BASELINE DETAILS · p. 14
  • GQR optimizes each query for 50 steps on the union of two retrievers’ top-10 lists, using a frozen multimodal encoder and a complementary text retriever.The evaluated zero-shot setting uses no labels, reranker, or feedback budget.Source: B BASELINE DETAILS · p. 14

C QUERY-LEVEL ADAPTATION COMPARED WITH GQR

Q-REACT Query achieves the strongest reported query-level retrieval result in this comparison. Task mode reaches nearly the same quality without per-query optimization or additional reranker calls after adaptation.

C QUERY-LEVEL ADAPTATION COMPARED WITH GQR

  • The comparison uses frozen Qwen3-VL-Embedding-8B; Query mode optimizes adaptation per query, while GQR uses a complementary text retriever with T = 50 steps.Q-REACT Task is included as a task-level adaptation reference.Source: C QUERY-LEVEL ADAPTATION COMPARED WITH… · p. 14
  • 0.6480 nDCG@10; Q-REACT Query beats Base by 5.81 points, GQR by 9.91, direct reranking by 1.62, and TTT-Embed Query by 1.16.Table 6 reports nDCG@10 on the [0, 1] scale and differences from Base multiplied by 100.Source: Table 6 · p. 15, C QUERY-LEVEL ADAPTATION COMPARED WITH… · p. 15
  • Q-REACT Task reaches 0.6462, only 0.18 points below Query mode, without further reranker calls or per-query optimization after adaptation.The authors describe this as a favorable balance between retrieval quality and online computation.Source: C QUERY-LEVEL ADAPTATION COMPARED WITH… · p. 15

D NORMALIZATION SCOPE AND RESIDUAL REGULARIZATION

Full-index normalization adds competition from unscored pages, but its retrieval effect varies with residual parameterization and sharing scope. The regularization analysis distinguishes shared from query-dependent displacements without establishing guarantees for unseen queries.

D NORMALIZATION SCOPE AND RESIDUAL REGULARIZATION

  • Matched student distributions, parameterizations, weights, and penalties give CE and forward KL the same minimizers and gradients; changing normalization scope breaks that equivalence.Full-index normalization adds a probability-mass term that changes gradients by increasing rewarded candidates’ total mass relative to the rest of the index.Source: D NORMALIZATION SCOPE AND RESIDUAL… · pp. 15–16
  • Unscored pages with larger student probabilities exert competitive pressure through cached embeddings, but zero teacher mass outside feedback is an assumption, not a verified relevance label.The derivation does not guarantee that any particular page score decreases after an update.Source: D NORMALIZATION SCOPE AND RESIDUAL… · p. 16
  • At b = 10, low-rank Task gains from full-index over Top-10 normalization are 1.94 points without context and 1.14 with context; fixed-vector variants lose 0.69 and 0.62.Low-rank Global instead favors Top-10 by 0.75 and 0.38 points, so full-index normalization is not optimal for every sharing mode.Source: D NORMALIZATION SCOPE AND RESIDUAL… · p. 16
  • The capacity interpretation is joint: Task uses eight rank-8 transformations and Global one, while the normalization factorial comparison tests only b = 10.The interaction between normalization and feedback coverage remains to be tested directly.Source: D NORMALIZATION SCOPE AND RESIDUAL… · pp. 16–17
  • The vector penalty constrains one shared displacement, whereas the low-rank penalty weights query-dependent displacements by rewarded-query directions; the derivation adds no new regularization principle.The analysis provides no guarantee of stability or retrieval improvement on unseen queries.Source: D NORMALIZATION SCOPE AND RESIDUAL… · p. 17

E RANK SCALING ACROSS FEEDBACK BUDGETS

The benefit of adapter rank depends on feedback coverage and parameter-sharing scope. Rank scaling is limited under sparse feedback, while broader feedback supports larger gains, especially as Global capacity increases.

E RANK SCALING ACROSS FEEDBACK BUDGETS

  • The analysis varies r ∈ {1, 2, 4, 8, 16, 32} at b ∈ {0.01, 1, 10}; each adapter contains 2Dr trainable parameters.Figure 5 shows five-backbone average and individual-backbone gains over Base; direct reranking is the rank-independent reference.Source: E RANK SCALING ACROSS FEEDBACK BUDGETS · p. 17, Figure 5 · p. 18
  • At b = 0.01, both modes maintain average gains of roughly 1–2 points across tested ranks; at b = 1, Global is non-monotonic while Task maintains larger gains with moderate variation.The individual backbones show similar capacity limits, including declining Global performance at larger ranks for WeMM models.Source: E RANK SCALING ACROSS FEEDBACK BUDGETS · p. 17
  • At b = 10, average Global gain rises from roughly 3 points at r = 1 to nearly 9 at r = 32; Task increases from about 5 to 8 points.The positive trend appears across all five backbones, with some small local fluctuations in Task curves.Source: E RANK SCALING ACROSS FEEDBACK BUDGETS · p. 17, Figure 5 · p. 18
  • At b = 10, Task leads at small ranks and largely plateaus between r = 8 and r = 16; Global continues improving, exceeds Task at r = 32 on every backbone.The authors interpret the pattern as consistent with Global needing more capacity across heterogeneous tasks, while Task transformations specialize more narrowly.Source: E RANK SCALING ACROSS FEEDBACK BUDGETS · p. 18

Related papers

Loading 2609.27688v1…