PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexVDR: Learning to Allocate Matryoshka Representation Budgets for Visual Document Retrieval

Haowen Xiang, Yibo Yan, Xin Zou, Shuliang Liu, Yi Cao, Mingdong Ou, Xuming Hu

arXiv:2610.30177v1cs.IR

TL;DR

Visual document retrieval needs rich representations to capture fine-grained page evidence, while making compact representations available does not decide which one a query should use. FlexVDR predicts route utility from early query features and selects among depth, width, and token budgets using a resource preference; across five backbones and three ViDoRe benchmark versions, it retains much of native retrieval quality while selecting compact routes.

  • Problem

    Visual document retrieval uses multi-vector representations to preserve fine-grained page evidence, but query encoding and vector-level matching are computationally demanding, and fixed configurations do not adapt to each query.

    Source: §1 · p. 1

  • Method

    FlexVDR predicts route-specific retrieval utility from early query features and combines it with a configurable resource preference to select among routes varying depth, width, and token budgets.

    Source: §1 · p. 2

  • Results

    Across five backbones on ViDoRe V1, V2, and V3 public, FlexVDR retains much of native retrieval quality while selecting compact routes for individual queries.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The same retriever can adjust computation for different resource preferences without retraining or switching models.

    Source: §1 · p. 2, §5 · p. 10

    Retrieval errors and biases inherited from the evaluated models and corpora may affect which documents are surfaced to users.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Visual document retrieval relies on rich multi-vector representations to capture fine-grained evidence in document pages, but encoding and matching these representations can be computationally expensive. Existing compression and elastic representation methods expose smaller configurations, yet making multiple budgets available does not determine which configuration a query should use. The allocation decision must estimate the retrieval value of alternative representations before incurring their full computation, while accommodating different resource preferences. We introduce FlexVDR, a framework for query-adaptive budget allocation within a single retriever. We construct a joint family of routes over encoder depth, embedding width, and token count. Retrieval supervision and distillation train the compact routes, while a native bypass preserves the full-budget reference. A Budget Allocator predicts route-specific retrieval utility from early query features and combines these predictions with a configurable resource preference to select a route. Experiments across five backbones on ViDoRe V1, V2, and V3 public show that FlexVDR retains much of the native retrieval quality while selecting compact routes for individual queries. A configurable resource preference allows the same retriever to adjust its computation without retraining or switching models.

1 INTRODUCTION

Visual document retrieval uses multi-vector representations to preserve fine-grained page evidence, but query encoding and vector matching are costly. FlexVDR addresses the gap between making compact representations available and deciding which one to use for each query.

1 INTRODUCTION

  • A single fixed depth, width, and token configuration imposes a common capacity–computation compromise even though relevant evidence varies by query.Queries may target text, tables, or figures, and late interaction matches query terms to page regions.Source: §1 · p. 1
  • The allocation decision must anticipate retrieval value before remaining encoder computation and remain adjustable across resource preferences without learning a new rule.Running every candidate route before selection would incur the computation that allocation is intended to avoid.Source: §1 · p. 2
  • FlexVDR predicts route-specific retrieval utility from early query features and combines it with a resource preference to select a route.The route family jointly varies encoder depth, embedding width, and query/document token count, and includes the native full route.Source: §1 · p. 2
  • The paper evaluates five backbones on ViDoRe V1, V2, and V3 public and examines computation across resource preferences.Source: §1 · p. 2

2 RELATED WORK

Prior work supplies visual multi-vector retrieval models and methods for varying representation capacity or computation. FlexVDR focuses on selecting a joint depth–width–token budget for each incoming query.

Multimodal multi-vector retrieval

  • Visual multi-vector retrievers use token-level matching for fine-grained page evidence, with ViDoRe benchmarks covering visually rich domains, harder query construction, and multimodal retrieval-augmented generation.ColBERT introduced late interaction for text retrieval, and ColPali transferred it to page images.Source: §2 · p. 3

Matryoshka representation learning

  • Matryoshka learning and compression methods vary representation width, depth, storage, or document-vector count, but FlexVDR combines depth, width, and token choices and selects a budget per query.Prior examples include nested dimensions, depth–dimension elasticity, residual compression, patch pruning, spatial pooling, and query-agnostic compression.Source: §2 · p. 3

Budget-oriented search

  • Prior budget-oriented methods adapt encoder depth, ANN search, document-region routing, token selection, or MaxSim scoring; FlexVDR selects a joint encoder-and-representation route before remaining query encoding and matching.Its route utilities are predicted from shared early query features separately from the resource preference.Source: §2 · p. 3

3 FLEXVDR: LEARNING AND ALLOCATING REPRESENTATION BUDGETS

FlexVDR learns compact routes around a frozen native retriever, trains an allocator to predict their query-specific utilities, and selects a route using a resource preference. The selected route then governs continued query encoding and matching against document representations from that route.

3.2 LEARNING SELECTABLE MATRYOSHKA REPRESENTATIONS

  • A route specifies encoder exit depth, embedding width, and a shared query/document output-token retention ratio, with the native full route included.The route family crosses candidate values on all three axes.Source: §3.2 · p. 4
  • Compact routes are trained with retrieval supervision and distillation from frozen native full-route scores, then the elastic retriever is frozen before allocator labels are generated.Nested-width supervision encourages leading coordinates to retain retrieval information when later coordinates are omitted.Source: §3.2 · p. 5

3.3 TRAINING THE BUDGET ALLOCATOR

  • The allocator predicts route utilities from features at the shallowest candidate exit, without using relevance labels, preliminary retrieval results, or full-depth selection features at query time.A fixed sparse random projection and multi-output ExtraTrees regression form the predictor.Source: §3.3 · p. 5
  • Route utilities are supervised by query-level nDCG@5 over a deterministic local document pool, preserving quality gaps among routes rather than only a best-route label.Squared-error regression fits the full utility vector independently of resource preference.Source: §3.3 · pp. 5–6

3.4 QUERY-ADAPTIVE BUDGET ALLOCATION AND RETRIEVAL

  • At query time, the allocator selects the route maximizing predicted utility minus a preference-weighted structural penalty; increasing the preference favors lower-penalty routes without refitting the predictor.The penalty combines encoding depth and matching volume, whose shared token-retention ratio gives relative matching volume τ^2d/D when token rounding is ignored.Source: §3.4 · p. 6
  • For fixed routes, penalties, and predictor, increasing the preference cannot increase selected structural penalty; bounded utility-prediction error also bounds regret in the utility–penalty objective.Pre-deployment calibration selects a preference that minimizes mean structural penalty subject to a permitted quality decrease; query encoding then resumes at the selected exit and matches same-route document representations.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate FlexVDR across five backbones and three ViDoRe benchmarks, examining retrieval quality, budget allocation, quality–computation trade-offs, and resource-preference sensitivity. The results show near-native quality with frequent use of compact routes, while allocation and quality responses vary by backbone and preference.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five backbones on ViDoRe V1, V2, and V3 public, using each task’s complete document corpus and nDCG@5 as the primary metric.Each backbone supports 64 routes through one elastic checkpoint; allocation comparisons share the trained representation family, and online FLOPs include query encoding, output heads, matching, and allocator arithmetic for learned policies.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 RETRIEVAL QUALITY AND REPRESENTATION BUDGETS

  • Across five backbones and three benchmarks, mean nDCG@5 decreases remain below two points, and V1/V2 decreases below one point, while FlexVDR assigns many queries to lower budgets.For ColPali, lower-budget shares are 66.60% on V1 and 83.62% on V2, with nDCG@5 scores of 83.16 versus 83.90 and 53.39 versus 53.82, respectively.Source: §4.2 · p. 7, Table 1 · p. 7
  • webAI and Nemotron remain the top two backbones on all three benchmarks while routing most queries to lower budgets.webAI assigns 99.97% of V1 queries and all V2 and V3 public queries to lower budgets; Nemotron assigns 88.32%, 98.18%, and 96.77%, respectively.Source: §4.2 · p. 8

4.3.1 VARIANT STUDY

  • Learned allocation improves quality–computation trade-offs over fixed and random choices: on ColQwen2.5, it gains 6.91 nDCG@5 points over the 75% Fixed 3D Budget while reducing FLOPs by 18.2%.At similar computation to random selection, it achieves 32.40 nDCG@5 versus 18.82; all four evaluated backbones have learned operating points with higher quality and lower FLOPs than random selection.Source: §4.3.1 · p. 8, Figure 3 · p. 8
  • On webAI, FlexVDR reduces FLOPs by 20.4% relative to Native Full with an nDCG@5 decrease of 0.86 percentage points.Width-fixed and token-fixed variants also retain near-native quality, reducing FLOPs by 19.0% and 20.4% with nDCG@5 decreases of 0.51 and 0.86 points, respectively.Source: Figure 3 · p. 8, §4.3.1 · p. 9
  • Fixing depth limits FLOPs reduction more than fixing width or token count on this workload, because every query still executes all encoder layers.Width-fixed and token-fixed policies retain the option to exit early; their relative quality versus the three-axis policy varies across operating points, with no uniform ranking established.Source: Figure 3 · p. 8, §4.3.1 · p. 9

4.3.2 HYPERPARAMETER ANALYSIS

  • At λ = 0.1, webAI uses 46.1% of Native Full FLOPs on the 1K workload, whereas ColQwen2.5 uses 90.5%.Increasing λ generally reduces computation by favoring compact routes, but computation at the same preference differs across backbones.Source: §4.3.2 · p. 9, Figure 4 · p. 9
  • Mean V1/V2 nDCG@5 retention ranges from 97.0% to 98.9% at λ = 0.05, but falls to 81.0%–90.8% at λ = 0.2.For the main results, one preference per backbone is selected using a calibration nDCG@5 decrease threshold of 0.5 percentage points, then held fixed across benchmarks and controller seeds.Source: §4.3.2 · p. 9, Figure 4 · p. 9

5 CONCLUSION

FlexVDR allocates representation budgets per query using early query features and a joint family of depth, width, and token routes. Across the evaluated backbones and benchmarks, it retains much of native retrieval quality with compact routes, while resource preferences adjust operating points without retraining.

5 CONCLUSION

  • FlexVDR uses early query features to select among joint depth–width–token representations, including the native full route.Its Budget Allocator predicts route utility and separates those predictions from the resource preference.Source: §5 · p. 10
  • Across five backbones on ViDoRe V1, V2, and V3 public, learned policies retain much of native retrieval quality while selecting compact routes.Quality and allocation responses differ across backbones.Source: §5 · p. 10
  • The same retriever supports different operating points without retraining by changing the resource preference.The conclusion describes allocation as a query-level decision that adapts to the desired preference.Source: §5 · p. 10

AI USE STATEMENT

The authors report AI assistance for code debugging, language polishing, and feedback on research content. They state that they reviewed the assisted material and take responsibility for the final content.

AI USE STATEMENT

  • GPT-5.3-Codex assisted with code debugging, while GPT-5.5 assisted with grammar checking and language polishing.The tools also assisted with feedback on methodology, result interpretation, and mathematical statements and proofs.Source: AI USE STATEMENT · p. 10
  • The authors state that they reviewed and verified AI-assisted content and take responsibility for the final content, including text, code, claims, and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work studies visual document retrieval with released models and existing benchmarks. The authors note that inherited model and corpus biases may affect which documents users see, and call for attention to permissions, privacy, licenses, and attribution.

ETHICS STATEMENT

  • The study’s scope is released models and existing document benchmarks.Source: ETHICS STATEMENT · p. 10
  • Inherited model and corpus biases may affect which documents are surfaced to users.Source: ETHICS STATEMENT · p. 10
  • Applications should respect document access permissions, privacy requirements, and model and dataset licenses, while preserving attribution to retrieved sources.Source: ETHICS STATEMENT · p. 10

A IMPLEMENTATION AND ROUTE CONSTRUCTION

The implementation builds a 64-route family crossing depth, width, and token retention, with Native Full as the maximal route. Compact projection heads and the Budget Allocator are trained separately while the native retrieval path remains frozen.

A IMPLEMENTATION AND ROUTE CONSTRUCTION

  • The route grid crosses depth–width pairs with four token-retention ratios, and Native Full uses maximal depth and width with full token retention.Each axis uses nominal retention ratios {1/4, 1/2, 3/4, 1}.Source: Table 2 · p. 12, A IMPLEMENTATION AND ROUTE CONSTRUCTION · p. 12
  • ColPali’s selected layers are 4, 9, 14, and 18, whose actual retained proportions are 22.22%, 50%, 77.78%, and 100%.Cost and depth-retention statistics use actual layers, not nominal quartile labels; valid tokens follow the released model’s output order.Source: A IMPLEMENTATION AND ROUTE CONSTRUCTION · p. 12
  • The retriever stays frozen while a projection bank is trained with AdamW for three epochs on 118,195 ColPali query–page rows.Training uses batch size four on each of eight GPUs and cross-device document negatives.Source: A IMPLEMENTATION AND ROUTE CONSTRUCTION · p. 12
  • The utility predictor is fitted only on the controller-training partition, while the elastic retriever uses all 118,195 rows.Document identities are split 80/10/10; local single-positive utilities differ from external multi-positive evaluation.Source: A IMPLEMENTATION AND ROUTE CONSTRUCTION · p. 12
  • Compact width/token streams share prefixes within a layer stream, but different layers and the transformed final-layer stream require separate storage or on-demand computation.Metadata, indexing structures, alignment, quantization, and temporary buffers add to the dense vector payload.Source: A IMPLEMENTATION AND ROUTE CONSTRUCTION · p. 13

B EXTERNAL DATASETS AND SCORING

The evaluation covers ViDoRe V1 and V2 task suites and a separate public V3 evaluation, using benchmark-specific scoring and fixed native references. Supplementary sections describe recall, Industrial variants, and V1/V2 route controls.

B EXTERNAL DATASETS AND SCORING

  • V1 has ten task configurations with 3,871 queries, and V2 has four configurations with 1,152 queries; Table 3 lists task query and corpus sizes.Each task includes all judged queries and unique corpus pages; ESG v2 and ESG Human v2 share a corpus but use different queries.Source: B EXTERNAL DATASETS AND SCORING · p. 13, Table 3 · p. 13
  • V1/V2 configurations share queries, images, relevance judgments, and scoring precision, while quality replay compares route rankings against each task’s complete corpus.Query-time selection uses early features and a fixed penalty table; offline enumeration is evaluation-only.Source: B EXTERNAL DATASETS AND SCORING · pp. 13–14

B.1 VIDORE V3 PUBLIC EVALUATION

  • V3 public evaluation uses 2,419 English queries from eight public datasets and complete document corpora, with metrics averaged equally across datasets.It reports nDCG@5/10, recall@1/5/10, and reciprocal rank; Table 6 provides supplementary nDCG@10 and MRR.Source: §B.1 · p. 14, Table 5 · p. 14, Table 6 · p. 15

B.2 SUPPLEMENTARY RECALL RESULTS

  • Table 7 reports macro-averaged recall@1 and recall@5 across ViDoRe benchmarks for FLEXVDR preferences and fixed native baselines.The supplied passage describes the table’s metrics and scope but does not provide its cell values.Source: Table 7 · p. 15

C.1 VARIANT STUDY ON INDUSTRIAL

  • The Industrial study evaluates four backbones on 283 queries and all 5,244 pages, while the supplementary V1/V2 controls compare routing and fixed budgets.Table 8 uses the trained representation family; its fixed routes assign one configuration to every query, unlike Industrial variants that retain two adaptive axes.Source: §C.1 · p. 15, §C.2 · p. 15, Table 8 · p. 16

D CONFIRMATION SCORES AND FULL-QUERY EVALUATION

At the selected preferences, the full V1/V2 evaluation reports small nDCG@5 changes from Native Full, while confirmation intervals include zero. The routing distributions and random-assignment controls describe how selected routes compare with alternatives.

D CONFIRMATION SCORES AND FULL-QUERY EVALUATION

  • All five 95% query-bootstrap intervals for confirmation-set Router minus Native Full include zero.The confirmation set comprises 224 queries disjoint from calibration queries but included in the full evaluation.Source: D CONFIRMATION SCORES AND FULL-QUERY… · p. 16, Table 10 · p. 16
  • The nDCG@5 changes from Native Full are −0.655±0.173, −0.182±0.108, −0.121±0.176, +0.159±0.216, and −0.207±0.132 percentage points for ColPali, ColQwen2, ColQwen2.5, webAI, and Nemotron, respectively.Table 11 aggregates all 5,023 V1/V2 queries, including calibration and confirmation subsets.Source: D CONFIRMATION SCORES AND FULL-QUERY… · p. 16, Table 11 · p. 17

E ROUTING DISTRIBUTIONS AT THE SELECTED PREFERENCES

F RANDOM-ASSIGNMENT CONTROLS

  • The matched-random control permutes the Router’s route choices within each task, preserving the exact route multiset for ColPali, ColQwen2, and ColQwen2.5.Table 15 reports the 14-task macro nDCG@5 and Router-minus-matched-random difference.Source: F RANDOM-ASSIGNMENT CONTROLS · p. 18, Table 15 · p. 18

G EXTERNAL CALIBRATION AND PREFERENCE SENSITIVITY

External calibration selects resource preferences under a quality tolerance, and sensitivity analysis varies that preference. The FLOPs protocol counts specified online arithmetic but excludes work that also affects end-to-end latency.

G EXTERNAL CALIBRATION AND PREFERENCE SENSITIVITY

  • The main preferences use an absolute nDCG@5 tolerance of 0.5 percentage points and remain fixed across benchmarks and controller seeds.Calibration uses 448 queries, reserves 224 disjoint confirmation queries, and chooses the lowest mean structural penalty within the quality tolerance.Source: G EXTERNAL CALIBRATION AND PREFERENCE… · p. 19
  • Figure 5 plots nDCG@5 relative to Native Full and full-route query share across resource preferences, using equal averaging over 14 V1/V2 tasks.The V1/V2 sweep evaluates ten preferences from cached seed-42 predictions.Source: Figure 5 · p. 19, G EXTERNAL CALIBRATION AND PREFERENCE… · p. 19

H ROUTE-CONDITIONED FLOPS PROTOCOL

  • Reported FLOPs exclude norms, activations, most elementwise operations, MaxSim reductions, TopK, tokenizer work, and data movement, so they do not represent end-to-end latency.Offline page encoding and index construction are also outside the online-query count.Source: H ROUTE-CONDITIONED FLOPS PROTOCOL · p. 21

I THEORETICAL ANALYSIS

The analysis characterizes how the selection rule trades predicted utility against structural penalty: stronger resource preference cannot increase the selected penalty, and bounded utility-prediction error bounds decision regret. Oracle query-adaptive selection can outperform fixed routes when different routes are uniquely best for different query subsets.

I THEORETICAL ANALYSIS

  • Increasing λ cannot increase the selected route’s structural penalty.This monotonicity concerns c(r), not evaluated FLOPs.Source: I THEORETICAL ANALYSIS · p. 21
  • If utility prediction error is at most δu(q) across routes, decision regret is bounded by 2δu(q).The comparison uses the same reference utility and fixed resource preference for the optimal and predicted selections.Source: I THEORETICAL ANALYSIS · p. 21
  • Oracle query-adaptive allocation is at least as good in expected utility as the best fixed route, and is strictly better when different routes uniquely maximize utility on positive-probability query subsets.The strict comparison follows from a positive expected gap for any fixed route under that condition.Source: I THEORETICAL ANALYSIS · p. 21
  • For λ > 0, each selected route is non-dominated in predicted utility and structural penalty.A route with no lower predicted utility and no higher penalty, with at least one strict improvement, would have a larger selection objective.Source: I THEORETICAL ANALYSIS · p. 21
Loading 2610.30177v1…