PaperlayerView exact revision on arXiv

Source-linked AI summary

Inverting Multi-Vector Visual Document Indices

Zhuchenyang Liu, Yao Zhang, Yu Xiao

arXiv:2610.09920v1cs.IRcs.CLcs.CV

TL;DR

The paper asks whether a visual document retriever’s stored patch-vector index can reproduce a page, despite often being treated as less sensitive than the page. It uses conditional document-image generation and infers the encoder, page shape, and shuffled-vector order from the index; raw-index inversions recover 47% of words and rank source pages first 98.4% of the time, while tests of protections and a second retriever show additional leakage.

  • Problem

    Multi-vector visual document indices are often treated as less sensitive than pages, but whether a page can be reproduced from its index had not been examined.

    Source: §1 · p. 1

  • Method

    A conditional flow-matching inverter generates pages from indices while the attack infers the encoder, page shape, and shuffled-vector order from the index alone.

    Source: §1 · p. 2

  • Results

    Raw-index inversions recover 47% of words and rank their source first 98.4%; pooling and shuffling cut word recall to about 8%, while a second retriever reaches 70.2% source-page top-1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Multi-vector visual document retrievers are vulnerable to inversion through their stored index, which should be protected like the documents it encodes.

    Source: §7 · p. 8

    The ordered inverter’s validation loss was still falling at 100k steps, and confidence leakage from invented content remains future work.

    Source: §8 · p. 9

Abstract

from arXiv · show

Prevailing multi-vector visual document retrievers store each page as about a thousand patch vectors, often in vector databases run by a third party. Since no one can read a page from its vectors, this index is easily treated as less sensitive than the page. However, because the index keeps one vector per patch in raster order, and each vector is computed by a vision-language model pre-trained to read documents, we hypothesize that whoever runs or breaches the store can reproduce a page from its index alone. We frame inversion as conditional document image generation and infer from the vectors what the attack needs: the encoder, the page shape and, for shuffled vectors, their order. On the ViDoRe v3 benchmark, pages inverted from raw indices recover 47% of the words and 45% of the sensitive tokens. Used as queries against the stored indices, they rank their source page first 98.4% of the time. We test two cheap protections, token pooling and shuffling, which both cut word recall to about 8%. A model that restores the order of a shuffled index raises the share of source pages ranked first from 3.8% to 93.5%, while inverting a pooled index remains open. To test generalisation, we apply the same attack unchanged to another multi-vector retriever: its inverted pages still rank their source page first 70.2% of the time, though its word recall stays below a nearest-neighbour baseline. Multi-vector visual document retrievers are therefore vulnerable to inversion through their stored index, which should be protected like the documents it encodes.

1 Introduction

The paper asks whether a multi-vector document index can reveal the page it represents, despite often being treated as less sensitive than the page. It proposes index-only inversion and reports substantial content recovery and source-page identification.

1 Introduction

  • A conditional flow-matching inverter generates the page while the attack infers the encoder, page shape, and shuffled-vector order from the index alone.The motivation is that vectors retain raster position and contextual information, and are produced by models pre-trained to read documents.Source: §1 · pp. 1–2
  • Raw-index inversions recover 47% of words and 45% of sensitive tokens, and rank the source page first 98.4% of the time.The evaluation uses 19,252 ViDoRe v3 pages disjoint from the roughly 683k public training pages.Source: §1 · p. 2
  • Pooling and shuffling each cut word recall to about 8%, but restoring shuffled-vector order raises source-page top-1 from 3.8% to 93.5%.Inverting a pooled index remains open.Source: §1 · p. 2
  • With the same attack recipe on ColQwen3.5-4.5B, inverted pages rank their source first 70.2% of the time.This tests generalisation to a second multi-vector retriever.Source: §1 · p. 2

2 Related Work

Prior work studies inversion of text embeddings and image features, as well as privacy attacks through retrieval-system interfaces. The paper identifies stored-index inversion for visual late interaction as unexamined.

2 Related Work

  • Text-embedding attacks recover input words or tokens, while image-feature inversion targets natural images or captions rather than legible, two-dimensional document pages.Document-page generation has also been studied with layout planning and OCR-based supervision, which this inverter does not use.Source: §2 · p. 2
  • Multi-vector retrieval work studies late interaction and compression through pooling or merging, but does not ask what the stored vectors reveal.These methods include ColPali for page images.Source: §2 · p. 2
  • Privacy attacks on retrieval-augmented systems use generator prompts or query-answer behavior; stored-index inversion for visual late interaction had not been studied to the authors’ knowledge.The cited protections address the operational interface, including keeping raw query text from the vector-database provider.Source: §2 · pp. 2–3

3 Threat Model and Problem

The threat model asks what an attacker can recover from a stored page index without side information. It distinguishes raw, pooled, and shuffled storage and evaluates both page content and page identity leakage.

3 Threat Model and Problem

  • The attacker observes a page’s stored index and must infer the encoder, page shape, and—if shuffled—the vector permutation to generate an image.The target page image is sampled conditionally on the observed index.Source: §3 · p. 3
  • Zero side information excludes page source, shape, metadata, and access to indexed pages; the attacker may use public encoders and public documents for training.Any page information beyond the stored index counts as side information.Source: §3 · p. 3
  • The three storage configurations are raw ordered vectors, vectors merged by hierarchical pooling, and raw vectors stored in a random order.Late-interaction scoring is order-independent, so shuffling leaves retrieval unchanged.Source: §3 · p. 3
  • Leakage is measured as OCR-readable words and sensitive tokens, plus whether retrieval ranks the source page first.Figure 2 lays out encoder identification, public-page training, and the configuration-specific sources of page shape and vector order.Source: §3 · p. 3, Figure 2 · p. 3

4 Method

The attack uses a conditional image generator and estimates the information needed to place its conditioning vectors and produce a page. Its components identify the encoder, infer page shape, and restore shuffled order when necessary.

4 Method

  • The attack first identifies the encoder, then trains the inverter and auxiliary inference models on public pages, and finally infers shape and order before inversion.The inverter uses an ordered variant when vector order is known and a set variant otherwise.Source: §4 · p. 3

4.1 Conditional flow-matching inverter

  • The conditional flow-matching inverter generates a page in a frozen VAE’s latent space, learning a conditional distribution rather than regressing a potentially blurred page.It minimizes the flow-matching loss and uses classifier-free guidance at inference.Source: §4.1 · p. 4

4.2 Identifying the encoder

  • Encoder identification filters public candidates by index dimensions, vector norms, and vector count, then selects the candidate most similar to public reference-page indices.The procedure does not use any indexed page.Source: §4.2 · p. 4

4.3 Recovering the page shape

  • For raw indices, the method estimates page width from vector similarity at sequence lags; for pooled indices, a set classifier selects among shapes compatible with vector count.Pooling removes the sequence-period cue, while the pooled vector count still constrains candidate shapes.Source: §4.3 · pp. 4–5

4.4 Restoring the order of a shuffled index

  • For shuffled indices, a permutation-equivariant position model predicts each vector’s normalized row and column, then a Hungarian assignment restores the grid order.The model also predicts aspect ratio to select a candidate page shape.Source: §4.4 · p. 5

5 Experimental Setup

The experiments attack two public multi-vector retrievers using public training pages and a held-out ViDoRe v3 evaluation set. Leakage is assessed through OCR-based content measures and source-page re-identification, across raw, pooled, and shuffled indices.

Attacked public encoders

  • The primary encoder, EA, is Tomoro-ColQwen3-8B; the unchanged attack is also tested on EB, ColQwen3.5-4.5B, with different training data and input resolution.EA training uses 682,818 pages; EB training uses 346,770 pages from three collections, excluding VDR-MEGA-2.Source: §5 · p. 5

Target corpus and fixed evaluation set

  • Evaluation uses 2,000 of ViDoRe v3’s 19,252 pages, sampled uniformly across eight professional domains; only 8 have near-identical training counterparts.The 2,000-page set covers 11 page shapes, and removing the 8 overlapping pages changes no reported number.Source: §5 · p. 5

Stored indices

  • Each encoder is tested on raw indices, while EA also uses hierarchical token pooling at factors 3 and 9 and shuffled indices.The shuffle is a single random permutation per page, shared across attacks; pooling uses the unmodified hierarchical token pooler.Source: §5 · p. 5

Sampling

  • Sampling uses inferred page shapes, 50 Euler steps, a shared noise seed, and guidance scale γ = 4 across both encoders and every setting.The guidance scale was selected once on 100 in-distribution validation pages.Source: §5 · p. 5

Evaluation measures

  • Content leakage is measured by OCR word recall, normalized edit distance, and sensitive-token recall; re-identification measures whether the source page ranks first.Text metrics cover the 1,927 evaluation pages with at least 20 reference words; the independent judge is the other encoder, avoiding reliance solely on the attacked encoder.Source: §5 · pp. 5–6

6 Results

The experiments show that raw multi-vector indices support substantial content recovery and near-certain source-page re-identification. Pooling sharply reduces word recovery, while shuffled vectors can be reordered; results on a second retriever show that leakage varies across encoders.

6.1 The attacker’s preliminaries

  • The attacker identifies the correct encoder in all 26,000 decisions; shape recovery reaches 99.2% for raw indices and 99.95% for shuffled ones.For pooled indices, shape classification reaches 99.1% at factor 3 and 95.7% at factor 9.Source: §6.1 · p. 6

6.2 The page is invertible from its raw index

  • 45.0% sensitive-token recall exceeds the kNN baseline’s 14.6%; the null model recovers 0.1% of words, while 75.0% of low-word-recall pages still re-identify.In the selected Figure 3 table band, 59 of 60 numeric cells are recovered exactly.Source: §6.2 · p. 6, Figure 3 · p. 7
  • 47.4% of reference words are recovered at NED 0.292, versus 0.858 for kNN and 0.958 for the null model; 98.4% of inverted pages rank their source first.Table 1 reports the comparison on 2,000 held-out pages; OCR metrics cover pages with at least 20 reference words.Source: §6.2 · p. 6, Table 1 · p. 7

6.3 Inverting indices under two protections

  • Pooling cuts word recall to 7.6% at factor 3 and 8.1% at factor 9; restoring shuffled order raises re-identification from 3.8% to 93.5%.The restored shuffled index reaches 23.1% word recall; words and identity recover more fully than reading order, as lines and blocks appear at the wrong height.Source: §6.3 · pp. 7–8, Figure 4 · p. 8

6.4 Generalisation to a second retriever

  • Under the unchanged attack, EB raw top-1 is 70.2% versus 15.4% for kNN, but word recall is 7.9% versus 22.3% for kNN.An EA control trained on EB’s data re-identifies 98.9% of pages; the authors cannot determine which differing encoder properties explain the gap.Source: Table 2 · p. 8, §6.4 · p. 8

7 Conclusion

The conclusion is that a document page’s multi-vector index can be inverted without side information. Raw indices yield substantial word recovery and source-page re-identification; pooling remains uninverted, while shuffled-index order can be restored.

7 Conclusion

  • The attack infers the public encoder and page shape from stored vectors, and infers vector order for shuffled indices before redrawing the page.The paper frames inversion as conditional document image generation using a flow-matching inverter.Source: §7 · p. 8
  • Raw-index inversions recover 47% of words and rank the source page first 98.4% of the time; restoring shuffled order recovers most re-identification under EA.The conclusion reports that pooled-index inversion remains open.Source: §7 · p. 8
  • The paper concludes that multi-vector visual document indices should be protected like the documents they encode.Source: §7 · p. 8

8 Limitations

The evaluation is limited by its encoder family, choices that may understate leakage, metric constraints, uncertainty about which recovered content is correct, and the limited protections tested.

8 Limitations

  • The attack is tested only on ColPali-style late-interaction retrievers with Qwen backbones; private-encoder stores remain outside the study’s scope.Encoder identification is evaluated only on 13 public encoders.Source: §8 · p. 9
  • Several untuned choices may make measured leakage conservative, including a still-falling validation loss, training data, sampler guidance, and position-model configuration.A better-matched training set alone raises word recall from 0.474 to 0.555.Source: §8 · p. 9
  • OCR-based metrics inherit source-transcript errors, sensitive-token recall uses a regex heuristic without human validation, and re-identification measures source faithfulness rather than an attack.The authors note that re-encoding a known document with the public encoder can link it to the store without inversion.Source: §8 · p. 9
  • The inverter samples from a learned distribution, so the authors cannot verify which parts of an inverted page are correct.They leave separating shared, high-confidence leakage from invented content across samples to future work.Source: §8 · p. 9
  • Pooling resistance holds only for the attacks tested, because the authors did not try to recover positions for pooled vectors or evaluate other listed protections.The untested protections include noise, quantisation, keyed projections, encryption or private retrieval, and store access control.Source: §8 · p. 9

Ethical Considerations

The section describes risks where indices are stored apart from pages, illustrates recovered content, and records the corpus, metric, and page-shape choices that bound the reported evaluation.

Ethical Considerations

  • Hosted indices can become an intruder’s only copy when page images are stored separately; the evaluation used public data and models, not deployed stores.The authors frame raw indices as needing protection like the documents they encode and say shuffling is not a protection.Source: Ethical Considerations · p. 9

A Whole-page gallery

  • The gallery examples recover 59 of 60 numeric cells exactly and pharma acronyms exactly, despite word-level errors elsewhere.The finance filing’s registration number is legible, though OCR misreads one malformed digit.Source: A Whole-page gallery · p. 11, Figure 5 · p. 12

B Corpus and grid table

  • Deduplication and shape filtering leave 715,515 pages across 26 shapes; evaluation pages with rare shapes are still processed rather than excluded.The resulting set includes 682,818 training, 14,357 validation, and 18,340 in-distribution test pages.Source: B Corpus and grid table · p. 12

C Metric definitions

  • OCR-based metrics inherit source-transcript errors; sensitive-token rules overapproximate breach-relevant text, while raw-index word precision (0.475) nearly matches recall (0.474).The authors say the overapproximation includes figures, dates, amounts, capitalised words, and acronyms, including ordinary sentence-initial words.Source: C Metric definitions · p. 20

D Recovering the page shape

  • The candidate aspect-ratio band is [0.12, 8.0], covering 99.98% of real pages; a true shape outside it cannot be recovered.The corpus spans aspect ratios from 0.051 to 21.0.Source: D Recovering the page shape · p. 20
  • On the fixed evaluation set, the plain periodic score recovers 0.992 of shapes under EA and 0.795 under EB; score choice is not uniform across encoders.The peak score is slightly ahead under EB on validation and the in-distribution test, while the plain score leads under EA.Source: D Recovering the page shape · p. 20
  • For pooled indices, the true shape is among the top two candidates in 99.97% of factor-3 cases and 98.5% of factor-9 cases.Balanced accuracy on ViDoRe v3 is 0.733 at factor 3 and 0.656 at factor 9, with errors concentrated on rare shapes.Source: D Recovering the page shape · p. 21
  • Under EA, using inferred rather than true shape changes raw-index word recall from 0.476 to 0.474 and top-1 from 0.986 to 0.984.The shape estimate matters more under EB, whose periodic estimator is weaker.Source: D Recovering the page shape · p. 21

E Encoder identification

The encoder-identification pipeline combines structural compatibility with embedding-space tests, while the inverter uses the identified encoder’s index vectors to generate pages. The architecture also infers page shape from the index, avoiding side information.

E Encoder identification

  • 99.9% of single-page decisions and every decision from ten pages identify the candidate correctly with the query probe.Structure keeps the true encoder in every decision but leaves 2.77 candidates on average.Source: E Encoder identification · p. 22
  • The structure test filters candidates by vector dimension, unit norm, and processor-compatible vector count, but shared properties leave multiple candidates in most cases.It never separates EA from its smaller sibling; only ColLFM2’s own store is settled by structure alone.Source: E Encoder identification · p. 22
  • The reference cloud identifies every target store in every decision; its single-page error rate is below 0.012% at 95% confidence.The query probe’s 31 single-page errors involve confusions among ColPali v1.2, ColPali v1.3, ColNomic, and ColQwen2.Source: E Encoder identification · p. 22
  • Only the inverter for the true encoder returns a faithful page; the other encoder’s output is unreadable and has fidelity at chance.The comparison re-encodes inverted pages with either EA or EB and scores them against the store.Source: E Encoder identification · p. 23

F Training and sampling details

  • The inverter replaces Qwen-Image’s text stream with index vectors and jointly attends across image and condition streams in 16 blocks.Its output head predicts the latent velocity field, which is decoded by the frozen VAE during sampling.Source: F Training and sampling details · p. 23, Figure 14 · p. 24
  • Rotary positions computed from input shape let one parameter set serve all page shapes, with the shape inferred from the index.Position embeddings are omitted for pooled indices and for shuffled indices without the position model.Source: F Training and sampling details · pp. 23–24

G Position models

The position model predicts where shuffled vectors belong, then assigns them to page slots so the ordered inverter can read them. Its recovered ordering substantially improves both source-page identification and content recovery.

Position model

  • The position model is a permutation-equivariant Transformer that predicts each vector’s normalized row and column, with a page-shape token and aspect-ratio head.It has 19.35M parameters; the per-vector probe is a 0.43M-parameter MLP applied independently to each vector.Source: G Position models · p. 24

Shape and assignment

  • 0.9995 of shapes under EA and 0.998 under EB are recovered, versus count-prior rates of 0.898 and 0.978, respectively.The model selects a factorization of the vector count using its aspect-ratio prediction, then runs again with that shape.Source: G Position models · p. 25

Position accuracy and impact

  • Under true-position Gaussian noise, the fraction of vectors placed within one slot on both axes falls from 0.989 at σ = 0.5 to 0.117 at σ = 4.The corresponding mean position errors are 0.26 and 2.85.Source: G Position models · p. 25
  • 0.231 word recall and 0.935 re-identification follow position-model reordering, versus 0.021 and 0.134 for random order and the per-vector probe, respectively.The position model’s mean position errors are 3.2 index rows and columns; its gains exceed the independent-noise curve at similar error.Source: G Position models · pp. 25–26

H Guidance ablation

Guidance improves ordered-index word recall, but its effect is smaller on protected indices. The controls and bootstrap comparisons distinguish index-derived content from prior-like output, while showing that re-identification can persist even when OCR recall is low.

H Guidance ablation

  • Guidance γ = 4 is the smallest value on the ordered inverters’ 4–8 plateau, but protected-index word recall gains at most 0.033 and NED does not improve.The ordered inverter gains 0.170 word recall; on protected indices, the reported NED values move from 0.867 to 0.848 for shuffled and 0.899 to 0.902 for ×9.Source: H Guidance ablation · pp. 26–27

I Controls in full

  • The null model drops the index condition; seed agreement compares OCR transcripts from two seeds, and the VAE reconstruction provides an upper bound.These controls use a 200-page subset spaced through the fixed evaluation set.Source: I Controls in full · p. 27

J Bootstrap intervals

  • The position model adds 0.147 word recall over the attack without it, while pooling removes 0.393 from the raw index; its word recall does not differ from kNN.With the position model, sensitive-token recall exceeds kNN by 0.066 and top-1 is 0.935 against 0.199.Source: J Bootstrap intervals · p. 27

K Re-identification and readability

  • Below word recall of 0.1, 75% of pages still rank first, and the overall Spearman correlation between word recall and rank is ρ = 0.13.The figure bins 1,927 raw-index pages by word recall and shows source-page ranks among 19,252 pages.Source: Figure 18 · p. 27
  • Layout and partially formed glyphs can rank a page first even when OCR cannot transcribe them; 99.8% of pages rank within the top 100.The lowest-ranked page is at rank 991.Source: K Re-identification and readability · p. 28

L Results by domain

Pooling reduces retrieval utility by a small average amount, with different losses across domains. Failures on pooled and shuffled indices differ, and pooled-index inversion remains an open problem rather than evidence of safety.

L Results by domain

  • Pooling reduces mean nDCG@10 by 0.8% at factor 3 and 2.2% at factor 9, with domain losses ranging from 5.2% for French financial filings to 0.1% for physics slides.The raw-index inversion exceeds the kNN baseline in every domain; its median word recall is 0.493 and 47.1% of pages exceed 0.5.Source: L Results by domain · p. 28

M Where the attack fails

  • Pooled and shuffled-without-position inverters produce plausible pages of the right kind whose content does not match the source.For the ×9 pooled index, two samples agree with each other at F1 0.094 and with the source at 0.076; the null model has word recall 0.001 and NED 0.958.Source: M Where the attack fails · pp. 28–29

N The open problem

  • Inverting a pooled index remains open: failure of the methods tried is not evidence that a pooled index is safe.The paper identifies soft positions, differentiable assignment, and greater model capacity as possible directions.Source: N The open problem · p. 29

O Reproducibility

The reproducibility account checks that stored indices match their source pages and fixes the evaluation set and comparison procedures. It also records release, licensing, and compute details.

O Reproducibility

  • Code, evaluation manifest, and protocol are slated for release, but trained inverters are withheld; sampling, batch order, permutations, and OCR settings are specified, while initialization was unseeded.Each model was trained once.Source: O Reproducibility · p. 29
  • EB averages 739 vectors at 0.757 Mpx per training page versus EA’s 1,198 vectors at 1.227 Mpx, with encoder-specific resampling conventions matched for inversion targets.EB allows at most 768 merged tokens per page versus EA’s 1,280.Source: O Reproducibility · p. 29
  • Re-encoding matched stored indices at cosine 0.999–1.000 versus 0.09–0.17 for other pages; grid checks covered 711,603 VisRAG pages, and 59.9M tokens had no padding rows.Source: O Reproducibility · p. 29
  • The kNN baseline mean-pools training indices, retrieves 20 nearest by cosine, reranks with exact late-interaction scores, and uses the retrieved page’s stored index for re-identification.Source: O Reproducibility · pp. 29–30
  • Only 8 of 2,000 evaluation pages (0.4%) exceeded the 0.9 nearest-training-page score-ratio threshold, and removing them changed no reported number.The ratio compares the nearest training page's late-interaction score with the evaluation page's self-score.Source: O Reproducibility · p. 30
  • All settings used the same 2,000-page manifest, recorded before inverter evaluation; its 200-page stride-10 control spans all eight domains.Source: O Reproducibility · p. 30

P Licences and compute

  • The paper lists licenses for datasets, encoders, the image VAE, and OCR, and states that public artefacts are used for research within their terms.Source: P Licences and compute · p. 30
  • Training seven inverters took 133.3 GPU-hours, position models took 1.5 GPU-hours, and sampling plus re-identification took about 1.7 hours per 2,000-page setting.Training ran on single NVIDIA H200 GPUs; evaluation used a partition of one H200.Source: P Licences and compute · p. 30
Loading 2610.09920v1…