PaperlayerView exact revision on arXiv

Source-linked AI summary

ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval

Peng Chunyi, Xu Zhipeng, Yan Yukun, Liu Zhenghao, Yu Shi, Mei Sen, Sun Yubo, Zhang Yongheng, Zhou Jie, Gu Yu, Yu Ge, Sun Maosong

arXiv:2608.15698v1cs.CVcs.IR

TL;DR

Visual document retrieval lacks fine-grained signals identifying supporting evidence within pages. ConceptFormer learns adaptive, query-conditioned latent concepts and improves average NDCG@10 by 16.7% over the strongest visual retriever and 22.1% over the strongest OCR-based text retriever.

  • Problem

    Existing visual retrievers mainly use coarse page-level supervision, while finer-grained signals may incompletely represent complex visual evidence.

  • Method

    ConceptFormer dynamically allocates query-conditioned latent concepts from relevant evidence regions to provide fine-grained supervision for query-document alignment.

  • Results

    ConceptFormer achieves the best average performance across six benchmarks, improving average NDCG@10 by 16.7% over the strongest visual retriever and 22.1% over the strongest OCR-based text retriever.

  • Takeaways & Limitations

    Adaptive latent concepts provide complementary relevance supervision and accommodate varying evidence scales in visually rich documents.

Abstract

from arXiv · show

Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.

1 Introduction

ConceptFormer addresses the coarse page-level supervision of visual document retrieval by learning continuous, query-conditioned latent concepts. It uses VLM-guided, dynamically sized concept sequences to provide fine-grained query-document alignment and improve visual document representations.

  • Motivation: Existing visual retrievers encode queries and document screenshots into dense representations but typically rely on coarse page-level contrastive supervision.This supervision provides relevance signals without specifying the supporting evidence.
  • Motivation: ConceptFormer builds on multimodal latent-reasoning approaches that use continuous latent representations instead of natural language for intermediate reasoning.Prior work encodes reasoning into latent visual states, shared latent spaces, or interleaved language and continuous visual representations.
  • Contribution: ConceptFormer represents query-relevant evidence as continuous, query-conditioned latent concepts that narrow the semantic gap between textual queries and visual documents.The concepts provide fine-grained query-document alignment signals.
  • Method: A strong vision-language model localizes relevant regions and estimates latent-concept capacity by mapping them onto the retriever’s visual token grid.ConceptFormer then dynamically allocates a latent concept sequence with the corresponding length.
  • Results: Experiments on diverse visual document retrieval benchmarks show that learned latent concepts improve retrieval beyond standard page-level matching.The resulting supervision enables more tailored visual document representations and provides complementary relevance information.

2 Related Work

Prior visual document retrieval methods commonly convert pages to text with OCR, while newer approaches model localized semantics, verbalize visual evidence, or learn continuous latent representations. These strategies address fine-grained query-document alignment but expose limitations in preserving visual structure or relying on textual evidence proxies.

  • Visual Document Retrieval: OCR-based pipelines retrieve pages through textual representations using sparse, dense, or LLM-based embeddings, but recognition errors can discard essential visual information.The omitted visual information includes spatial layout and other page-level structure.
  • Fine-Grained Visual Modeling: ColPali improves query-page matching through multi-vector interactions that capture localized page-level semantics.
  • Fine-Grained Visual Modeling: Other methods verbalize visual evidence into textual descriptions that serve as auxiliary supervision for retrieval.ReAlign uses VLM reasoning to identify query-relevant regions and describe cropped visual evidence for retriever training.
  • Continuous Latent Reasoning: Recent vision-language research constructs intermediate reasoning representations in continuous latent spaces to preserve query-relevant visual information.LVR learns latent visual states, while MCOUT and LanteRn develop unified or interleaved latent reasoning mechanisms.

3 Methodology

ConceptFormer extends standard contrastive visual document retrieval with adaptive, query-conditioned latent concepts that connect localized visual evidence to semantic relevance. A training-time vision-language model determines concept capacity, while retrieval-guided alignment teaches the concepts to preserve ranking behavior; inference retains the original retrieval pipeline.

  • Standard Retrieval Formulation: Visual document retrieval encodes queries and image-form pages into a shared semantic space, ranks candidates by similarity, and trains with contrastive learning against positive and negative pages.Pages contain heterogeneous information such as text, layout, tables, charts, and visual symbols; cosine similarity is a typical ranking function.
  • ConceptFormer Overview: ConceptFormer introduces continuous, query-conditioned latent concepts as an intermediate representation linking localized visual grounding with query-level semantic relevance.The latent concept space is grounded in page regions and captures semantic evidence required for document ranking.
  • Concept Learning via Matryoshka Representations: A training-time vision-language model identifies query-relevant evidence regions, whose structured annotations guide semantic-complexity estimation and adaptive latent concept capacity.Evidence annotations follow a reasoning-crop-observation process, coupling textual reasoning with localized regions for fine-grained verification.
  • Concept Learning via Matryoshka Representations: ConceptFormer maps evidence regions onto the retriever’s visual patch grid and aggregates covered-patch scales into a latent concept length, assigning larger capacity to charts, maps, tables, and cross-region relations.Samples with more localized evidence receive shorter latent concept segments, while visually or relationally complex evidence receives larger capacity.
  • Visual-Language Alignment via Latent Concepts: The retriever generates continuous concept states from adaptive latent tokens, aggregates them into a relevance representation, and aligns its candidate-ranking distribution with the query-induced distribution using KL divergence.The joint objective combines page-level query-document discrimination with concept-ranking alignment, and inference requires no latent concept generation.

4 Experimental Methodology

ConceptFormer is evaluated on diverse visual document retrieval data using Recall@10 and NDCG@10, with comparisons spanning OCR-based text and image-based visual retrievers. Training uses Qwen3.6-Plus to determine latent-concept capacity while testing retriever backbones initialized with Phi3V-4B and Qwen2.5-VL-7B-Instruct.

  • Datasets: The training set contains approximately 38,000 query-document pairs spanning industrial documents, infographics, web pages, tables, and open-domain document pages.Pairs are collected from DocVQA, InfoVQA, VisualMRC, OpenWikiTable, DUDE, and MHDocVQA.
  • Evaluation Metrics: Evaluation uses Recall@10 and NDCG@10, with statistical significance tested by permutation test at P< 0.05.
  • Baselines: ConceptFormer is compared against OCR-based text retrievers and image-based visual retrievers.OCR systems operate on PaddleOCR-extracted text, while visual systems retrieve directly from document page images without OCR extraction.
  • Implementation Details: Qwen3.6-Plus is used only during training to determine the adaptive capacity of latent concepts, while retriever backbones use Phi3V-4B and Qwen2.5-VL-7B-Instruct.This setup enables evaluation across different vision-language encoder architectures.
  • Implementation Details: All ConceptFormer variants train for 3 epochs with AdamW, effective batch size B= 128, and latent concept representation loss weight λ= 0.2.Experiments use four A100 GPUs with 40GB memory each, and models share training and inference configurations unless otherwise specified.

5 Evaluation Results

ConceptFormer achieves the strongest average retrieval performance across six visual document retrieval benchmarks, with gains over both visual and OCR-based text retrievers. Ablations and analyses show that latent concept supervision, adaptive capacity, and distributed concept states improve query–document alignment.

  • Overall Performance: 16.7% and 22.1% relative improvements in average NDCG@10 over the strongest visual retriever and OCR-based text retriever, respectively.ConceptFormer achieves the best average performance across six visual document retrieval benchmarks.
  • Overall Performance: ConceptFormer preserves visual layouts, charts, and non-textual cues while its latent concept space improves semantic alignment beyond global page representations.These advantages explain its gains over OCR-based and existing visual retrievers.
  • Backbone Analysis: Over 10% additional improvement with Qwen over Phi3V shows that ConceptFormer benefits from stronger backbones while remaining effective with weaker ones.The Phi3V variant already outperforms the strongest existing visual retriever by approximately 4% on average.
  • Latent Concept Representations: ConceptFormer outperforms textual and visual concept proxies, while textual verbalization provides more informative semantic supervision than direct visual-feature aggregation.The latent representation therefore improves learning of query-related evidence compared with both proxy alternatives.
  • Relevance Supervision: ConceptFormer benefits from relevance-distribution alignment and mean pooling removal, indicating that query-relevant information is distributed across multiple latent concept states.Its Query-Concept Matching objective outperforms a contrastive alternative, and the full model outperforms ConceptFormer without mean pooling.
  • Loss-Weight Sensitivity: λ = 0.2 yields the best average performance on Qwen2.5-VL, whereas larger weights do not consistently improve retrieval and may compromise query–page discrimination.At λ = 0, concept-level ranking supervision is removed, leaving conventional page-level contrastive learning.

6 Conclusion

ConceptFormer is a latent concept learning framework for visual document retrieval that adaptively encodes query-conditioned relevance, bridges visual grounding with semantic relevance, and improves retrieval across diverse benchmarks.

  • ConceptFormer introduces adaptive latent concepts to encode query-conditioned relevance in visual document retrieval.
  • The framework bridges localized visual grounding and semantic relevance through query-conditioned latent concepts.
  • ConceptFormer dynamically allocates concept capacity according to evidence coverage, capturing fine-grained information while retaining the standard retrieval pipeline at inference.
  • Experiments across diverse benchmarks show consistent improvements for ConceptFormer.

A Additional Experimental Details · A.1 Datasets and Evaluation Tasks · A.2 Baseline Checkpoints

The supplementary experiments train ConceptFormer on approximately 38K query-document pairs and evaluate it across six visual document retrieval benchmarks. Baselines use publicly released OCR-based, visual, and multimodal retriever checkpoints under official preprocessing and inference settings when available.

  • A.1 Datasets and Evaluation Tasks: Approximately 38K query-document pairs from six datasets form ConceptFormer’s training data.The datasets are DocVQA, InfoVQA, VisualMRC, OpenWikiTable, DUDE, and MHDocVQA.
  • A.1 Datasets and Evaluation Tasks: Each positive query-page pair additionally receives query-related region proposals for adaptive concept capacity allocation.
  • A.1 Datasets and Evaluation Tasks: The processed training set is stored as a merged JSONL file containing queries, positive pages, bounding-box annotations, and corresponding text.
  • A.1 Datasets and Evaluation Tasks: ConceptFormer is evaluated on six visual document retrieval benchmarks spanning varied document formats and evidence structures.
  • A.1 Datasets and Evaluation Tasks: The evaluation tasks include retrieving infographics, charts, presentation slides, textbook pages, and statistical charts relevant to natural-language questions.The named benchmarks are InfoVQA, ChartQA, SlideVQA, TQA, and OWID Charts.
  • A.2 Baseline Checkpoints: OCR-based retrieval extracts page text with PaddleOCR before applying the corresponding sparse or dense text retriever.
  • A.2 Baseline Checkpoints: Visual and multimodal retrievers directly encode original page images, using publicly released checkpoints and official preprocessing and inference settings whenever available.Table 8 lists checkpoints used in the main and supplementary experiments.

A.3 Implementation Details · A.4 Evidence Proposal and Concept Capacity Allocation

ConceptFormer uses a Qwen2.5-VL-7B-Instruct-based retriever with LoRA fine-tuning and normalized EOS representations. Training-time evidence proposals dynamically allocate latent concept capacity according to query-related visual-region coverage.

  • A.3 Implementation Details: The retriever is initialized from Qwen2.5-VL-7B-Instruct, while the Phi3V setting uses Tevatron/dse-phi3-docmatix-v1 and VDocRetriever-Phi3-vision initialization.
  • A.3 Implementation Details: Three training epochs with AdamW use 4 GPUs, gradient accumulation over 4 steps, and per-device batch size 8, yielding an effective batch size of 128.
  • A.3 Implementation Details: LoRA fine-tuning uses rank 8, scaling factor 64, dropout 0.1, BF16 precision, and learning rate 1 × 10−4.LoRA is applied to q_proj, k_proj, v_proj, o_proj, down_proj, up_proj, and gate_proj.
  • A.3 Implementation Details: Retrieval reads EOS representations, applies ℓ2 normalization, and computes probabilities with temperature 0.01; latent concept states are mean-pooled.The representation objective uses forward KL with λ=0.2 and no additional textual-distribution KL objective.
  • A.4 Evidence Proposal and Concept Capacity Allocation: Concept capacity is dynamically tied to retriever-specific coverage of query-related regions rather than a fixed token budget.Localized evidence receives shorter sequences, whereas larger charts, maps, tables, or cross-region relations receive greater capacity; “fixed latent tokens = 0” denotes this allocation.
  • A.4 Evidence Proposal and Concept Capacity Allocation: The training-time evidence proposer uses Qwen3.6-Plus to return query-related regions from each query and positive page image, but is absent during retrieval inference.It decodes at temperature 0.2 and processes requests with 16 concurrent workers.
  • A.4 Evidence Proposal and Concept Capacity Allocation: Proposed bounding boxes provide spatial grounding for concept-capacity allocation, while retained textual labels supply semantic descriptions and textual concept proxies for ablation.

A.5 Sensitivity to the Representation-Loss Weight

The latent concept representation-loss weight controls concept-side ranking supervision and must balance distribution matching against the discriminative query-page objective. λ=0.2 is used as the common default because it is optimal for Qwen2.5-VL and near-optimal for Phi3V.

  • Effect of λ: λ=0 removes concept-side ranking supervision, reducing training to standard page-level contrastive learning.Positive λ encourages latent concepts to reproduce query-induced ranking behavior over candidate pages.
  • Backbone sensitivity: λ=0.2 gives the best average result on the main Qwen2.5-VL backbone.Performance is non-monotonic as λ increases, suggesting overly strong distribution matching can interfere with the discriminative query-page objective.
  • Backbone sensitivity: For Phi3V, λ values from 0.1 to 0.5 remain close, with the numerically best result at λ=0.5.λ=0.2 remains near the optimum for Phi3V and is therefore selected as the common default.

B Retrieval Geometry Metrics

ConceptFormer’s latent concept representation is compared with visual and textual concept proxies using three retrieval geometry metrics. These metrics assess query discrimination, page discrimination, and alignment with the original query’s document-ranking distribution.

  • Representation Comparison: The analysis compares visual concept proxy ˆv_i, textual concept proxy ˆr_i, and latent concept representation ˆc_i for each sample.The same comparison framework is applied across all three representation types.
  • Query Discriminability: Query Discriminability measures how clearly a representation distinguishes the matched query from mismatched queries.A larger D_q(a_i) indicates better separation from unrelated queries.
  • Page Discriminability: Page Discriminability measures separation between the positive page and irrelevant pages.A larger D_p(a_i) indicates stronger separation from negative pages.
  • Retrieval Alignment: Retrieval Alignment measures whether a representation induces a document-ranking distribution close to that of the original query.A higher R(a_i) indicates closer ranking distributions.
Loading 2608.15698v1…