PaperlayerView exact revision on arXiv

Source-linked AI summary

KoViDoRe: Korean Visual Document Retrieval

Yongbin Choi, Yongwoo Song, Mujeen Sung

arXiv:2608.20840v1cs.IRcs.CV

TL;DR

Existing Korean visual document benchmarks provide limited coverage of complex documents and generally evaluate single-page retrieval, despite real information needs spanning multiple pages. KoViDoRe addresses this gap with a Korean benchmark built from publicly available documents and a multi-stage curation pipeline, and the authors report that current multimodal retrieval models struggle on the resulting task. They also release Ko-VDR Train Public as a training resource for Korean multimodal retrieval.

  • Problem

    Existing benchmarks provide limited Korean coverage and largely formulate visual document retrieval as single-page search, missing multi-page evidence aggregation.

  • Method

    KoViDoRe uses publicly available Korean documents and a multi-stage pipeline combining structured parsing, synthetic query generation, relevance mapping, and human verification.

  • Results

    Current multimodal retrieval models struggle with Korean visual document retrieval, especially for structured content, diverse query types, and evidence distributed across multiple pages.

  • Takeaways & Limitations

    KoViDoRe and Ko-VDR Train Public provide a benchmark and training resource for Korean visual document retrieval.

  • Takeaways & Limitations

    Query generation relies on parsed markdown representations and image captions rather than raw visual inputs, which may lose fine-grained layout, color, or chart-specific information.

Abstract

from arXiv · show

Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. However, existing benchmarks remain largely centered on English and provide limited coverage of Korean visual documents with complex structures. Furthermore, most existing Korean resources primarily evaluate single-page retrieval, failing to capture realistic scenarios that require evidence aggregation across multiple pages. To address these gaps, we introduce KoViDoRe, a benchmark for Korean visual document retrieval. The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures. We develop a multi-stage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and context-based strategies, and relevance mapping with human verification. Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types. Motivated by this finding, we further curate a large-scale training dataset, Ko-VDR Train Public, to support the development of retrieval models tailored to Korean visual documents. Together, KoViDoRe and Ko-VDR Train Public provide a unified benchmark and training resource for Korean visual document retrieval.

1 Introduction

KoViDoRe addresses gaps in Korean visual document retrieval by evaluating multi-page evidence aggregation in realistic documents. The authors also release a large-scale Korean training dataset after finding that existing models struggle with this setting.

  • Motivation: Existing VDR benchmarks largely treat pages independently, although real documents distribute relevant information across multiple pages.This limitation affects scenarios requiring evidence aggregation and reasoning over a set of pages.
  • Benchmark: KoViDoRe is a Korean benchmark designed to retrieve and synthesize distributed evidence from realistic visual documents.Queries can require multi-hop inference, numerical comparison, and cross-sectional aggregation with explicit supporting-page mappings.
  • Scope: The benchmark targets Korean and enterprise contexts that remain insufficiently represented in prior retrieval resources.The introduction identifies limited coverage of Korean visual documents and a gap in existing benchmarks.
  • Resources: Ko-VDR Train Public is released as a large-scale dataset for training Korean multimodal retrieval models.It is aligned with the proposed task and supports development and evaluation in Korean multimodal settings.
  • Findings: Existing multimodal retrieval models struggle to retrieve evidence distributed across multiple pages on Korean visual documents.The difficulty becomes especially pronounced as the number of required supporting pages increases.

2 Related Work

Prior multimodal retrieval advances support visually rich document search, but Korean evaluation remains underexplored and commonly uses single-page formulations. KoViDoRe complements these resources with multi-page, higher-complexity retrieval over realistic Korean documents.

  • Multimodal Retrieval: Multimodal retrieval models jointly represent textual, visual, and layout information for page-level search in structured documents.These approaches extend retrieval beyond text-only semantic matching.
  • Korean Evaluation: Multimodal retrieval datasets and evaluations remain concentrated on English and European languages, leaving Korean performance underexplored.The paper connects this gap to the interaction between Korean linguistic characteristics and document layout.
  • Existing Benchmarks: Recent benchmarks broaden evaluation with visually rich documents, multimodal signals, and realistic query formulations.Examples include ViDoRe, Jina-VDR, REAL-MM-RAG, UniDoc-Bench, MIRACL-VISION, and IRPAPERS.
  • Korean Resources: Korean resources such as Jina-VDR, MIRACL-VISION, and SDS KoPub VDR still differ from realistic structured Korean document settings or remain single-page.The paper notes limitations involving corpus composition and query formulation.
  • KoViDoRe: KoViDoRe emphasizes multi-page evidence aggregation and higher reasoning complexity for realistic Korean visual document retrieval.Its queries require relationships between distributed information that cannot be satisfied by a single page alone.

3 Dataset Curation

KoViDoRe is curated from authentic Korean documents through structured parsing, complementary query-generation strategies, and staged relevance filtering with human verification. The resulting benchmark covers diverse layouts, query types, formats, and multi-page evidence needs.

  • Pipeline: The curation pipeline comprises document collection, structured parsing, query generation, and relevance mapping adapted to Korean document ecosystems.The design is inspired by ViDoRe V3 while targeting Korean real-world data sources.
  • Source Collection: The corpus uses publicly available Korean government, policy, and enterprise-style documents with complex layouts, tables, figures, and multi-column structures.The collection prioritizes authentic sources rather than synthetic or simplified formats.
  • Structured Parsing: Pages are parsed into page-level and element-level markdown, with components such as text blocks, tables, figures, charts, and diagrams plus visual captions.This representation preserves document structure while exposing non-textual content for downstream processing.
  • Query Generation: Queries are generated from both cross-section summaries and local page or multi-page contexts to capture global relationships and localized information needs.The pipeline also controls query types and formats, including multi-hop reasoning, numerical comparison, and aggregation.
  • Relevance Mapping and Filtering: Relevance annotations combine generation-stage signals, page-level mapping, consistency and rule-based filtering, and final human verification.The process prioritizes reliable annotations over coverage and removes low-quality or answer-revealing queries.
  • Benchmark Statistics: The benchmark contains 57 documents, 6,729 pages, 706 queries, and 2,073 relevance annotations across four domain-specific subsets.The HR and Energy subsets contain more pages per query, suggesting greater multi-page aggregation requirements.

4 Experiments

The experiments evaluate multimodal retrieval models on KoViDoRe using standardized page-ranking metrics across Korean document domains. Results show persistent difficulty with structured, multi-page evidence, while Korean-specific training improves retrieval.

  • Evaluation Setup: KoViDoRe evaluates Korean visual document retrieval as a page-ranking task using graded page relevance and nDCG@10.Pages labeled 2 contain sufficient answer information, while pages labeled 1 provide supporting evidence.
  • Evaluation Setup: Models span CLIP-based, late-interaction, and multimodal embedding architectures across Cybersecurity, Energy, Economic, and Human Resources domains.The evaluation covers small, medium, and large models using separate domain-specific retrieval indices.
  • Main Results: Larger models generally perform better, but gains vary by domain and model scale alone does not determine effectiveness.Small models perform poorly, whereas medium late-interaction models improve; training objective and data composition also matter.
  • Main Results: jina-embeddings-v4 achieves the best overall performance and significantly outperforms other models across all domains.
  • Main Results: Retrieval remains limited in Economic and Human Resources, where complex structures and multi-page information make relevant evidence harder to retrieve.Strong models also struggle with structured visual content and queries requiring evidence aggregation across multiple pages.
  • Ko-VDR Train Public: Fine-tuning colSmol-500M and colqwen2-v1.0 on Ko-VDR Train Public consistently improves performance across domains.The gains show that Korean-specific training data can help smaller models compete with larger multimodal embedding models.

5 Ablation Study

The ablation study examines how query complexity and training-data composition affect retrieval. Multi-page queries are harder, while public and private Korean supervision provide complementary benefits overall.

  • Query Complexity: Performance generally decreases as the number of relevant pages per query increases.Both evaluated models perform best for single-page queries, with declines that are not strictly monotonic.
  • Query Complexity: Qwen3-VL-Embedding-8B consistently outperforms Qwen3-VL-Embedding-2B across groups while showing similar performance trends.The shared trend indicates that queries requiring evidence integration across pages remain challenging for both model sizes.
  • Training Data Composition: Training with Korean-specific private data improves performance over the base model, while public data produces larger gains across most subsets.The comparison varies only training-data composition while holding the remaining training configuration fixed.
  • Training Data Composition: Combining private and public data achieves the best overall performance across most subsets, especially Cybersecurity, Energy, and Human Resources.The Economic subset differs, with private data alone achieving the highest performance there.

6 Conclusion

KoViDoRe introduces a Korean visual-document benchmark centered on multi-page evidence integration, alongside a large-scale Korean training dataset. Experiments show current models struggle with this setting, while Korean-specific training improves retrieval.

  • Contributions: KoViDoRe evaluates queries requiring retrieval and integration of information distributed across multiple pages.Its documents come from publicly available Korean sources with diverse layouts and a multi-stage curation pipeline with human verification.
  • Findings: Current multimodal retrieval models struggle particularly with structured content and diverse query types in Korean visual documents.
  • Contributions: Ko-VDR Train Public is a large-scale training dataset designed for Korean visual document retrieval.The experiments demonstrate improved retrieval performance from Korean-specific training data.
  • Implications: Together, KoViDoRe and Ko-VDR Train Public are intended to facilitate future research on Korean visual document retrieval.

Limitations

The paper identifies limitations in visual fidelity, relevance mapping, reproducibility, and evaluation scope. These constraints affect query quality, training reproducibility, and the breadth of conclusions.

  • Data Curation: Query generation uses parsed markdown and image captions rather than raw visual inputs, potentially losing fine-grained layout, color, and chart information.The authors plan to incorporate vision-language models to capture visual information more fully.
  • Data Curation: Consistency-based relevance mapping may still introduce noise because generation and mapping stages can be imperfectly aligned.
  • Reproducibility: The private Korean VQA dataset cannot be publicly released because of licensing restrictions, limiting full reproducibility of the training setup.
  • Evaluation Scope: The experiments focus on evaluating existing retrieval models, constraining the scope of the reported findings.

A Appendix

The appendix supplies materials supporting transparency and reproducibility, including benchmark comparisons, model-selection rationale, query-generation definitions, prompts, examples, and metadata.

  • The appendix compares KoViDoRe with SDS KoPub VDR and explains the technical rationale for selected models.
  • It documents formal query-category definitions, complete prompts, representative query-page examples, and source-collection metadata.

A.1 Comparison to SDS KoPub VDR

KoViDoRe differs from SDS KoPub VDR mainly in query-to-document mapping: it targets multi-page evidence aggregation rather than single-page retrieval.

  • SDS KoPub VDR largely maps each query to one relevant page, whereas KoViDoRe associates queries with multiple pages.
  • 2.94 relevant pages per query is the reported average for KoViDoRe.

A.2 Document Parsing and Query Generation Models

The pipeline uses layout-aware parsing for structured Korean documents and reasoning-oriented Korean language generation for complex retrieval queries.

  • Document parsing: Upstage Document Parse extracts page- and element-level markdown while decomposing pages into text, tables, figures, charts, and diagrams.
  • Document parsing: The parser generates captions for visual elements, enabling semantic interpretation of non-textual content.
  • Query generation: Solar-Pro3 generates queries from markdown conditioned on textual and visual information extracted during preprocessing.
  • Query generation: Solar-Pro3 is selected for complex Korean queries requiring multistep reasoning and realistic retrieval scenarios.

A.3 Query Category Definitions

The query-construction pipeline combines section summaries and local context to create realistic Korean retrieval queries requiring implicit multi-page synthesis, specific entities, and natural formats.

  • Relevant-page distribution: Most queries involve two or three relevant pages, while larger evidence sets occur less frequently but provide challenging aggregation cases.
  • Query generation: Summary-based generation uses full-page section summaries and aggregated cross-section summaries to support queries connecting information across sections.
  • Realistic retrieval constraints: Queries must require implicit synthesis across pages without artificial page references, using section headers, captions, or unique keywords instead.
  • Realistic retrieval constraints: Queries should use specific entities and conceptual references rather than copied exact figures or document-aware framing.
  • Query formats: Question, instruction, and keyword formats impose distinct requirements for sentence endings, commands, noun phrases, and explicit semantic relationships.
Loading 2608.20840v1…