PaperlayerView exact revision on arXiv

Source-linked AI summary

KoViDoRe: Korean Visual Document Retrieval

Yongbin Choi, Yongwoo Song, Mujeen Sung

arXiv:2608.20840v1cs.IRcs.CV

TL;DR

Korean visual retrieval benchmarks provide limited coverage of Korean documents and often treat pages independently, although real documents can distribute evidence across pages. KoViDoRe benchmarks multi-page Korean document retrieval, and evaluations find that current multimodal models struggle, especially as the number of supporting pages increases.

  • Problem

    Multimodal retrieval benchmarks offer limited coverage of Korean visual documents, while single-page formulations do not capture evidence distributed across multiple pages in real-world documents.

    Source: §1 · p. 2, §2.1 · p. 3

  • Method

    KoViDoRe reformulates Korean visual document retrieval as multi-page evidence retrieval and builds a benchmark from diverse Korean documents using a multi-stage curation pipeline with human verification.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Existing multimodal retrieval approaches struggle under KoViDoRe’s multi-page formulation, especially as the number of required supporting pages increases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    KoViDoRe and Ko-VDR Train Public provide resources for evaluating and developing retrieval models for Korean visual documents.

    Source: §1 · p. 2

    Query generation uses parsed markdown and image captions rather than raw visual inputs, which may lose fine-grained visual information and affect query quality.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. However, existing benchmarks remain largely centered on English and provide limited coverage of Korean visual documents with complex structures. Furthermore, most existing Korean resources primarily evaluate single-page retrieval, failing to capture realistic scenarios that require evidence aggregation across multiple pages. To address these gaps, we introduce KoViDoRe, a benchmark for Korean visual document retrieval. The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures. We develop a multi-stage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and context-based strategies, and relevance mapping with human verification. Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types. Motivated by this finding, we further curate a large-scale training dataset, Ko-VDR Train Public, to support the development of retrieval models tailored to Korean visual documents. Together, KoViDoRe and Ko-VDR Train Public provide a unified benchmark and training resource for Korean visual document retrieval.

1 Introduction

KoViDoRe addresses the gap in Korean visual document retrieval benchmarks by evaluating multi-page evidence aggregation in realistic documents. The paper also releases aligned training data and reports that existing retrieval models struggle as evidence spans more pages.

1 Introduction

  • Existing visual document retrieval benchmarks largely treat pages independently, although real-world information often spans multiple pages and requires evidence aggregation.Prior reasoning queries are typically limited to a single page or narrow context.Source: §1 · p. 2
  • KoViDoRe benchmarks Korean visual document retrieval by requiring models to retrieve the full set of pages whose combined content answers a query.Its queries can require multi-hop inference, numerical comparison, and cross-sectional aggregation.Source: §1 · p. 2
  • KoViDoRe adapts synthetic query generation for Korean enterprise-style documents, producing diverse queries with explicit mappings to their supporting pages.The queries include multi-hop inference, numerical comparison, and cross-sectional aggregation.Source: §1 · p. 2
  • Existing retrieval models struggle with Korean visual documents when evidence is distributed across multiple pages, especially as the number of required pages increases.The results point to a limitation of single-page retrieval paradigms.Source: §1 · p. 2
  • The authors release Ko-VDR Train Public, a large-scale dataset aligned with the benchmark to support training retrieval models for Korean multimodal settings.Source: §1 · p. 2

2 Related Work

Prior work advances multimodal retrieval for visually rich documents, but Korean coverage and realistic multi-page retrieval remain limited. KoViDoRe addresses this gap with Korean queries requiring evidence aggregation across pages.

2.1 Multimodal Retrieval Models

  • Multimodal retrieval models, including ColPali and late-interaction architectures, encode page text, visual features, and layout as a unified retrieval unit (Faysse et al., 2024; Xiao et al., 2025).Dual-encoder and dense approaches also use vision-language models to capture textual and visual semantics (Ma et al., 2024).Source: §2.1 · pp. 2–3
  • Multimodal retrieval datasets remain concentrated on English and European languages, leaving Korean model behavior underexplored despite interactions among morphology, spacing, domain expressions, visual structure, and layout.Existing visual document retrieval benchmarks, including ViDoRe (Macé et al., 2025; Loison et al., 2026), Jina-VDR (Günther et al., 2025), and others, have expanded multimodal evaluation but leave Korean relatively underrepresented.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Vision Document Retrieval Benchmarks

  • Jina-VDR includes Korean and broadens visual-document and query diversity, but its broad scenario coverage may be less representative of real-world Korean retrieval; MIRACL-VISION uses a primarily Wikipedia-derived corpus unlike structured, visually complex Korean documents.Source: §2.2 · p. 3
  • SDS KoPub VDR (Lee et al., 2025) benchmarks Korean documents with complex layouts, but maps each query to only one relevant page.Its single-page formulation does not address information distributed across multiple pages.Source: §2.2 · p. 3
  • KoViDoRe complements existing benchmarks with Korean queries that require evidence aggregation across multiple pages and relationships among distributed information.It targets higher reasoning complexity and more realistic document distributions.Source: §2.2 · p. 3

3 Dataset Curation

KoViDoRe’s curation pipeline adapts document collection, structured parsing, query generation, and relevance mapping to Korean document sources and real-world characteristics. It combines automated filtering with human verification to produce a benchmark spanning diverse queries and multi-page evidence.

3.1 Source Collection

  • Documents come from public Korean government, policy, and enterprise sources, prioritizing freely usable KOGL Type 1 and 2 or otherwise unrestricted materials and authentic complex layouts.The sources include the Korean public data portal1 and official institutional websites; the authentic documents preserve features such as tables, figures, and multi-column structures.Source: §3.1 · pp. 3–4

3.2 Document Processing and Parsing

  • Page-level processing with Upstage Document Parse produces page- and element-level markdown plus captions for visual elements, preserving structure while enabling fine-grained access to text, tables, figures, charts, and diagrams.Pages are treated as the fundamental unit, and the parser organizes content into semantically meaningful sections.Source: §3.2 · p. 4

3.3 Query Generation

  • Solar-Pro generates queries from document text and visuals using summaries for global, cross-section needs and local context windows for localized needs, with controlled formats and types including multi-hop, comparison, and aggregation.Cross-section summaries combine randomly sampled single-section summaries, while context prompts use page-level or local multi-page windows.Source: §3.3 · pp. 4–5

3.4 Relevance Mapping and Filtering

  • Relevance mapping compares generation-stage signals with page-level supporting evidence, retains consistent pairs, filters low-quality queries, and uses human review to refine unnatural or ambiguous annotations.This prioritizes annotation reliability over coverage, including in challenging multimodal and multi-page cases where relevant pages may be difficult to identify.Source: §3.4 · pp. 5–6

3.5 Dataset Statistics

  • The benchmark contains 57 documents, 6,729 pages, 706 queries, and 2,073 relevance annotations; HR and Energy have more pages per query, and query types and formats vary across subsets.Multi-hop, open-ended, and comparison-based queries appear frequently, while question- and instruction-style formats are more common than keyword queries.Source: Figure 3 · p. 5, Figure 4 · p. 5, §3.5 · p. 6

4 Experiments

Across KoViDoRe experiments, jina-embeddings-v4 achieves the best overall performance, while Korean-specific fine-tuning improves retrieval and enables a smaller model to surpass a larger baseline. Strong models still struggle with complex visual structures and evidence distributed across pages.

4.1 Experimental Setup

  • Evaluation ranks pages for Korean queries with graded page-level relevance and nDCG@10 across four domains, using MTEB adapted for multimodal retrieval.Source: §4.1 · p. 6

4.2 Main Results

  • jina-embeddings-v4 achieves the best overall performance across domains; larger models generally score higher, but gains vary by domain.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 Analysis

  • Economic and Human Resources scores remain relatively low, while complex layouts, structured visual content, and evidence spread across pages challenge strong models, especially on queries with multiple relevant pages.Source: §4.3 · p. 7

4.4 Ko-VDR Train Public

  • Ko-VDR Train Public contains 310,226 Korean visual-document query-page pairs, filtered by relevance agreement and rules; unlike benchmark construction, it receives no additional human verification.Source: §4.4 · pp. 7–8
  • Fine-tuning improves both models across domains; colqwen2-v1.0 becomes competitive with strong multimodal embedders and surpasses Qwen3-VL-Embedding-8B despite its smaller scale.Source: Table 3 · p. 8, §4.4 · p. 8
  • The fine-tuning setup mixes Ko-VDR Train Public with private Korean visual QA supervision to strengthen structured visual understanding, particularly for tables and figures.Source: §4.4 · p. 8

4.5 Interpretability

  • Figure 5 shows fine-tuned colqwen2-v1.0 assigning high similarity to “선물 시장,” indicating focus on query-relevant Korean text.Source: §4.5 · p. 8, Figure 5 · p. 8

5 Ablation Study

The ablations show that multi-page queries are more challenging as relevant pages increase, while Korean-specific and public training data improve retrieval, with their combination generally performing best. Training-data effects vary for the Economic subset, where private training performs best.

5.1 Effect of the Number of Relevant Pages

  • nDCG@10 generally decreases as queries require more relevant pages, and the 8B model consistently outperforms the 2B model without eliminating multi-page retrieval difficulty.The decline is not strictly monotonic; the results frame queries requiring evidence aggregation across pages as more challenging.Source: §5.1 · pp. 8–9, Figure 6 · p. 9

5.2 Effect of Training Data Composition

  • Combining private and public training data achieves the highest average performance and performs best across most subsets, while public data alone improves most subsets over the base model.The study attributes the combined gains to complementary Korean-specific supervision and large-scale public data coverage.Source: §5.2 · p. 9, Table 4 · p. 9
  • For the Economic subset, private-data-only training performs best, which the authors hypothesize reflects its table-heavy queries and the private data’s TableVQA-style supervision.These queries often require interpreting complex multi-column tables distributed across document pages.Source: §5.2 · p. 9

6 Conclusion

KoViDoRe is a benchmark for Korean visual-document retrieval that emphasizes queries requiring information integration across multiple pages. The paper also reports that Korean-specific training improves retrieval performance and introduces a public training dataset to support this task.

  • KoViDoRe benchmarks Korean visual-document retrieval with queries requiring multi-page information integration, using diverse-layout public documents and a human-verified, LLM-based curation pipeline.Source: §6 · p. 9
  • Current multimodal retrieval models struggle with Korean visual-document retrieval, particularly for structured content and diverse query types.Source: §6 · p. 9
  • Ko-VDR Train Public is a large-scale Korean-specific training dataset, and experiments show that Korean-specific training improves retrieval performance.The authors state that KoViDoRe and Ko-VDR Train Public will facilitate future research on Korean visual-document retrieval.Source: §6 · p. 9

Limitations

KoViDoRe’s limitations concern possible loss of visual information and annotation noise, restricted reproducibility due to private training data, and the absence of a purpose-built retrieval architecture.

Limitations

  • Query generation uses parsed markdown and image captions rather than raw visuals, potentially losing layout, color, and chart patterns and affecting query quality.The authors plan to incorporate vision-language models to better capture visual information and reduce this loss.Source: Limitations · p. 10
  • Consistency-based filtering reduces manual relevance-mapping effort, but imperfect alignment between generation and mapping may still introduce noise.Source: Limitations · p. 10
  • The private Korean VQA dataset improves training performance but cannot be released due to licensing restrictions, limiting full reproducibility to public components.Source: Limitations · p. 10
  • The experiments evaluate existing retrieval models without proposing a Korean visual document retrieval architecture; future work may explore tailored models and training strategies.Potential directions include approaches that directly incorporate visual inputs during query generation.Source: Limitations · p. 10

A Appendix

The appendix provides supplementary materials for transparency and reproducibility, including benchmark comparisons, model-selection rationale, query-generation definitions, pipeline prompts, examples, and source-document metadata.

  • It includes a detailed SDS KoPub VDR comparison, model-selection rationale, query-generation category definitions, pipeline prompt templates, representative query-page examples, and metadata for all source collections.Source: A Appendix · p. 12

A.1 Comparison to SDS KoPub VDR

KoViDoRe complements SDS KoPub VDR (Lee et al., 2025) by evaluating multi-page evidence aggregation rather than predominantly single-page retrieval, reflecting information distributed across reports.

A.1 Comparison to SDS KoPub VDR

  • KoViDoRe associates each query with an average of 2.94 relevant pages, contrasting with SDS KoPub VDR’s largely single-page query mapping (Lee et al., 2025).This formulation targets queries that require synthesizing information from multiple document regions, such as financial trends or policies distributed across a report.Source: §A.1 · p. 12
  • The HR subset reaches a relevant-pages-per-query ratio of 3.30, highlighting KoViDoRe’s emphasis on multi-page evidence gathering.The passage describes this as complementing existing resources by measuring retrieval across multiple pages to meet diverse informational needs.Source: §A.1 · p. 12

A.2 Document Parsing and Query Generation Models

The pipeline uses Upstage Document Parse to preserve layout-aware structure in complex Korean documents and Solar-Pro3 to generate queries from textual and visual content. These model choices support downstream query generation, relevance mapping, and challenging retrieval scenarios.

A.2 Document Parsing and Query Generation Models

  • Upstage Document Parse produces page- and element-level Markdown, decomposes pages into structured components, and captions visual content for semantic interpretation.Its components include text blocks, tables, figures, charts, and diagrams.Source: §A.2 · p. 12
  • Reported DP-Bench structure-preservation performance and layout-aware handling of complex Korean documents motivate Upstage Document Parse for downstream query generation and relevance mapping.Source: §A.2 · p. 12
  • Solar-Pro3 generates queries from Markdown conditioned on textual and visual information, supporting complex Korean queries requiring multistep reasoning.The model is described as reasoning-oriented and designed for structured, context-aware generation.Source: §A.2 · p. 12

A.3 Query Category Definitions … A.7 Distribution of Relevant Pages per Query

The appendices define KoViDoRe’s query categories and generation guidance, show example query-page pairs, document collection metadata, and describe the distribution of relevant pages per query.

A.3 Query Category Definitions

  • Tables 5 and 6 define the query-type and query-format categories used in the dataset construction pipeline.The tables cover query-type categories and their definitions, and query-format categories and their definitions.Source: §A.3 · p. 12, Table 5 · p. 14, Table 6 · p. 14

A.4 Prompt Templates

  • Figures 9–12 present prompts for summary generation, relevance mapping, and summary-based and context-based query generation.The prompts include controls for query realism, diversity, and formulation.Source: §A.4 · p. 12, Figure 9 · p. 15, Figure 10 · p. 16

A.5 Example Query-Page Pairs

A.6 Document Metadata

A.7 Distribution of Relevant Pages per Query

  • Most queries have two or three relevant pages, while less frequent queries with more relevant pages support evaluation of broader evidence aggregation.The distribution is presented as reflecting varying query complexity, including a substantial share requiring multi-page aggregation.Source: §A.7 · p. 13, Figure 7 · p. 13
  • The query-generation guidance requires specific, realistic Korean queries that implicitly connect information across pages without mentioning page locations or exact document values.It also rules out document-aware framing and exact document titles, reflecting a user who knows topic areas, entities, and periods but not document structure or exact values.Source: §A.7 · pp. 17–18
  • Question and instruction formats require complete interrogative sentences and commands, while keyword queries must be coherent noun phrases rather than sentences or keyword bags.Keyword phrases connect nouns with Korean particles, and the read-aloud test flags queries that split into independent chunks without a grammatical bridge.Source: §A.7 · pp. 18, 21

Related papers

Loading 2608.20840v1…