PaperlayerView exact revision on arXiv

Source-linked AI summary

HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers

Issa Sugiura, Shuhei Kurita, Yusuke Oda, Naoaki Okazaki

arXiv:2606.01132v1cs.CV

TL;DR

Non-English chart and table VQA lacks realistic, diverse benchmarks, leaving multilingual generalization unclear. The paper constructs HakushoBench from Japanese governmental white papers and finds that open-weight VLMs remain substantially behind proprietary models on complex understanding.

  • Problem

    Existing chart and table benchmarks are heavily English-centric, while Japanese benchmark coverage is limited and JGraphQA is small, visually narrow, and relatively simple.

  • Method

    HakushoBench collects and filters chart and table images from 33 Japanese governmental white papers, then adds manually annotated high-difficulty QA pairs with verification.

  • Results

    HakushoBench is more challenging than JGraphQA; the best open-weight model reaches 58.6% accuracy, with a 34.9-point gap separating the best proprietary and open-weight models.

  • Takeaways & Limitations

    Governmental white papers provide a scalable source for realistic, diverse Japanese chart and table benchmarks, while open-weight models still fall short on complex understanding.

  • Takeaways & Limitations

    HakushoBench is limited to Japanese governmental white papers, so it does not directly cover other low-resource languages or all real-world visual styles and domains.

Abstract

from arXiv · show

Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, non-English counterparts remain scarce, leaving it unclear whether this progress generalizes across languages. A key obstacle is the difficulty of collecting realistic and diverse non-English chart and table images at scale. To address this, we leverage governmental white papers as a scalable source for benchmark construction beyond English, as they contain naturally occurring charts and tables across diverse formats and domains and are freely accessible in many countries. As a first instantiation, we introduce HakushoBench, a challenging Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated QA pairs, designed to assess deep and holistic understanding of charts and tables, rather than local visual cues alone. Experiments across a broad range of VLMs demonstrate that HakushoBench remains challenging for open-weight models: the best open-weight model achieves only 58.6% accuracy, and a 34.9-point gap between open-weight and proprietary models highlights substantial room for improvement in complex chart and table understanding. We release our dataset and code.

1 Introduction

HakushoBench addresses the scarcity of challenging, diverse Japanese chart and table VQA benchmarks by using governmental white papers as a realistic data source. It contains 2,053 images and remains difficult for open-weight VLMs.

  • Chart and table VQA matters because real-world documents often convey information that cannot be expressed in text alone.
  • Existing benchmarks are English-centric, while visual composition, terminology, writing direction, density, and reasoning requirements vary across languages and cultures.
  • JGraphQA contains only around 200 Japanese examples with limited visual diversity and simple questions, enabling even 3B-scale VLMs to exceed 80% accuracy.
  • Governmental white papers provide publicly available charts and tables spanning diverse domains and visual formats for benchmark construction.
  • HakushoBench comprises 2,053 unique images from 33 Japanese governmental white papers and more than 10 image types, with questions requiring holistic integration and multi-hop reasoning.
  • 58.6% accuracy is achieved by the best open-weight model, while HakushoBench shows a 34.9-point gap between the best proprietary and open-weight models.

2 Related Work

Chart and table VQA benchmarks have progressed from synthetic or simple English datasets toward more diverse real-world sources, but non-English evaluation remains limited. HakushoBench contributes a realistic and challenging Japanese benchmark based on governmental white papers.

  • Early chart VQA datasets used synthetic charts or focused on straightforward extraction, while later benchmarks introduced real-world images and more complex reasoning.
  • ChartQAPro and CharXiv broadened image sources through dashboards, infographics, and research-paper charts, yet benchmark development remains primarily English-focused.
  • Japanese document benchmarks include JDocQA from public-institution PDFs and BusinessSlideVQA from Japanese business slides.
  • JGraphQA uses a narrow corporate-slide source, around 200 examples, and comparatively simple extraction and arithmetic questions.
  • HakushoBench uses governmental white papers to provide realistic, visually diverse, and challenging Japanese chart and table VQA.

3 Construction of HakushoBench

HakushoBench is built by collecting and filtering chart and table images from Japanese governmental white papers, then creating and independently verifying difficult short-answer questions. The resulting benchmark spans many image types and domains.

  • The construction pipeline collects chart and table images from Japanese white papers, filters unsuitable images, annotates difficult questions, and independently verifies every QA pair.
  • White papers are selected through HTML editions because directly extracting figures and tables from PDFs remains challenging for current OCR models.
  • 18,539 collected images are manually reduced to 5,903 candidate chart and table images after removing photographs, low-resolution images, and near duplicates.
  • Questions must require the image, be natural and unambiguous, and have a short answer; evaluation uses an LLM-based judge tolerant of surface-form variation.
  • Accepted QA pairs target global integration, multi-hop reasoning, counting, external knowledge, and fine-grained visual perception.
  • Annotators may skip images lacking sufficiently challenging questions, preventing artificially simple QA pairs from entering the dataset.
  • HakushoBench contains over ten times more examples than JGraphQA and covers a broader range of image types while remaining competitive with ChartQAPro and CharXiv in scale and diversity.

4 Exploring HakushoBench

HakushoBench combines broad topic coverage with substantial visual and question diversity. Its examples require holistic interpretation, including multi-value computation, spatial reasoning, and comparison of embedded text.

  • Dataset statistics: HakushoBench contains 2,053 examples from 33 white papers across six topics, with Economy and Society comprising 30.4% and 23.3%, respectively.Each question may receive multiple difficulty flags; Global is the most frequent.
  • Visual diversity: The benchmark classifies images into 11 visual-format categories, including charts, maps, tables, infographics, dashboards, and other formats.The taxonomy adds Map and Table to the ChartQAPro categories, with Gemini 3 Pro classification manually verified on 100 examples.
  • Visual diversity: HakushoBench has higher embedding diversity than JGraphQA, with mean pair-wise cosine distance of 0.365 versus 0.275.Diversity is measured using SigLIP2 image embeddings.
  • Question complexity: Representative questions require extracting and computing multiple table values, interpreting map locations and legends, and comparing embedded textual descriptions.These tasks illustrate information-dense images and reasoning beyond local visual cues.

5 Experiments

The experiments evaluate diverse open-weight and proprietary VLMs under direct-answering and chain-of-thought prompting, using accuracy across HakushoBench and four comparison benchmarks.

  • Models: The evaluation covers multilingual and Japanese-centric open-weight models alongside GPT-4o, GPT-5.1, and Gemini 3 Pro.Open-weight models include Qwen3-VL, InternVL3.5, Sarashina2.2-Vision, and LLM-jp-4-VL.
  • Prompt settings: Direct prompting requests concise answers, whereas CoT prompting additionally instructs models to reason step by step.Both settings are evaluated for each model.
  • Evaluation: Accuracy is judged by GPT-5.1, and each evaluation is repeated three times with the mean score reported.The LLM judge tolerates minor surface-level phrasing variations.
  • Compared benchmarks: Models are additionally evaluated on JGraphQA, ChartQA, ChartQAPro, and CharXiv to compare HakushoBench with established benchmarks.JGraphQA uses the cleaned and corrected JGraphQA-Verified version.

6 Results

HakushoBench is difficult for open-weight models and exposes substantial differences across model families and prompting strategies. Even the strongest model exhibits perception, knowledge, and counting errors.

  • Main results: 93.5% is Gemini 3 Pro’s highest score, compared with 58.6% for the best open-weight model, Qwen3-VL 8B.GPT-5.1 scores 67.9%, showing substantial variation among proprietary models.
  • Main results: CoT prompting improves GPT-5.1, Qwen3-VL, and InternVL3.5 by more than 10 points, but yields limited gains or degradation for several other models.Manual analysis associates weak gains with failure to engage in reasoning or repetitive, incoherent chains.
  • Benchmark comparison: Qwen3-VL 8B scores 58.6% on HakushoBench versus 88.8% on JGraphQA, while Sarashina2.2-Vision 3B scores 37.7% versus 81.0%.The accuracy spread is broadly consistent across image and question types, except for infographics.
  • Benchmark comparison: 34.9 points separate the best proprietary and open-weight models on HakushoBench, versus 8.1 points on JGraphQA and 1.4 points on ChartQA.The HakushoBench gap is comparable to ChartQAPro’s 30.7-point gap.
  • Error analysis: Gemini 3 Pro’s remaining errors include misreading scatter-plot spatial relationships, identifying geographic locations, and off-by-one counting.The analysis covers 50 randomly sampled incorrectly answered questions.

7 Conclusion

HakushoBench is a diverse Japanese chart and table VQA benchmark sourced from governmental white papers. Results show that it remains challenging for open-weight models and complex visual understanding.

  • Conclusion: HakushoBench contains 2,053 VQA pairs from 33 governmental white papers across 10 distinct image types.The benchmark provides broad domain coverage and visual diversity.
  • Conclusion: The best open-weight model reaches only 58.6% accuracy, while the benchmark reveals a large proprietary–open-weight gap.These results indicate that open-weight VLMs still fall short on complex chart and table understanding.

Limitations

HakushoBench has three main limitations: it is restricted to Japanese governmental white papers, contamination cannot be completely ruled out, and it offers limited discrimination among frontier models.

  • Language and domain coverage: The dataset is limited to Japanese and may not cover the visual styles and domains of other real-world documents because it uses only governmental white papers.The construction approach can nevertheless be extended to other languages and cultural contexts because analogous reports exist in many countries.
  • Potential data contamination: Potential contamination from derivative web content cannot be completely ruled out, although the QA pairs were newly created through manual annotation.Using the most recent edition of each white paper was intended to mitigate contamination risk.
  • Saturation at the frontier: Gemini 3 Pro reaches 93.5%, leaving limited headroom for distinguishing among frontier models.Constructing harder questions could reduce saturation, but may risk producing unnatural questions that depart from realistic use cases.

Ethical Considerations

The benchmark uses publicly released government white-paper images, with manual filtering indicating negligible privacy, personal-data, and NSFW risks; its code is Apache 2.0 licensed, while raw images are not distributed.

  • Public data sources and safety: All images come from publicly released Japanese governmental white papers, making privacy, personally identifiable information, and NSFW risks negligible.Manual filtering found no problematic content.
  • Data and code release: HakushoBench and its evaluation code are released under the Apache 2.0 License, but only image URLs are distributed.The raw image data are not included in the distribution.

E Accuracy Spread by Question Type on HakushoBench

Figure 9 presents model accuracy spread on HakushoBench, grouped by question type, while the comparison figures report per-model Direct and CoT accuracy on four benchmarks.

  • HakushoBench: Figure 9 shows the accuracy spread across models on HakushoBench grouped by question type.The passage identifies the figure’s grouping but does not report individual accuracy values.
  • Comparison benchmarks: Figures 10–13 visualize each model’s Direct and CoT accuracy on ChartQA, ChartQAPro, CharXiv, and JGraphQA as bar charts.These figures provide the comparison-benchmark performance views referenced alongside Figure 9.

G Image Showcases for Comparison Benchmarks

The paper provides representative image showcases for HakushoBench and four comparison benchmarks, showing one randomly sampled image per image-type category for each benchmark.

  • Comparison benchmarks: Figures 14–17 show one randomly sampled image per image-type category for ChartQA, JGraphQA, ChartQAPro, and CharXiv.The figures are presented as image showcases for comparison benchmarks.
  • Performance context: Figures 10–13 report model performance on ChartQA, ChartQAPro, CharXiv, and JGraphQA, complementing the image showcases.The captions identify the corresponding benchmark-performance figures.
Loading 2606.01132v1…