PaperlayerView exact revision on arXiv

Source-linked AI summary

PalmLeaf-VQA: A Multi-Script Visual Question Answering Benchmark for Historical Palm-Leaf Manuscript Understanding Across Diverse Regions

Nimol Thuon, Jun Du, Panhapin Theang

arXiv:2609.31651v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Historical manuscripts are underrepresented in modern vision-language benchmarks, leaving limited evidence about MLLMs’ ability to reason beyond transcription and layout parsing. PalmLeaf-VQA provides a multi-script benchmark of visually grounded manuscript questions; on it, Gemini 3.5 Flash reaches only 58.00% exact-match accuracy under constrained-answer prompting.

  • Problem

    Existing historical-document benchmarks provide limited evaluation of multimodal reasoning beyond transcription and layout parsing, despite manuscripts’ material, spatial, and cultural attributes.

    Source: §1 · p. 1

  • Method

    PalmLeaf-VQA benchmarks manuscript-aware visual question answering with 923 images and 7,384 pairs across eight collection groups and eight visually grounded categories.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    58.00% exact-match accuracy: Gemini 3.5 Flash achieves this strongest reported result under constrained-answer prompting, while persistent errors span rare-script recognition, localized physical-feature analysis, line counting, and multi-attribute prediction.

    Source: §7 · p. 9

  • Takeaways & Limitations

    PalmLeaf-VQA establishes a benchmark for evaluating visual question answering over historical palm-leaf manuscripts as both textual documents and physical artifacts.

    Source: §7 · p. 9

    The eight collection groups do not represent the full geographic, linguistic, chronological, or material diversity of palm-leaf manuscripts, and incomplete metadata precludes claiming fully manuscript-disjoint splits.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

Historical manuscripts remain largely absent from modern vision-language benchmarks, leaving open how well multimodal large language models (MLLMs) handle culturally diverse, degraded, and non-Latin document images. We introduce \textbf{PalmLeaf-VQA}, a multi-script visual question answering benchmark for historical palm-leaf manuscript understanding across South and Southeast Asian traditions. PalmLeaf-VQA contains \textbf{923 curated manuscript images} and \textbf{7,384 question--answer pairs} from eight collection groups: Balinese, Grantha, Jathakam, Kambaramayanam, Kannada, Khmer, Sundanese, and Tamil. Unlike recognition-oriented resources, the benchmark targets manuscript-aware visual reasoning over preservation-relevant cues, including physical condition, line structure, material and coating, binding holes, margins, symbols, drawings, and localized visual artifacts. We evaluate recent proprietary and open-weight MLLMs under open-answer and constrained-answer prompting and provide fine-grained analysis across collections, question categories, and task types. The strongest evaluated model reaches only \textbf{58.00\% exact-match accuracy} on the held-out test split, revealing substantial limitations in current MLLMs for rare-script, degraded-layout, and preservation-oriented document understanding. PalmLeaf-VQA provides a standardized benchmark for advancing culturally grounded and layout-aware multimodal document analysis.

1 Introduction

Historical palm-leaf manuscripts require models to reason about written content alongside material, spatial, and cultural attributes, yet existing benchmarks offer limited evidence on this capability. PalmLeaf-VQA introduces a multi-script benchmark and evaluates current MLLMs, which remain far from solving the task.

1 Introduction

  • Existing historical-document benchmarks provide limited evaluation of multimodal reasoning beyond transcription and layout parsing, while current VQA resources rarely require joint reasoning over manuscripts’ scripts, materials, layouts, and degradation.The gap leaves limited evidence about whether current MLLMs understand historical manuscripts rather than modern document types.Source: §1 · p. 1
  • Historical palm-leaf images combine rare or non-Latin scripts with irregular layouts, degradation, binding structures, and preservation cues.These features make manuscripts difficult to interpret as material objects as well as written content.Source: §1 · p. 1
  • PalmLeaf-VQA evaluates historical palm-leaf understanding across 923 images and 7,384 question–answer pairs from eight collection groups, using image-disjoint splits.Its categories address both manuscripts as textual documents and as physical artifacts.Source: §1 · pp. 1–2
  • The benchmark covers eight manuscript-aware categories and evaluates classification, counting, layout reasoning, binary recognition, and preservation-aware attribute understanding.The categories include physical condition, line count, symbols and drawings, material and coating, binding and holes, and scribe and margins.Source: §1 · p. 2, Figure 1 · p. 2
  • The evaluation finds that historical palm-leaf manuscript understanding remains far from solved for current general-purpose MLLMs.The study uses zero-shot evaluation of recent proprietary and open-weight models across the benchmark’s categories.Source: §1 · p. 2

2 Related Work

VQA and MLLM benchmarks have expanded from natural images to text-rich and document-centric tasks, while historical palm-leaf research has focused mainly on recognition, segmentation, and restoration. PalmLeaf-VQA instead assesses broader visual questions about manuscripts as documents and physical artifacts.

2.1 Document VQA and MLLM Evaluation

  • VQA benchmarks progressed from natural-image reasoning to scene text, scanned documents, infographics, charts, multi-domain documents, and long PDF-style inputs.Broader MLLM evaluations also assess perception, reasoning, knowledge, and instruction following, but rarely test historical manuscripts jointly across scripts, materials, layouts, and degradation.Source: §2.1 · pp. 2–3

2.2 Historical and Palm-Leaf Manuscript Analysis

  • Historical document analysis addresses degraded materials, while palm-leaf manuscripts add elongated geometry, nonpaper supports, low-contrast strokes, binding holes, coatings, surface damage, and complex non-Latin scripts.These characteristics make palm-leaf documents a particularly challenging subset.Source: §2.2 · p. 3
  • Prior palm-leaf resources primarily target recognition, segmentation, or restoration; PalmLeaf-VQA evaluates broader visual questions about manuscripts as textual documents and physical artifacts.Existing resources include work on binarization, line segmentation, glyph and word recognition, transliteration, enhancement, damage segmentation, and layout analysis.Source: §2.2 · p. 3

3 PalmLeaf-VQA Benchmark

PalmLeaf-VQA unifies curated palm-leaf manuscript collections as a manuscript-aware visual question answering benchmark. It combines balanced visual categories, culturally informed annotation, and image-disjoint splits while noting that available metadata does not support a fully manuscript-disjoint claim.

3 PalmLeaf-VQA Benchmark

  • PalmLeaf-VQA reformulates curated palm-leaf manuscript collections as a unified benchmark for manuscript-aware visual question answering, with image-disjoint splits and eight question–answer pairs per image.Table 2 reports dataset statistics by collection group.Source: §3 · p. 3, Table 2 · p. 3

3.1 Source Collections and Curation

  • The benchmark draws on public and research-oriented resources, removes unusable or duplicate captures, and applies cropping or framing normalization only when needed while preserving manuscript features.Its eight collection groups include Balinese, Grantha, Jathakam, Kambaramayanam, Kannada, Khmer, Sundanese, and Tamil; these labels do not always map one-to-one to scripts.Source: §3.1 · p. 3

3.2 Question Taxonomy and Answer Design

  • Each image has one question in each of eight visually grounded categories, mapped to five VQA task types: classification, counting, binary recognition, layout reasoning, and attribute recognition.Answers may be single- or multi-label, using canonical vocabularies, standardized binary labels, numeric line counts, and unordered sets for multi-label responses.Source: §3.2 · pp. 3–4

3.3 Annotation and Validation

  • Twelve annotators with relevant regional, script, or collection familiarity wrote and independently reviewed each question–answer pair, resolving disagreements through adjudication and excluding unresolved cases.Annotations were checked for visual answerability, category consistency, and agreement with image evidence.Source: §3.3 · p. 4

3.4 Quality Control, Splits, and Release

  • The 923 images are divided into image-disjoint training, validation, and test splits of 553, 92, and 278 images, respectively.Where reliable identifiers were unavailable, grouping used source folders, filenames, metadata, and near-duplicate clusters, so the benchmark is not claimed to be fully manuscript-disjoint.Source: §3.4 · p. 4

4 Evaluation Protocol

The evaluation tests models on image-disjoint manuscript examples under open-answer and constrained-answer prompting, with fixed normalization and exact-match scoring. It also analyzes results across skills and prompting conditions and compares model performance with human responses on a balanced subset.

4.1 Task and Test Setting

  • Models receive only the image and question, with zero-shot evaluation on 2,224 question–answer pairs from 278 image-disjoint test images.Validation finalizes normalization rules and prompts before test evaluation; training examples are not used in reported experiments.Source: §4.1 · p. 4

4.2 Prompting Conditions

  • Open-answer prompting permits free-form responses, whereas constrained-answer prompting supplies an output format and, when applicable, a canonical answer vocabulary.The image, question, and reference remain unchanged, separating visual-understanding errors from lexical and formatting variation.Source: §4.2 · pp. 4–5

4.3 Answer Normalization and Metrics

  • Exact-match accuracy compares normalized predictions and references, while additional metrics capture partial correctness and within-one accuracy for line counts.Normalization standardizes case, punctuation, script and collection names, binary labels, numeric answers, and unordered multi-label sets; invalid or unparseable outputs count as incorrect.Source: §4.3 · p. 5

4.4 Fine-Grained and Human Evaluation

  • The analysis breaks down performance by collection, question category, task type, answer cardinality, and prompting condition, and evaluates humans on a balanced test subset.Three independent human responses are collected per question; human EM and F1 average 768 responses, while W1 uses 96 responses to 32 line-count questions.Source: §4.4 · p. 5

5 Experiments

The experiments evaluate proprietary and open-weight MLLMs on the held-out benchmark under open-answer and constrained-answer prompting, then compare results across collections, categories, and human responses. Performance remains limited and varies with prompting, collection, and the kind of visual reasoning required.

5 Experiments; 5.1 Models and Experimental Setup

  • The study measures recent proprietary and open-weight MLLMs, plus text-only controls, on the same 2,224 held-out questions under both answer settings.Models are evaluated without task-specific fine-tuning, using exact-match accuracy as the primary metric and mean answer F1 and within-one line-count accuracy as complementary measures.Source: §5 · p. 5, §5.1 · p. 5

5.6 Human Baseline

  • 87.50% human EM, 92.40% F1, and 96.88% W1 on a balanced 256-question subset exceed the corresponding model EM scores by 28.91 points for Gemini 3.5 Flash and 27.73 for ChatGPT 5.5.Eight domain-experienced participants supplied three responses per question.Source: Table 5 · p. 6, §5.6 · p. 7

5.2 Overall Performance

  • Under constrained-answer prompting, Gemini 3.5 Flash is strongest, scoring 58.00% EM, 67.25% F1, and 93.88% W1; under open-answer prompting, it also leads with 43.03% EM.Its F1 exceeds EM by more than nine points under constrained prompting, suggesting models may recover partially correct information without producing the full canonical answer.Source: §5.2 · p. 6

5.3 Effect of Prompting and Answer Cardinality

  • Constrained-answer prompting improves EM for all evaluated models, averaging a 17.52-point gain across eleven paired comparisons.Gemini 3.5 Flash rises from 43.03% to 58.00% EM; even with constrained prompting, the strongest model remains incorrect on more than two-fifths of questions.Source: §5.3 · p. 6, Figure 3 · p. 7

5.4 Collection-Level Performance

  • 14.8 points separate Gemini 3.5 Flash's 65.8% Balinese result from its 51.0% Kambaramayanam result, while collection leaders vary across models.InternVL3.5-8B leads Grantha at 67.3%, and ChatGPT 5.5 leads Kannada and Tamil at 59.4% and 55.2%; collection difficulty is not explained simply by sample size.Source: §5.4 · pp. 6–7, Table 4 · p. 6

5.5 Category-Level Performance

  • Material and coating and binding and holes are recognized more reliably than physical condition, margins, and other physical features.Line-count models often recover approximate structure but miss the exact count by one; script or collection identification also remains difficult and model-dependent.Source: §5.5 · p. 7, Figure 4 · p. 8

5.7 Error Analysis

  • The recurring failures are rare-script confusion, missed localized physical features, line-count ambiguity, and incomplete or non-canonical answers.Constrained prompting reduces output drift but does not remove visual-reasoning limitations; progress requires combining rare-script representation, localized attention, preservation-aware reasoning, precise counting, and controlled multi-label generation.Source: §5.7 · pp. 7–8

6 Discussion and Limitations

PalmLeaf-VQA extends document VQA to historical manuscripts as textual documents and physical artifacts, but current MLLMs still struggle across several manuscript-specific tasks. The benchmark's coverage and annotations have defined limits, and the authors identify extensions for future work.

6.1 Main Findings

  • The strongest model reaches 58.00% EM, while constrained-answer prompting improves EM by 17.52 points on average; rare-script identification, localized damage, line counting, and multi-attribute prediction remain difficult.Performance also varies substantially across collections, with script appearance, image quality, page geometry, and preservation condition affecting model behavior.Source: §6.1 · p. 8

6.2 Scope and Limitations

  • The eight collection groups do not cover palm-leaf manuscripts' full geographic, linguistic, chronological, or material diversity, and incomplete metadata prevents claiming fully manuscript-disjoint splits.Annotations cover visually observable properties, not transcription, translation, dating, authorship, or deeper philological interpretation; exact-match scoring may penalize semantically reasonable non-canonical answers.Source: §6.2 · p. 8

6.3 Future Directions

  • Future extensions include newly digitized traditions, stronger provenance, richer preservation labels, region-grounded annotations, and broader expert evaluation.The authors also propose connecting visual inspection with transcription and historical metadata for broader cultural-heritage analysis.Source: §6.3 · p. 8

7 Conclusion

PalmLeaf-VQA evaluates manuscript-aware visual reasoning across eight collection groups and eight categories using 923 images and 7,384 question–answer pairs. The strongest evaluated model reaches 58.00% exact-match accuracy, while the benchmark provides a reproducible basis for developing preservation-aware multimodal analysis.

7 Conclusion

  • PalmLeaf-VQA contains 923 curated images and 7,384 annotated question–answer pairs across eight collection groups and eight manuscript-aware categories.Its tasks cover classification, counting, layout reasoning, and preservation-related visual understanding.Source: §7 · p. 9
  • 58.00% exact-match accuracy is the strongest result, achieved by Gemini 3.5 Flash under constrained-answer prompting.Persistent errors span rare-script recognition, localized physical-feature analysis, line counting, and multi-attribute prediction.Source: §7 · p. 9
  • PalmLeaf-VQA provides a reproducible benchmark for developing multimodal models with stronger rare-script representations, localized visual attention, structural precision, and preservation-aware reasoning.Source: §7 · p. 9

Supplementary Material

The supplementary material provides details omitted from the main paper because of space limitations, covering dataset methods, evaluation, experiments, analysis, and release information.

Supplementary Material

  • The supplement expands on information omitted from the main paper because of space limitations.Source: Summary · p. 13
  • It details dataset harmonization, annotation and validation, machine-readable formats, prompting and model configurations, answer normalization, and human evaluation.Source: Summary · p. 13
  • It also provides extended experiments, error analysis, licensing, and release organization details.Source: Summary · p. 13

A Additional Dataset Construction Details

This appendix documents PalmLeaf-VQA’s dataset curation, annotation, splitting, standardization, packaging, and model-evaluation setup, including limits on split independence and image redistribution. It also specifies answer normalization and scoring, human evaluation and agreement measurement, and prediction-record and failure logging.

A.1 Source Harmonization

  • Source images were assigned benchmark identifiers and provenance links, with manuscript features preserved rather than geometrically normalized.Framing adjustments were limited to removing unrelated borders or acquisition artifacts; writing placement, holes, margins, and degradation were retained.Source: §A.1 · p. 13

A.2 Image Filtering and Quality Control

  • Images with insufficient visual evidence or severe capture problems were excluded, but interpretable natural degradation was retained as part of benchmark difficulty.Exact duplicates were removed and verified near-duplicates grouped before splitting; images from a known manuscript were kept together when reliable identifiers were available, so splits are image-disjoint rather than fully manuscript-disjoint.Source: §A.2 · p. 13, §A.3 · p. 13, §A.4 · pp. 13–14

A.5 Annotation Workflow and Validation

  • Twelve regionally, script-, or collection-familiar researchers annotated the data, with independent review and adjudication for visually unsupported or ambiguous cases.The workflow also normalized labels, counts, punctuation, and multi-label formatting.Source: §A.5 · p. 14

A.6 Question Construction Principles

  • Each image receives one visually grounded question per category, while transcription, translation, dating, author identification, and deeper philological interpretation are excluded.Questions use consistent phrasing and predefined categorical, binary, numerical, or multi-label answer formats.Source: §A.6 · p. 14

A.7 Answer Types and Canonicalization

  • Answers are canonicalized by type, and each question–answer pair is stored as an independent JSON record linked to its image and split.Multi-label answers are unordered canonical sets, binary answers use yes/no, and visible-line counts are integers; provenance and licensing information reside in separate artifacts.Source: §A.7 · p. 14, §A.8 · p. 14, Table 7 · p. 15

A.9 Source Provenance and Release Organization

  • The release separates annotations from image permissions and includes provenance metadata; its supplementary sample contains 80 images and 640 question–answer records.Restricted sources are accompanied by annotations, identifiers, retrieval instructions, and processing scripts rather than redistributed images; the sample spans all collection groups and question categories where redistribution permits.Source: §A.9 · p. 15, §A.10 · p. 15

B.2 Evaluation Conditions

  • Models were evaluated zero-shot on the same 2,224 test questions in open-answer and constrained-answer conditions, with image and question held constant.Open-weight inference used two 80GB A800 GPUs, while proprietary models ran through APIs; the text-only controls received no manuscript image.Source: §B.2 · p. 16, §B.1 · p. 16

B.4 Open-Answer Prompt

  • Constrained prompts add category-specific formats and vocabularies, while answer rules specify canonical labels, unordered multi-label sets, yes/no, or integer counts.Open-answer prompts request image-grounded responses without external knowledge; outputs were stored with model, prompt condition, raw and normalized responses, references, and metrics.Source: §B.4 · p. 16, §B.5 · p. 17, §B.6 · p. 17

B.7 Prediction Record Format

  • Prediction records include the sample, model, prompt condition, prediction, reference, and scoring fields, and API failures, empty outputs, refusals, and unparseable responses are logged.These cases are logged rather than silently removed.Source: §B.7 · p. 17

C.1 Normalization Procedure

  • Normalization standardizes answer forms without changing meaning, while exact match compares normalized answers and requires complete-set equality for multi-label questions.Processing includes case and whitespace normalization, punctuation cleanup where appropriate, canonical labels and aliases, binary-label standardization, and numerical parsing for line counts; repeated labels and invalid or unparseable outputs are handled explicitly.Source: §C.1 · pp. 17–18, §C.2 · p. 18

C.3 Partial-Credit Metrics

  • Multi-label scoring reports precision, recall, F1, and Jaccard similarity; line-count scoring uses count-exact and within-one accuracy.The released scoring code specifies behavior for empty predicted or reference sets.Source: §C.3 · p. 18, §C.4 · p. 18

D.1 Balanced Subset Construction

  • The human-evaluation subset contains 256 questions, with four questions for each of eight collection groups crossed with eight question categories.This stratification gives equal representation to collections and categories rather than overrepresenting larger collections or easier question types.Source: §D.1 · p. 18

D.2 Participants and Assignment

  • Human evaluation used eight domain-experienced participants, with three independent responses per question and no self-evaluation of questions they annotated.Participant instructions restricted judgments to visible image evidence and the supplied constrained-answer vocabulary.Source: §D.2 · p. 18

D.3 Evaluation Instructions

  • Participants received each manuscript image and question with the models’ constrained-answer vocabulary, and were told to use visible evidence rather than external historical or linguistic knowledge.Required response formats were one canonical answer, yes/no, one integer, or all applicable canonical labels, depending on question type.Source: §D.3 · p. 18

D.4 Human-Score Aggregation

  • Human EM and F1 average scores across 768 individual responses, while majority voting is not used.Human W1 is computed over the 32 visible-line-count questions; model EM and F1 are computed over the 256-question subset and model W1 over those 32 questions.Source: §D.4 · p. 18, Table 11 · p. 19

D.5 Response Agreement

  • Response consistency is measured by mean pairwise exact agreement after applying the same normalization used for model evaluation.Agreement is averaged within each category and then macro-averaged across the eight categories to accommodate heterogeneous answer spaces.Source: §D.5 · p. 19

D.6 Human Baseline Results

Human responses provide a strong baseline, while the supplement details the evaluation release and complementary analyses across categories, collections, answer cardinality, and line-count errors.

D.6 Human Baseline Results

  • 87.50% EM, 92.40% F1, and 96.88% W1 were achieved by human participants, with 86.0% mean pairwise exact agreement.Human–model EM gaps were 28.91 points for Gemini 3.5 Flash and 27.73 points for ChatGPT 5.5.Source: §D.6 · p. 19

D.7 Human Evaluation Release

  • The permitted public human-evaluation release includes sampled-question identifiers, anonymous participant IDs, responses, scores, agreement statistics, assignments, and instructions, but excludes names and personally identifying information.The release is conditional on permission.Source: §D.7 · p. 19

E Extended Experimental Results

E.1 Complete Category-Level Results

  • Category-level tables report all evaluated models across eight question categories and both prompting conditions, with exact-match accuracy and applicable task-specific metrics.Machine-readable tables are identified as results/per_category_metrics.csv.Source: §E.1 · p. 19

E.2 Complete Collection-Level Results

  • Collection-level supplement tables add open-answer results and, where informative, F1 or Jaccard similarity to the main paper’s constrained-answer accuracy.Machine-readable tables are identified as results/per_category_metrics.csv.Source: §E.2 · p. 19

E.3 Answer-Cardinality Analysis

  • Single-answer and multi-answer questions are reported separately, with multi-answer evaluation including exact match, precision, recall, F1, and Jaccard similarity.These metrics account for cases where a model recovers only a subset of required labels.Source: §E.3 · p. 20

E.4 Line-Count Error Analysis

  • Line-count analysis reports count-exact accuracy, W1, and absolute counting-error distributions to distinguish near misses from larger layout failures.Source: §E.4 · p. 20

E.5 Additional Ablations

The supplement lists additional prompting, normalization, resolution, and image-conditioning comparisons, and describes aggregate and qualitative analyses of model failures.

E.5 Additional Ablations

  • The listed ablations compare open-answer with constrained-answer prompting, vocabulary instructions, answer normalization, image resolution, and image-conditioned models with text-only controls.Completed open-versus-constrained and text-only-control results are reported in machine-readable tables; the passage does not give their outcomes.Source: §E.5 · p. 20

F Extended Error Analysis

  • The extended error analysis combines aggregate failure-category counts with examples examining how prompt format, visual ambiguity, localized evidence, and answer cardinality affect predictions.Source: F Extended Error Analysis · p. 20

F.1 Aggregate Error Distribution

The analyses identify recurring visual and answer-generation failures in PalmLeaf-VQA, while defining limits on the benchmark’s intended use and emphasizing cultural context.

F.1 Aggregate Error Distribution

  • Category failure counts are comparable because each model receives the same number of questions per category; the released distribution covers 30,897 errors across 20 model–prompt runs.The figure pools 30,897 incorrect row-level predictions; summary-only runs without row-level files are excluded, and visible-line-count errors use exact-count correctness.Source: §F.1 · p. 20, Figure 5 · p. 20

F.2 Qualitative Comparison of Prompting Conditions

  • Constrained-answer prompting reduces lexical variation and output drift, but visual-reasoning errors persist, including missed localized features and incorrect line counts.Open answers may substitute broad or non-canonical labels for benchmark labels.Source: §F.2 · pp. 20–21

F.3 Summary of Failure Patterns

  • Four dominant failures are rare-script or collection confusion, missed localized features, line-count ambiguity, and incomplete or drifting answers.The paper says progress requires more than stronger OCR, naming rare-script representation, localized attention, preservation-aware reasoning, precise counting, and controlled multi-label generation.Source: §F.3 · p. 21

G.1 Intended and Unsupported Uses

  • PalmLeaf-VQA is intended for research and evaluation, not historical authentication, ownership decisions, automated conservation intervention, cultural attribution without expert review, or complete transcription and translation.Source: §G.1 · p. 21

G.2 Cultural-Heritage Considerations

  • Predictions should not replace manuscript experts, conservators, source institutions, or relevant communities, and users should respect collection access conditions and cultural context.The paper notes that palm-leaf manuscripts may contain culturally, historically, or religiously significant material.Source: §G.2 · p. 21

G.3 Planned Release Components

  • The authors plan to release the dataset and associated resources, subject to source-specific redistribution permissions, through GitHub and Hugging Face.Planned materials include results, evaluation code, prompts, and documentation.Source: §G.3 · p. 21
Loading 2609.31651v1…