PaperlayerView exact revision on arXiv

Source-linked AI summary

KhatianDoc: A Human-Verified Benchmark Diagnosing Multimodal LLM Failure on Bengali Legal Land Records

Tasmiad Hasan, Arafat Zaman Ratul, Sarker Sadman Saalim, S. M. Shah Nawaz Hossain, Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi

arXiv:2609.03597v1cs.CL

TL;DR

KhatianDoc addresses the absence of benchmarks for handwritten Bengali legal land records using an unsupported base-16 fraction system. It builds and verifies a four-task dataset, then evaluates six multimodal LLMs under a fixed zero-shot protocol. The models show systematic failure across legal QA and arithmetic, while the paper openly audits and corrects evaluation artifacts.

  • Problem

    No benchmark tests whether machines can read RS Khatians, authoritative Bengali land records whose handwritten Ana-Ganda base-16 numerals lack mainstream font, OCR, and tokenizer support.

  • Method

    KhatianDoc combines 107 real RS Khatian records, four tasks, lawyer-verified manual ground truth, positional-token anonymization, and fixed zero-shot evaluation of six multimodal LLMs.

  • Results

    Across six multimodal LLMs, 39.3% of the stratified evaluation set receives zero correct answers from every model, and every emitted arithmetic number underperforms a context-free mean baseline.

  • Takeaways & Limitations

    KhatianDoc documents an absent capability in current multimodal systems and provides verified ground truth for future systems.

  • Takeaways & Limitations

    All 107 documents come from one Mouza in Munshiganj, so performance gains may not transfer to more varied Khatian corpora without further evaluation.

Abstract

from arXiv · show

Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs, no mainstream font, and no coverage in any OCR pipeline or tokenizer. The handwritten records that carry these fractions, RS Khatians, are the authoritative title record for millions of parcels and a frequent subject of civil litigation, yet no benchmark has asked whether a machine can read one. We introduce KhatianDoc, a four-task benchmark built from 107 real RS Khatian records from the Vumi (land) Office of Munshiganj, Bangladesh: symbol recognition, base-16-to-decimal conversion, structured field extraction, and legal document question answering over 1,634 QA pairs. Ground truth was transcribed by hand, verified by a land-law practitioner to full agreement, and anonymized through positional tokens that keep the referential distinctions multi-hop questions depend on. We evaluate six multimodal LLMs (8B to 72B+, open and closed) under a fixed zero-shot protocol. Five QA categories, 39.3% of our stratified set, return zero correct answers from every model; on the arithmetic task, every model that emits a number does worse than a constant-mean baseline, with exact- and near-match scores coinciding: decorrelation, not approximation. Auditing our own metrics surfaced two artifacts in opposite directions: we correct a refusal-scoring bug and report the fixed scores beside the originals, and flag an inflated metadata metric as an upper bound. KhatianDoc documents not a performance gap but the absence of a capability, with verified ground truth for future systems. Code and data, with a redacted image release, are publicly available.

1 Introduction

KhatianDoc targets a legal-document gap created by handwritten Bengali land records and the unsupported Ana-Ganda base-16 system. Its benchmark and audit show widespread model failure rather than gradual partial competence.

  • Motivation: KhatianDoc addresses RS Khatians, authoritative Bengali land records whose handwritten Ana-Ganda glyphs lack standard font, OCR, and tokenizer support.The records carry direct legal force in property disputes and title information for millions of parcels.
  • Benchmark: The benchmark uses 107 real RS Khatian records and four tasks spanning symbol recognition, base-16 arithmetic, field extraction, and legal document QA.The QA task contains 1,634 question–answer pairs.
  • Diagnosis: 39.3% of the evaluation set yields zero correct answers from every tested model, while every numeric arithmetic output exceeds the constant dataset-mean baseline’s error.These results indicate a hard capability floor under the reported protocol.
  • Contributions: KhatianDoc is presented as the first benchmark for Bengali RS Khatian records and the first machine-checkable, legally verified Ana-Ganda encoding.The benchmark contribution is paired with a diagnosis of multimodal-model failure.
  • Contributions: The evaluation diagnoses universal failure on complex legal reasoning and sub-baseline performance on non-decimal arithmetic across six frontier multimodal LLMs.The paper frames these findings as a capability diagnosis rather than a smooth performance gap.
  • Contributions: The methodology reports two corrected evaluation artifacts, preserves multi-hop referential distinctions through positional tokens, and releases redacted images.The artifacts include a refusal-scoring correction and an inflated metric flag.

2 Related Work

Prior legal, document-understanding, multilingual, and reasoning benchmarks largely use typed Latin-script or base-10 text. KhatianDoc instead targets handwritten legal records whose binding numerals are invisible to those systems and preserves distinctions needed for multi-hop QA.

  • Document IE and legal DocVQA: Existing document-understanding and legal benchmarks overwhelmingly assume typed Latin-script forms or clean digital prose.The cited corpora do not address handwritten Bengali legal tables with unsupported numerals.
  • Multilingual and low-resource legal NLP: Multilingual legal resources generally target typed modern prose in base-10, whereas RS Khatians are handwritten, abbreviated, tabular, and use an absent numeral system.The gap is characterized as script- and numeral-system-level invisibility.
  • Quantitative and symbolic reasoning: Common arithmetic and reasoning benchmarks use base-10, while KhatianDoc instantiates a positional base-16 fraction with four nested subunits.The paper links the benchmark’s sub-baseline arithmetic results to structural novelty rather than mere difficulty.
  • Privacy-preserving legal AI: Generic deidentification can collapse distinct parties into one token, making multi-hop share comparisons unanswerable by construction.KhatianDoc uses positional tokens to retain the referential distinctions required by those questions.

3 The KhatianDoc Benchmark

KhatianDoc is a human-built, legally verified benchmark of real handwritten RS Khatians, organized into four tasks and anonymized without erasing referential distinctions. Its hardest material is the unsupported Ana-Ganda base-16 fraction system.

  • 3.1 Source documents: The corpus contains 107 full-page handwritten RS Khatian scans from one Vumi office and one Mouza, preserving the office-issued document boundaries.The records cover ownership, plot boundaries, and fractional inheritance shares.
  • 3.2 Ground-truth construction: Every field was transcribed manually without OCR and independently checked by a land-law lawyer, reaching complete agreement on symbol identity and decimal value.The verified transcription is the benchmark’s ground truth.
  • 3.3 Privacy-preserving anonymization: Personal fields are replaced with consistent positional tokens across structured records, questions, answers, and evidence spans so multi-hop questions can distinguish parties.The design avoids the referential collapse caused by a single generic name token.
  • 3.4 The Ana-Ganda-Kora-Kranti-Til system: Ana-Ganda-Kora-Kranti-Til is a base-16 positional fraction system in which 16 Ana make one plot and four subunits refine shares below one Ana.Its dedicated glyphs lack standard font and OCR support, motivating machine-checkable encoding.
  • 3.5 Task definitions and statistics: The four tasks progress from isolated glyph recognition and fraction-to-decimal conversion to field extraction and document QA.Task 1 contains 95 symbol crops; Task 2 reuses the same strings as text inputs; Task 3 extracts metadata and 261 row records.
  • 3.5 Task definitions and statistics: Task 4 contains 1,634 question–answer pairs, including 428 complex questions requiring multi-step reasoning, with results standardized on a stratified 300-question subset.The subset is 73.7% simple and 26.3% complex.

4 Experimental Setup

The study evaluates six open and closed multimodal LLMs under one fixed, deterministic zero-shot protocol. Task-specific output caps and metrics are predefined across recognition, arithmetic, extraction, and QA.

  • Models and protocol: Six multimodal LLMs, spanning closed APIs and open weights from 8B to over 70B parameters, are evaluated through one common access path.Identical retry and rate-limit logic reduces provider-specific client code as a confound.
  • Models and protocol: All tasks use fixed zero-shot deterministic instructions without in-context examples or chain-of-thought elicitation.Blocked or dropped responses are logged as empty-string outcomes rather than discarded.
  • Metrics: Scoring uses CER and EM for symbol recognition, exact decimal match and MAE for arithmetic, metadata EM and row-level F1 for extraction, and EM with ANLS for QA.Generation limits are set separately for each task to match expected output length.

5 Results

Across four tasks, the six multimodal models show uniformly weak performance, with Task 4 revealing a zero-success floor on complex reasoning categories and Task 2 falling below a constant-mean arithmetic baseline.

  • Overall results: No model clears 26% on any primary metric except Task 3 metadata EM, whose score is inflated by constant and null fields rather than extraction.The reported metadata metric is therefore not directly comparable to the row-level extraction results.
  • Task 4 legal QA: 39.3% of the stratified Task 4 evaluation set consists of five categories that return zero correct answers from every model.Four categories have no successes across 118 questions; total_area is included in the 39.3% group despite a few correct answers from two models.
  • Task 2 arithmetic: Task 2 models with numeric outputs achieve MAE between 0.40 and 0.42, worse than the constant-mean baseline MAE of 0.237.The baseline always predicts the dataset mean, 0.3935, while the models receive the conversion rule in the prompt.
  • Task 2 arithmetic: Exact Decimal Match and Near Match within 0.005 coincide for every model, indicating no close-but-inexact arithmetic predictions.The paper interprets this pattern as outputs being decorrelated from the input string rather than approximate conversions.
  • Task 1 and Task 3: Task 1 character error rates remain uniformly high from 76.89 to 96.56 across models ranging from 8B open systems to 72B+ closed systems.The narrow, scale-independent band indicates no visible improvement with model scale or access conditions.

6 Failure Analysis and Discussion

Failure analysis finds systematic, model-family-specific substitutions and script errors, while a scoring audit identifies one bug that penalized correct refusals and another metric that rewarded non-reading shortcuts.

  • 6.1 Symbol collapse and numeral-script substitution: Task 1 models collapse toward characteristic substitutes rather than random glyph noise, including the Bengali Rupee Mark, digit-slash notation, and Brahmi characters.Gemini 2.5 Flash Lite uses U+09F2 on 11 of 95 items, while other families exhibit distinct substitution patterns.
  • 6.1 Symbol collapse and numeral-script substitution: Figure 2 separates simple lookups from complex reasoning categories, showing zero exact-match cells for every complex category and fraction share across all six models.Only scattered non-zero cells appear among the simple lookup categories.
  • 6.1 Symbol collapse and numeral-script substitution: GPT-4o Mini emits Western Arabic digits on 97 of 107 Task 3 documents despite Bengali-script prompts and references.Because scoring uses exact-string match, a numerically correct Western-digit output is still marked wrong against a Bengali-script reference.
  • 6.2 Two places our own metrics needed correcting: The refusal-normalized rescoring raises Task 4 EM from 13.33% to 21.67% for Gemini 2.5 Flash Lite and from 11.33% to 18.33% for Llama 4 Scout.The correction maps canonical refusal sentences and ground-truth shorthand for absent information into one scored class.
  • 6.2 Two places our own metrics needed correcting: Task 3 metadata EM is treated as an upper bound because constant district and upazila fields and frequent null total_area values can earn matches without image extraction.Row-level F1 becomes the metric of record because it credits only fields recovered from the image.
  • Discussion: The audited failures remain systematic and model-family-specific rather than random, with distinct dominant patterns across Qwen, Gemma, Gemini, Llama, and GPT-4o Mini.The audit reports that the core Task 4 reasoning categories and row-level extraction numbers are unaffected by the two artifacts.

7 Conclusion

KhatianDoc contributes a legally verified benchmark resource and a diagnosis of systematic multimodal-LLM failure on Bengali land records. Its results show universal zero-shot failures on substantial legal QA and arithmetic below a context-free baseline, while openly auditing misleading metrics.

  • KhatianDoc provides 107 real Khatian records with hand-verified ground truth and positional-token anonymization that preserves multi-hop referential structure.
  • 39.3% of the stratified evaluation set returns zero correct answers from every model, while every model emitting an arithmetic number underperforms a context-free baseline.
  • The authors identify and disclose two evaluation artifacts: one inflating metadata scores through constant and null fields, and another penalizing correct refusals.

Limitations

The benchmark’s scope is constrained by geographically narrow data, limited symbol-string coverage, zero-shot-only evaluation, automatic legal-QA scoring, and image-only inputs. These boundaries limit how broadly its results and improvements should be generalized.

  • All 107 documents come from one Mouza and district, so performance gains may not transfer to more varied Khatian layouts without further evaluation.
  • Task 1 contains 95 crops and 52 unique strings, likely underrepresenting rarer compound fractions.
  • The study evaluates zero-shot performance only, leaving the effects of fine-tuning, retrieval augmentation, and in-context examples unmeasured.
  • Automatic exact or near-exact scoring can disagree with legal correctness, and human legal-expert evaluation of complex QA remains future work.
  • Image-only evaluation leaves OCR-augmented performance untested, although recognized text could substantially shift Task 3 and Task 4 failures.

Ethical Considerations

The paper balances research access with privacy and deployment safeguards for legally consequential land records. It uses anonymized text, redacted public images, controlled access to originals, and an explicit warning against unaudited deployment.

  • The source records are official government survey documents containing personally identifying land-ownership information, motivating privacy protections.
  • Released ground-truth files replace owner and residence identifiers with positional tokens before annotation, training, or evaluation, including Task 4 questions and evidence spans.
  • Original unredacted scans are used for reported results under access control, while vetted researchers may access them through the Vumi-office channel.
  • Public redacted images mask names, addresses, administrative headers, form identifiers, and signatures, excluding masked fields from image-based scoring.
  • The public and secure protocols measure the same four tasks but differ by omitted masked metadata, and the paper’s reported numbers come from secure originals.
  • The benchmark warns that deploying models shown to fail on fraction math and multi-hop reasoning could produce incorrect ownership outputs with real legal consequences.
  • KhatianDoc is intended to support research toward privacy-respecting tools, not replace lawyers, surveyors, or citizens doing this work.

C Dataset Statistics and Task Examples

The dataset contains 107 annotated Khatian records, 95 symbol crops, 261 rows, 573 owner entries, and ten legal-QA categories. Its task examples range from structured JSON extraction to multi-hop fraction arithmetic and counterfactual reasoning.

  • Task examples: Task 4 spans ten categories, including six simple and four complex question types.
  • Task examples: Task 3 requires exact JSON fields for Khatian metadata, plot information, land class, shares, areas, and owner lists.
  • Task examples: Legal fraction-math questions require filtering rows, extracting Ana-Ganda strings, converting and summing values, and multiplying by total area.
  • Task examples: Counterfactual questions require multi-hop extraction, base-16 division, and base-16 addition after a share transfer.
  • Task examples: Table 10 illustrates family-specific out-of-vocabulary collapse in raw Task 1 outputs.

E Numeral-Script Substitution

Task 3 reveals a script mismatch in model outputs, while metric audits identify metadata and refusal-scoring artifacts that affect evaluation. The benchmark therefore distinguishes apparent extraction performance from image-grounded recovery and corrected scoring.

  • Numeral-script substitution: GPT-4o Mini outputs Western digits almost exclusively despite Bengali-script prompts and documents.Table 11 reports predicted-digit proportions in Bengali versus Western Arabic script across six Task 3 models.
  • Metric artifacts: Metadata EM can approach 50% without visual extraction because district and upazila are corpus constants and total_area is often null.The paper treats metadata EM as an upper bound and uses row-level F1 for image-grounded extraction claims.
  • Metric artifacts: A refusal-string mismatch scored compliant answers as incorrect when prompts and references used different strings for absent information.Normalizing both strings to one semantic class corrects the penalty for 14 genuinely absent-field answers.
  • Task interfaces: Task 2 requires converting Ana-Ganda fractions under rules mapping 16 Ana to 1.0 and successive units through Ganda, Kora, Kranti, and Til.Task 4 separately requires concise Bengali answers and an exact absence string when information is missing.
Loading 2609.03597v1…