PaperlayerView exact revision on arXiv

Source-linked AI summary

Impact of Iterative Fine-Tuning on Transcription Accuracy in Complex Historical Sanskrit Manuscripts

Kartik Chincholikar, Kaushik Gopalan, Mihir Hasabnis

arXiv:2608.18696v1cs.CVcs.AI

TL;DR

Historical Sanskrit manuscripts are difficult to digitize, while annotation requires specialized expertise and is expensive and time-consuming. The paper introduces an iteratively fine-tunable traditional OCR pipeline and reports substantial accuracy gains over successive pages, outperforming Gemini 3.5 Flash after adaptation.

  • Problem

    Historical manuscript digitization requires specialized language and script expertise, making annotation expensive and time-consuming.

  • Method

    The paper uses a two-step OCR pipeline for layout analysis and Unicode transcription, iteratively fine-tuned at layout and appearance levels, alongside a richly annotated PAGE-XML dataset.

  • Results

    Up to 34.1% lower TextEdit and 37.5% lower Page-CER enabled the adapted pipeline to outperform Gemini 3.5 Flash on all three manuscripts and both metrics.

  • Takeaways & Limitations

    Externally verifiable text-line locations and Unicode outputs support using the dataset for post-training and fine-tuning modern multimodal large language models.

  • Takeaways & Limitations

    The pipeline is brittle because of its stepwise design, heuristic text-line processing, and dependence on the frozen CRAFT character detector.

Abstract

from arXiv · show

Digitizing the text from handwritten historical manuscripts is required to make them easily accessible, preservable, and to enable historical scholars to study them in new ways. Historical manuscripts, however, often exhibit complex heterogeneous layouts and non-standard appearance due to period-specific writing styles, page textures, camera noise, and other nuisance factors, making them difficult to perform OCR on. To tackle this challenge, we introduce a local traditional OCR pipeline, which can be iteratively fine-tuned on the target manuscript at the layout-level and the appearance-level. By adapting to the target manuscript distribution, the proposed Traditional OCR pipeline makes better predictions on subsequent pages, causing iterative reduction in human annotation effort, which is expensive and time-consuming as it requires historical domain expertise. Using this pipeline, we digitize text from three complex historical Sanskrit manuscripts and introduce a dataset with granular layout-level annotations, along with Unicode annotations in the standard PAGE-XML format. We demonstrate quantitative gains due to iterative fine-tuning of the proposed traditional OCR pipeline, and also benchmark the performance of leading Multi-Modal Large Language Models on the introduced Dataset. Code and dataset are available at: https://github.com/flame-cai/gnn-synthetic-layout-historical/.

1 Introduction

Historical manuscript digitization improves access and scholarly analysis but is costly because it requires expert annotation and must handle substantial layout- and appearance-level variation. This work addresses these challenges with an iteratively fine-tunable traditional OCR pipeline and a richly annotated Sanskrit manuscript dataset.

  • Motivation: Unicode digitization protects fragile manuscripts while enabling faster search and analysis of word usage and idea frequency.The digital format avoids risks to original manuscripts and supports historical research.
  • Challenges: Specialized historical language and script expertise makes manuscript digitization expensive and time-consuming, contributing to scarce annotated data.The supplied passage identifies expert collection and annotation demands as a source of data scarcity.
  • Challenges: Historical pages exhibit layout-level shifts from heterogeneous structures and appearance-level shifts from scribal, period-specific, and physical variations.Examples include marginalia, interlinear glosses, footnotes, irregular curved text-lines, and variation in writing conventions and degradation.
  • Method: The pipeline digitizes three complex Sanskrit manuscripts through layout analysis followed by transcription of segmented text-line images into Unicode.Its two-step process first segments individual text-line images and then transcribes them into machine-readable text.
  • Contributions: The open-source Traditional OCR pipeline supports iterative human-supervised fine-tuning at layout and appearance levels, progressively reducing annotation burden on subsequent pages.The proposed adaptation targets both manuscript layout and visual appearance.
  • Contributions: The introduced dataset provides high-quality layout- and appearance-level annotations in standard PAGE-XML format.The dataset is designed for richly annotated historical manuscript digitization.

2 Literature Review

The literature contrasts inconsistent multimodal LLM OCR on complex historical manuscripts with traditional pipelines that separately analyze layout, segment text lines, and recognize Unicode text. Because segmentation errors propagate to recognition and historical layouts challenge existing methods, manual supervision may remain necessary.

  • Multi-Modal LLMs: Modern Multi-Modal LLM OCR remains inconsistent on historical documents with complex, dense layouts and non-standard appearance because their distributions differ from pre-training data.Existing benchmark datasets mostly cover printed or standard digital documents.
  • Traditional OCR Pipelines: Traditional OCR pipelines address scarce data through text-line segmentation followed by recognition of segmented images as machine-readable Unicode text.The two stages are layout analysis and text recognition OCR.
  • Layout Analysis: Projection-profile segmentation works when manuscript layouts are known a priori, whereas newer methods predict bounding polygons or pixel-level masks end to end.Dense pixel-level prediction methods are discussed as a more recent alternative to early projection-profile approaches.
  • Layout Analysis: LineTR and CurT instead predict geometric parameters for piecewise line segments or cubic Bézier curves, using inductive geometric priors to address layout variation.These methods extract information from manuscript images before predicting text-line geometry.
  • Text Recognition: Text recognition methods progress from CNN features with BiLSTM or RNN sequence modeling, CTC loss, and decoding toward Transformer-based visual and language modeling.Layout analysis and text-line segmentation remain prerequisites, and their errors can impair downstream recognition, sometimes requiring manual supervision.

3 Dataset

The dataset contains three historical Sanskrit manuscripts with distinct page layouts and provides granular layout- and appearance-level annotations in standard PAGE-XML. Its annotation protocol specifies text-region and reading-order assumptions, while the pipeline targets manuscripts meeting character-segmentation and script-recognition criteria.

  • Dataset composition: The dataset comprises three historical Sanskrit manuscripts, categorized as Moderate, Dense, and Circular Layout Manuscripts according to their page layouts.The manuscripts are Yajnavalakyasmritih (Acharadhyayah), Muhurta Martanda, and Tantra Raj With Yantra And Mantra Uddhara.
  • Annotation Methodology: Annotations operate at the Layout-level and Appearance-level, representing characters or grapheme clusters as nodes connected to neighboring characters in the same text-line.Nodes belonging to the same text-line share a label, and text-regions are similarly labeled.
  • Annotation assumptions: The protocol annotates text-regions only when their internal text-line reading order is unambiguous, while marginalia and commentary may have ambiguous text-region order.Reference symbols and numbers linking main text to commentary are excluded from annotation.
  • Selection Criteria: The Traditional Pipeline targets manuscripts for which CRAFT character segmentation performs satisfactorily and a pre-trained OCR model exists for the manuscript’s script.The authors expect these criteria to apply to most Sanskrit manuscript images archived in culture-preservation projects such as eGangotri and Gyan…
  • Dataset format: The dataset includes rich granular annotations at both levels and is exported in the standard PAGE-XML format.The annotations cover Layout-level and Appearance-level information.

4 Method

The method is a local traditional OCR pipeline that adapts separately to manuscript layout and appearance through iterative human correction and fine-tuning. It detects character locations, analyzes text-line structure, rectifies curved text, and recognizes Unicode text while progressively reducing annotation demands.

  • Pipeline overview: The pipeline fine-tunes at both the layout level and appearance level to handle variations in text placement and visual style.Layout variation includes dense pages, marginalia, interlinear text, and curved or circular lines; appearance variation includes handwriting conventions, paper texture, and uneven scans.
  • Layout annotation: A frozen CRAFT U-Net detects character or grapheme-cluster locations, which a GNN-based layout backbone uses to segment text lines.The GNN represents manuscript characters or grapheme clusters as nodes and predicts their text-line connectivity.
  • Iterative fine-tuning: Human-corrected layout graphs fine-tune the target-manuscript GNN, while corrected text-line transcriptions fine-tune the CNN–BiLSTM–CTC recognizer for appearance-level shifts.The GNN learns from corrected graphs, whereas the recognizer uses text-line image and corrected Unicode-text pairs.
  • Text-line preparation and recognition: Curved text is unwrapped into rectangular text-line images before the CNN–BiLSTM–CTC predicts Unicode text.Local tangent and normal directions define the crop’s horizontal and vertical axes, respectively.
  • Annotation cost: The OCR recognizer’s annotation cost subsumes the GNN’s because recognition requires both corrected layout crops and their Unicode transcriptions.The GNN requires only a corrected graph, while OCR training requires text-line images cut from that layout and corresponding corrected text.

5 Experiment Setup

The experiments use five-fold manuscript-specific evaluation with disjoint fine-tuning and held-out test pages, measuring transcription accuracy through TextEdit and Page-CER while benchmarking four off-the-shelf multimodal OCR systems.

  • Data Splits: Each manuscript is evaluated in five fixed-seed folds, reserving three pages for fine-tuning and all remaining pages for held-out testing.Fine-tuning gains are quantified using 1, 2, and 3 pages across the five folds, with fine-tuning and test sets disjoint within each fold.
  • Multi-modal large language model pipeline: Four off-the-shelf multimodal OCR systems are benchmarked: Gemini-3.5-Flash, OpenAI GPT-5.6-Terra, Claude-Sonnet-5, and Sarvam Vision Document Digitization.Gemini, OpenAI, and Claude receive the same resized manuscript-page image and end-to-end prompt requesting diplomatic Unicode Devanagari transcription and normalized text-line polygons.
  • Metrics: Transcription accuracy is measured with TextEdit and Page-CER, using the exact OmniDocBench simple_match score for TextEdit.TextEdit ignores line order and geometry, pairing each ground-truth line with at most one predicted line while counting unmatched lines as missing or extra text.
  • Metrics: For TextEdit, each non-empty PAGE-XML TextLine is parsed as an atomic item using its direct TextEquiv/Unicode transcription, while TextRegion membership is ignored.The matcher receives each text line as an individual text block.
  • Metrics: Page-CER sorts ground-truth and predicted lines top-to-bottom and left-to-right, joins their strings, and computes Character Edit Distance.This procedure calculates character error rate after imposing the reading order defined by the sorted lines.

6 Results

Results show that Gemini 3.5 Flash is the strongest off-the-shelf multimodal model, while the traditional pipeline is competitive without adaptation and improves through iterative fine-tuning. Manual layout correction and appearance-level OCR adaptation further improve recognition under manuscript-specific heterogeneity.

  • Off-the-Shelf Multi-modal LLMs: Gemini 3.5 Flash performs best across all three manuscripts and both TextEdit and Page-CER metrics, with Sarvam Vision consistently second.All multimodal models rank manuscript difficulty identically: Moderate < Dense < Circular.
  • Off-the-Shelf Traditional Pipeline: The off-the-shelf traditional pipeline performs comparably with Gemini 3.5 Flash, outperforming it on Page-CER for Moderate and Dense Layout Manuscripts but underperforming on Circular Layout.These comparisons use the 0-page fine-tuned rows for the traditional pipeline and Gemini 3.5 Flash’s rows, with lower metric values better.
  • Iteratively Fine-tuned Traditional Pipeline (Fully Automatic): Iterative fine-tuning of the traditional pipeline’s GNN and CNN–BiLSTM–CTC adapts it to target-manuscript heterogeneity and improves subsequent OCR performance.Table 3 evaluates fully automatic inference and inference preceded by manual correction of predicted layouts using TextEdit and Page-CER.
  • Iteratively Fine-tuned Traditional Pipeline (with Manual Layout Correction): Manual layout correction at inference time prevents costly layout-analysis mistakes that can cause disastrous downstream line-level recognition errors.The experiment applies manual correction to predicted layouts on the test set before downstream text recognition OCR.
  • Fine-tuning Ablations: For the Moderate Layout manuscript, OCR-only fine-tuning helps more than GNN-only fine-tuning because the pretrained GNN is already near ground truth, while recognition rapidly adapts at the appearance level.The ablations compare GNN-only, OCR-only, and combined fine-tuning.

7 Discussion

The traditional pipeline matches Gemini 3.5 Flash without adaptation but surpasses it after iterative fine-tuning, while manual layout correction removes object-level errors. Its task-specific local adaptability suits scarce, costly annotation settings, though its heuristic, stepwise design remains brittle.

  • Without target-manuscript adaptation, the traditional pipeline achieves comparable accuracy to Gemini 3.5 Flash.
  • Fine-tuning on three corrected pages reduces TextEdit by up to 34.1% and Page-CER by up to 37.5%, outperforming Gemini 3.5 Flash on all three manuscripts.The improvements apply to both metrics.
  • Fine-tuning each new page reduces the human annotation effort required for the next page, especially when data is scarce and annotation is costly.
  • Manual layout correction at inference time removes object-level errors such as incorrectly merged or split text-lines and incorrectly predicted missing or extra nodes.Remaining errors arise from textline unwrapping and processing and the fine-tuned CNN–BiLSTM–CTC text-line recognition model.
  • The specialized Traditional pipeline is well suited to scarce Sanskrit-manuscript data and expensive annotation, but its stepwise heuristics and dependence on frozen CRAFT make it brittle.
  • The pipeline bootstraps a richly annotated dataset with layout-level annotations and Unicode transcriptions in graph-based and standard PAGE-XML formats.Its final text-line locations and Unicode text content can be externally verified for post-training and fine-tuning modern multimodal models.

8 Supplementary Material

The supplementary material specifies a controlled comparison of three VLMs using identical manuscript images and prompts, while documenting the prompt format, model endpoints, and Sarvam’s distinct API workflow. The transcription task requires diplomatic Unicode Devanagari text and visual text-line geometry in normalized coordinates.

  • VLM evaluation: Gemini-3.5-Flash, OpenAI GPT-5.6-Terra, and Claude-Sonnet-5 received the same manuscript-page image and end-to-end prompt for Unicode Devanagari transcription and text-line polygons.The geometry uses normalized 0–1000 coordinates and follows conventions established in Pix2Seq and PaLI.
  • Prompt specification: The prompt required raw valid JSON, normalized coordinates, and a tight polygon or acceptable box for every visual text-line.Coordinates run from [0,0] at the top-left to [1000,1000] at the bottom-right, with polygon points represented as [y,x].
  • VLM baselines: The four end-to-end VLM baselines used documented model identifiers and HTTP endpoints, with Gemini, OpenAI, and Claude returning JSON and Sarvam returning HTML.Gemini, OpenAI, and Claude received the resized page image followed by the same VLM_END_TO_END_PROMPT, whereas Sarvam received the image alone.
  • Sarvam API: Sarvam Vision requests carried language=sa-IN and output_format=html without a model parameter, while sarvam-vision was retained only as a provider-specification and cache-fingerprint identifier.The identifier was not sent over the wire.
  • Sarvam API: Sarvam Vision processed one page per asynchronous job, costing five calls in sequence: create, upload, start, status polling, and download.Image and result ZIP transfers used presigned object-storage URLs rather than api.sarvam.ai.
Loading 2608.18696v1…