PaperlayerView exact revision on arXiv

Source-linked AI summary

Devanagari Handwritten Character Recognition Using TrOCR: A Transformer-Based Model with Real-Time Web Deployment

Amrit Baskota, Samyam Budhathoki, Shubham Ghimire, Abiskar Ghimire, Sarwesh Phuyal, Baskaran P

arXiv:2609.31681v1cs.CV

TL;DR

Complex Devanagari handwriting makes accurate character recognition challenging. This paper fine-tunes pretrained TrOCR for handwritten characters, reporting 3.95% CER and 96.05% character-level accuracy, with web deployment latency below five seconds per request.

  • Problem

    Complex Devanagari symbols, character combinations, marks, and positional changes make handwritten character recognition challenging.

    Source: §I · p. 1

  • Method

    The paper fine-tunes a pretrained TrOCR encoder-decoder model to recognize handwritten Devanagari images.

    Source: §III · p. 3, B. Model Training and Fine-Tuning… · p. 4

  • Results

    CER of 3.95% and character-level accuracy of 96.05% were reported; the deployed application had latency <5s per request.

    Source: A. Model Performance and Accuracy · p. 5, §VII · p. 6

  • Takeaways & Limitations

    The study presents a foundation for scalable digitization of handwritten Devanagari characters, while word- and sentence-level recognition remains future work.

    Source: §VII · p. 6, §VI · p. 6

    The dataset contains isolated characters, limiting the model’s ability to recognize words and sentences.

    Source: §VI · p. 6

Abstract

from arXiv · show

Devnagari is a one of the ancient language of the Indian subcontinent consisting of 36 vowels, 14 consonants and 10 numerals. The accurate recognition of handwritten Devnagari characters is challenging due to high complexity of Devnagari scripts. This paper presents a method to fine tune the pre trained TrOCR model to accurately recognize Devanagari handwritten characters. The Methodology consists of a preprocessing mechanism where input images are standardized into RGB format, tokenized in batches and integrated with Hugging Face Dataset. The pre-trained microsoft/trocr-base-handwritten model is fine-tuned over a dataset of nearly 5000 character images that are uniformly partitioned in the ratio 8:1:1 for training, evaluation and testing. Further optimization is done is the training process through mixed precision training, gradient checkpointing, and early stopping mechanism. The model achieves a character error rate (CER) of 3.95% and a character-level accuracy of 96.05%, outperforming the previous CNN based models. A scalable web application is developed using Vite.js, Golang, and FastAPI which practically deploys the OCR model and serves character recognition task with a latency less than 5 seconds per request. This study demonstrates the use of TrOCR model to build a scalable handwritten Devnagari character recognition system and also a foundation to future research on Devnagari Script Recognition using Transformers.

I. INTRODUCTION

Complex Devanagari character combinations, marks, and positional changes make OCR challenging; this paper proposes evaluating TrOCR against traditional CNN approaches. Related work spans segmentation-based methods, neural recognizers, and transformer-based recognition.

I. INTRODUCTION

  • The paper proposes comparing TrOCR with traditional CNN approaches for handwritten Devanagari recognition.The comparison is framed around character-recognition efficiency, processing speed, and error rate.Source: §I · p. 1

II. RELATED WORK

  • Devanagari OCR must handle joined or compound characters and matras, while traditional SVM and HMM methods rely on explicit character segmentation.The introduction also identifies handwriting-style variation as a recognition challenge.Source: §II · pp. 1–2
  • A prior neural-network system for Devanagari characters achieved 75.6% accuracy on noisy characters.Source: §II · p. 2
  • A transformer-based handwritten-string method reported CER values of 0.241, 0.164, and 0.127 after adding a decoder and training data.Source: §II · p. 2
  • Prior work reported CER of 0.024 for TrOCR in historical-document recognition [12], while a template-matching character system reported 92.66% accuracy [14].Source: §II · p. 2

III. PROPOSED MODEL

The proposed system applies transformer-based TrOCR to handwritten Devanagari images, in a setting where complex characters challenge recognition and segmentation-based traditional methods. Its encoder-decoder architecture converts image patches into character outputs.

III. PROPOSED MODEL

  • The proposal targets Devanagari recognition challenges including merged and compound characters and matras, which can complicate explicit segmentation.Source: §III · p. 2
  • The image Transformer encoder divides each 384×384 input into 16×16 patches and produces 768-dimensional features through 12 transformer blocks.Learnable spatial information is added to patch tokens, and a cross-attention bridge connects visual and textual modalities.Source: §III · p. 3
  • The text decoder uses token and position embeddings, masked attention, and cross-attention to generate Devanagari characters successively.Source: §III · p. 3

A. Dataset Preprocessing

The dataset pipeline standardizes character images and labels for TrOCR, then fine-tunes a pretrained encoder-decoder model with training safeguards and efficiency optimizations. The reported training and evaluation losses decrease over training.

A. Dataset Preprocessing

B. Model Training and Fine-Tuning Hyperparameters

C. Evaluation Metrics and Performance Analysis

Model performance is evaluated by comparing predictions with actual labels using Word Error Rate (WER) and Character Error Rate (CER). The evaluation framework also uses error-rate and accuracy metrics to assess validation predictions.

C. Evaluation Metrics and Performance Analysis

D. Web Based Integration

The web application lets users submit Devanagari handwriting by uploading an image or drawing in a canvas. A Golang backend forwards resized images to a FastAPI-hosted TrOCR model and displays its JSON recognition output.

D. Web Based Integration

  • Users can upload a Devanagari image or write characters on an interactive canvas.The canvas simulates natural handwriting input.Source: D. Web Based Integration · p. 5
  • The Golang backend receives uploaded files over HTTP, resizes them to 384x384 pixels, and sends them to the FastAPI server hosting TrOCR.Source: D. Web Based Integration · p. 5
  • TrOCR returns the identified Devanagari script in JSON, which the web interface displays to the user.Source: D. Web Based Integration · p. 5

A. Model Performance and Accuracy

The model reports low character error and high character-level accuracy on the Devanagari test set, alongside observations about visually similar characters. Training and evaluation losses decline during training, while mixed precision and gradient checkpointing reduce GPU use.

A. Model Performance and Accuracy

  • 3.95% CER and 96.05% character-level accuracy were achieved on a test set of 467 character images.The test set comprised 10% of the dataset; the paper says the metrics validate robustness to handwritten variations.Source: A. Model Performance and Accuracy · p. 5, Table I · p. 5
  • TrOCR distinguished visually similar characters, including ड and ङ, where CNN models may struggle with untidy or closely written characters.The paper attributes this to TrOCR examining overall image structure and context.Source: A. Model Performance and Accuracy · p. 5

B. Training Efficiency and Convergence

  • Mixed-precision training and gradient checkpointing reduced GPU usage by nearly 30%, allowing longer training without hardware constraints.The preprocessing pipeline also used batch-wise RGB conversion and normalization for consistent inputs.Source: B. Training Efficiency and Convergence · p. 5
  • Evaluation loss fell from 0.9974 at 1% training completion to 0.031 after six epochs, while training loss fell from 1.3843 to 0.0009.The reported loss trajectories are described as stable convergence; Fig. 6 is identified as training versus evaluation losses.Source: B. Training Efficiency and Convergence · p. 5, Figure 6 · p. 5

C. Comparative Analysis with Existing Models

TrOCR is reported to outperform conventional approaches on handwritten Devanagari recognition, and the deployed application provides real-time character inference. The deployment standardizes input images and reports under-five-second request latency.

C. Comparative Analysis with Existing Models

  • TrOCR's self-attention is reported to handle positional differences and matras better than conventional approaches, improving accuracy and CER.Table II is identified as a metrics comparison, but the supplied passage does not provide its cell values.Source: Table II · p. 5, C. Comparative Analysis with Existing… · p. 5

D. Model Deployment and Performance

  • The deployed application standardizes input images to 384x384 pixels and reports end-to-end latency below five seconds per request.The passage says this standardization preserves critical image features while minimizing computational overhead.Source: D. Model Deployment and Performance · p. 6
  • The Golang-FastAPI backend is reported to serve numerous concurrent user requests with quick responses and minimal latency.Source: D. Model Deployment and Performance · p. 6

VI. FUTURE WORK

The study reports high character-recognition accuracy and a web deployment, while identifying isolated-character data as a limit on word- and sentence-level recognition. It proposes expanding the dataset to handwritten words and sentences.

VI. FUTURE WORK

  • The isolated-character dataset limits recognition of words and sentences, motivating expansion to handwritten word- and sentence-level images.The proposed sources include ancient Sanskrit manuscripts and modern writing from varied ages and regions.Source: §VI · p. 6

VII. CONCLUSION

  • 3.95% CER and 96.05% character accuracy are reported for the fine-tuned TrOCR model on a dataset of Devanagari numerals, vowels, and consonants.Source: §VII · p. 6
  • The OCR model is integrated into a Golang, FastAPI, and Vite.js web application with latency below five seconds per request.The conclusion describes the framework as a foundation for digitizing handwritten Devanagari across regions, age groups, and timelines.Source: §VII · p. 6
Loading 2609.31681v1…