Padārtha: Ontology-Grounded Fine-Grained NER Benchmark for Classical Sanskrit

Sujoy Sarkar; Pretam Ray; Paramhans Shah; Manoj Balaji Jagadeeshan; Akash Gairola; Arjuna S R; Pawan Goyal

Source-linked AI summary

Padārtha: Ontology-Grounded Fine-Grained NER Benchmark for Classical Sanskrit

Sujoy Sarkar, Pretam Ray, Paramhans Shah, Manoj Balaji Jagadeeshan, Akash Gairola, Arjuna S R, Pawan Goyal

arXiv:2608.29324v1Aug 29, 2026cs.CL

TL;DR

Literary Sanskrit NER lacks schemas and annotated data suited to entities expressed through culturally specific naming conventions. Padārtha grounds fine-grained annotation in Nyāya–Vaiśeṣika and benchmarks generative and task-specific systems, finding comparable coarse-level performance but persistent difficulty at fine granularity.

Problem
Literary and classical Sanskrit NER is constrained by imported modern-news tagsets, scarce expert-annotated data, and context-dependent naming conventions.
Method
Padārtha grounds an 18-category, 10-node ontology in Nyāya–Vaiśeṣika, maps it to five coarse tags, and annotates Sanskrit epic entity mentions using scholarly index links and expert correction.
Results
Task-specific Sanskrit systems are comparable with refined generative models, but neither performs well in fine-grained contexts; oracle ablations show granularity, not only detection, drives difficulty.
Takeaways & Limitations
An emic schema supplies category boundaries independent of the annotation task while retaining compatibility with existing tools through coarse-tag mappings.
Takeaways & Limitations
Experiments use machine-segmented text and a classical verse epic, so sandhi challenges are unaddressed and applicability to prose requires further investigation.

Abstract

from arXiv · show

Annotation schemas are not neutral. When applied to classical literature, tag sets developed for modern journalistic texts impose source-culture definitions on texts they were never designed to describe. We instead ground a schema in the tradition of the text itself introducing \textit{Padārtha}, the first ontology-grounded fine-grained Named Entity Recognition (NER) benchmark for Sanskrit, built on the \textit{Mahābhārata} epic. Our tag set derives from \textit{Nyāya-Vaiśesika}, a classical Indian ontological system, yielding 18 fine-grained categories organized under 10 ontological nodes and mapped onto five standard coarse tags, ensuring interoperability with existing benchmarks. Expert annotators label over 12.6K entries from a scholarly index of named entities, linked to corresponding mentions in the \textit{Mahānāma} corpus, producing fine-grained annotations for 108,335 entity mentions across 73,632 verses, along with a 5,000-verse expert-verified test set sampled to stress rare mentions. We present the first systematic benchmarking of generative NER against traditional architectures for Sanskrit, finding that fine-tuned generative models perform comparably to task-specific systems. However, all systems show a sharp decline from coarse to fine granularity and struggle with out-of-entity mentions unseen during training. The limitation is not due to data scarcity alone, as fine-tuned models recall unseen entities far worse than seen ones and tend to default to the majority sense under lexical ambiguity.

1 Introduction

Padārtha addresses the mismatch between imported modern NER schemas and classical Sanskrit literature by grounding fine-grained categories in Nyāya-Vaiśeṣika ontology. It introduces a large linked benchmark and evaluates generative and traditional systems under increasingly fine-grained and difficult entity conditions.

Research gap: Literary NER remains underdeveloped because standard tagsets lack domain-specific granularity, annotated historical data are scarce, and rare mentions are vulnerable to out-of-entity failures.Classical texts also exhibit lexical variation, context-dependent naming, and skew toward a few major entities.
Motivation: An emic schema grounds category boundaries in an intellectual tradition internal to the text rather than extending imported tagsets ad hoc.The paper argues that Nyāya-Vaiśeṣika provides independently articulated categorial boundaries for annotation.
Contribution: Padārtha is introduced as the first fine-grained NER benchmark in Sanskrit, based on the Mahābhārata.The epic combines diverse naming conventions with the computational challenges of morphologically rich Sanskrit.
Ontology-grounded tagset: The tagset contains 18 fine-grained leaves under 10 ontology nodes, mapped to five standard coarse tags for benchmark compatibility.The five coarse tags are PER, LOC, NORP, MISC, and TIME.
Dataset: 12.6K index entries yield annotations for 108,335 mentions across 73,632 verses, with a hand-corrected 5,000-verse test set.Labels are projected through links between the Mahānāma corpus and Sørensen’s Index.
Benchmarking and analysis: Benchmarking covers decoder-only LLMs, CRF, encoder, and encoder-decoder systems, while analyses examine granularity, memorization, ambiguity, information difficulty, and errors.Performance degrades as categorical granularity increases, and fine-tuned models recall unseen entities substantially less effectively while defaulting to majority senses under ambiguity.

2 Tagset Design and Dataset Construction

The tagset adapts Nyāya-Vaiśesika ontology to Sanskrit NER, retaining hierarchical fine-grained distinctions while mapping them to interoperable coarse tags. Padārtha projects expert labels from an index onto Mahānāma mentions and adds a manually corrected rare-mention test set.

Tagset Design: 15 of the 18 categories are inherited from the extended Vaiśesika dravya hierarchy, with additional categories covering groups, speech, and expanded places.Categories with insufficient corpus support were generally collapsed into parent nodes, except when sibling meanings remained distinct.
Tagset Design: 18 fine-grained categories are organized hierarchically under Nyāya-Vaiśesika-derived ontology nodes and mapped to five coarse NER tags.The hierarchy preserves shared ontological roots while allowing terminal categories to diverge.
Dataset Construction: Derived forms, relational identifiers, and epithets are treated as references to the entities they identify.This convention addresses classical Sanskrit’s frequent use of forms such as Kuntīputra and Vṛkodara instead of repeated proper names.
Dataset Statistics: The dataset exhibits a long-tailed mention distribution, with surface forms spanning a wider rank range than lemmatized forms because of morphological variation.The final dataset covers eighteen fine-grained categories and five coarse groupings.
Data Split: 5,000 verses were selected with rare-name sampling, and 73.14% of unique surface forms were unseen during training.The unseen forms accounted for 50.47% of all test mentions, while verse-level sampling allowed non-rare entities to appear as well.

3 Experimental Setup

The experimental setup compares non-neural, encoder, encoder-decoder, and decoder-only models across fine-tuning and in-context paradigms. Outputs are normalized to BIO tags and evaluated with strict span-and-label matching at coarse and fine granularity.

Models: Four architectural paradigms are evaluated: CRF, multilingual encoder, Sanskrit encoder-decoder, and decoder-only language models.The decoder-only models are tested under both fine-tuning and few-shot settings.
Models: The CRF baseline uses token identity, character prefixes and suffixes, and a ±1 token context window.It establishes a non-neural baseline based on shallow surface features.
Fine-tuning: MuRIL-large uses span-based BIO sequence labeling, while ByT5-Sanskrit preserves its original token_tag generation format with substituted NER labels.ByT5-Sanskrit was pretrained on Sanskrit segmentation and morphological tagging.
Decoder Models: Qwen2.5-7B-Instruct and Gemma-4-12B-it are fine-tuned with LoRA using the Inline Bracketed output format.Few-shot experiments use a two-stage mention-detection-then-classification cascade with 5-shot nearest-neighbor demonstrations.
Evaluation: All outputs are converted to unified BIO tags and scored with strict IOB2 evaluation requiring exact span boundaries and labels.Precision, Recall, Micro-F1, and Macro-F1 are reported at both coarse and fine granularity.

4 Results

Fine-tuned neural models outperform the CRF baseline, while few-shot decoder-only models underperform it; performance declines from coarse to fine granularity and unseen entities remain difficult.

Fine-tuned neural models substantially outperform the CRF baseline, while few-shot decoder-only models underperform it at both granularities.
MuRIL retains the highest Macro-F1, while Qwen-FT narrowly leads on Micro-F1 relative to MuRIL.
All models degrade from coarse to fine prediction, with average declines of −13.8pp in Macro F1 and −9.0pp in Micro F1.The larger Macro-F1 decline reflects low-support categories and increased disambiguation demands.
Gemini 3 Flash achieves the highest Micro F1 at both granularities, while MuRIL retains the best Macro F1.
All models recall seen entities better than unseen entities, while Gemini retains substantially higher unseen recall at 55.6–67.2%.CRF and fine-tuned models show the largest seen–unseen disparities, indicating heavy reliance on memorization.
Fine-tuned models fall into a majority-sense trap on minority-sense homonyms, with wrong-label rates of 27.4%–32.6%.The test set contains 1,227 minority-sense mentions.

5 Error Analysis

Error profiles differ by training paradigm, while fine-grained prediction increases wrong-type errors and boundary problems, especially under few-shot classification.

Gradient-trained models are dominated by Omitted Mentions, whereas few-shot models show comparatively more Completely-O errors.This pattern holds across both granularities and architectures.
Wrong-Type errors rise consistently from coarse to fine across nearly every model, showing that fine-grained classification is harder.Qwen-FS is the sole exception, with Wrong-Type errors dropping from 16.8% to 6.7%.
Qwen-FS’s Wrong-Type exception is associated with Omitted Mentions rising from 26.5% to 76.3%.The increase is driven by UNKNOWN being used for 75.1% of fine-grained predictions, versus 8.0% for Gemma-FS and 2.3% for Gemini.
Boundary errors are asymmetric across models, with under-extension dominating over over-extension, especially in few-shot systems.
Wrong-Type errors mainly reflect confusion among fine-grained categories sharing a coarse type or classification driven by entity relationships.Examples include confusion between mythical mobile and immobile artifacts and tagging Dānu by relationship rather than entity type.

6 Ablation Study

Oracle-span ablations show that mention detection is a major bottleneck, while fine-grained classification remains difficult even with identical oracle detection.

Replacing predicted spans with gold spans raises Gemma’s F1 from 45.26/32.48 to 66.1/50.1 micro/macro, identifying detection as the bottleneck.
With oracle detection, coarse classification exceeds fine-tuned Gemma’s micro F1 of 62.55 but trails its macro F1 of 53.10 with 50.1.
Fine-level oracle settings reach 47.5/34.9 and 45.7/29.4 F1 with and without the coarse label, respectively.Both trail coarse-oracle performance despite identical oracle detection, showing that granularity itself drives difficulty.

7 Conclusion

Padārtha grounds Sanskrit fine-grained NER in the Nyāya–Vaiśeṣika ontology and benchmarks models on the resulting classical-literary task. The benchmark finds comparable performance between task-specific and refined generative systems, but poor fine-grained performance overall.

Padārtha grounds its tagset in Nyāya–Vaiśeṣika rather than adapting a schema designed for modern news.
18 categories sit under 10 hierarchical nodes and collapse to five standard coarse tags, preserving compatibility with existing tools.
108,335 mentions span 73,632 verses, with a hand-corrected 5,000-verse test set built from 12.6K indexed entries.
Task-specific Sanskrit NER designs are comparable with refined generative models, but neither performs well at fine-grained classification.
Fine categories rely more on surrounding context than on the mention itself, limiting memorization-based modeling.

Limitations

The benchmark leaves several practical and methodological boundaries unaddressed. These include Sanskrit sandhi, transfer from epic verse to prose, class imbalance, and source-corpus noise.

Sanskrit sandhi is not addressed, leaving its additional challenges for future work.
Applicability beyond classical epic verse to prose texts may be limited and requires further investigation.
Class imbalance in the dataset is left for future investigation.
OCR and automatic segmentation errors are present, but their impact is outside the study’s scope.

Ethics Statement

The dataset is derived from published, copyright-free literary sources and a publicly available corpus, with resources designated for CC-0 release. No explicit bias analysis was performed.

The annotations derive from published, copyright-free sources and a publicly available corpus, with all resources appropriately cited.
The dataset, annotations, and code will be released under a CC-0 license.
No explicit bias analysis has been performed on the dataset.

AI Assistance

AI assistants supported writing refinement and code debugging during development. The supplied passages do not provide further detail about the experimental role of AI assistance.

AI assistants were used to refine textual clarity, grammar, and phrasing during writing.
AI assistants helped debug code used in the experimental pipeline.

B Implementation Details

The paper combines neural, generative, few-shot, and CRF NER systems with error, per-label, and information-theoretic analyses. It also situates the benchmark within Sanskrit NER resources and defines instruction templates and error categories for fine-grained evaluation.

Model configurations: MuRIL, ByT5, Qwen2.5-7B, and Gemma-4-12B are fine-tuned using task-specific sequence-labeling or generative configurations.MuRIL uses SpanMarkerNER, ByT5 uses sequence-to-sequence tag generation, and the larger language models use LoRA fine-tuning.
Data and evaluation: The benchmark includes coarse-grained and fine-grained train–test distributions, with Table 10 documenting entity-mention allocation across both schemas.The passage identifies Table 10 as the source for these distributions but does not provide the allocation values.
Error analysis: The evaluation diagnoses eight discrepancy types spanning hallucinated mentions, invalid types, boundary errors, false positives, and false negatives.The categories include OOD Mention, OOD Type, Wrong Type, under-extension, over-extension, overlap, Completely-O, and Omitted Mention.
Error analysis: Table 11 reports error distributions across models and both granularities, while Figure 6 reports per-label F1 with color intensity and exact cell values.Figure 6 uses darker green for higher F1 and darker red for lower F1.
V-information analysis: V-information analysis separates entity-only from context-only classification by fine-tuning MuRIL-large on datasets that retain either the entity span or its surrounding context.Pointwise V-information compares each model against a null-input baseline estimated from training label frequencies.
Benchmark context: The benchmark addresses a stated gap in domain-native, gold-standard Sanskrit NER data and systematic comparable benchmarking across approaches.Prior resources described in the supplied passages include coarse, silver-standard, translated, or distantly supervised datasets, while Mahānāma focuses primarily on linking and coreference.