PaperlayerView exact revision on arXiv

Source-linked AI summary

Padamitra: Grounded Glossary Generation for Classical Sanskrit

Manoj Balaji Jagadeeshan, Sai Pragnaan Marala, Pawan Goyal

arXiv:2608.25038v1cs.CL

TL;DR

Sanskrit glossary generation requires recovering semantically meaningful phrases and assigning meanings grounded in a specific translation, but existing approaches and metrics leave this structured task insufficiently evaluated. The paper formalizes the task, builds a 31,316-example benchmark with two metrics, and evaluates several prompting and fine-tuning settings. Instruction fine-tuning substantially outperforms prompting, while segmentation and word-order behavior remain important challenges.

  • Problem

    Existing methods do not directly evaluate Sanskrit phrase recovery with translation-grounded meanings, and model behavior, segmentation supervision, and phrase-level semantic faithfulness remain unclear.

  • Method

    The paper formalizes grounded glossary generation, constructs a benchmark from the Valmiki Ramayana and Srimad Bhagavatam, and evaluates models with Jaccard and Meaning Faithfulness.

  • Results

    Instruction fine-tuning substantially outperforms prompting across the evaluated model variants, while explicit segmentation supervision and word ordering affect performance.

  • Takeaways & Limitations

    The study makes interpretable Sanskrit lexical decomposition an explicit evaluable objective and identifies segmentation quality as important for semantic faithfulness.

  • Takeaways & Limitations

    The metrics rely on surface matching and embedding similarity and may penalize morphologically equivalent but orthographically distinct sandhi resolutions.

Abstract

from arXiv · show

We introduce grounded glossary generation, a structured task requiring models to recover semantically meaningful Sanskrit phrases and produce translation-grounded meanings from a sloka-translation pair, formalizing the traditional patha commentary practice as an evaluable NLP objective. We construct a benchmark of 31,316 sloka-translation-glossary triples from the Valmiki Ramayana and Srimad Bhagavatam, paired with two metrics: Jaccard for phrase recovery and Meaning Faithfulness for semantic consistency. Across zero-shot, few-shot, and instruction fine-tuned variants of Gemma-3n-E4B, Gemma-3-12B, Phi-4, and Qwen3.5-9B, instruction fine-tuning substantially outperforms prompting, while explicit segmentation yields gains. Error analysis identifies over-segmentation of sandhi and samasa compounds as the dominant failure mode, pointing to morphological modeling as the key bottleneck for faithful Sanskrit lexical decomposition.

1 Introduction

Classical Sanskrit glossary generation recovers semantically meaningful phrases and meanings grounded in an authoritative translation. The paper benchmarks this task and finds instruction fine-tuning substantially outperforms prompting, while segmentation and word order remain important challenges.

  • Motivation: Classical Sanskrit interpretation is difficult because sandhi and samasa create surface forms whose semantically meaningful segmentation is non-trivial.Traditional patha commentary addresses this by mapping resolved verse phrases to contextual meanings.
  • Task formulation: Grounded glossary generation maps each resolved Sanskrit phrase in a sloka to a meaning faithful to its authoritative English translation.This makes interpretable lexical decomposition an explicit, evaluable objective rather than an opaque by-product of translation.
  • Research gaps: Prior work leaves Sanskrit glossary generation underexplored because token alignment, model behavior, phrase-level recovery, semantic faithfulness, and segmentation supervision remain insufficiently addressed.The task requires sandhi resolution, boundary detection, and translation-grounded meaning assignment.
  • Contributions: 31,316 sloka–translation–glossary triples from the Valmiki Ramayana and Srimad Bhagavatam form the benchmark.The benchmark is paired with Jaccard for glossary key recovery and Meaning Faithfulness for translation-grounded semantic consistency.
  • Findings: Instruction fine-tuning substantially outperforms zero-shot and few-shot prompting, while explicit segmentation provides limited gains and over-segmentation remains a dominant failure mode.Instruction-fine-tuned models are also susceptible to word-order issues.

2 Problem Statement

The paper defines grounded glossary generation as structured phrase recovery and translation-grounded meaning assignment for Sanskrit sloka–translation pairs. Unlike translation and lexicon induction, it jointly addresses morphologically complex boundaries and contextual semantic faithfulness.

  • Task definition: Grounded glossary generation takes a Sanskrit sloka and its English translation as input and produces glossary pairs of Sanskrit phrases and translation-grounded meanings.Each key is a semantically meaningful phrase, while each value is grounded in the provided translation.
  • Illustration: A representative example aligns Sanskrit phrases such as नैिमषे with contextual meanings including “in the forest known as Naimisaranya.”The glossary also captures compounds, relational meanings, and inflected forms from the translation.
  • Distinction from related tasks: The task differs from machine translation, bilingual lexicon induction, and context-free definition modeling by requiring phrase-boundary recovery and meanings faithful to one specific translation.Its objective is not merely fluent output or plausible isolated glosses.

3 Dataset

The dataset contains structured Sanskrit sloka, English translation, and glossary mappings curated from the Ramayana and Srimad Bhagavatam. Normalized train, validation, and test splits are separated to reduce leakage.

  • Sources: The dataset is curated from two publicly available educational resources: the Ramayana and Srimad Bhagavatam.Each instance includes a Sanskrit sloka, English translation, and glossary mapping.
  • Processing: Semi-structured glossary annotations are parsed into structured key-value dictionaries with light Unicode and whitespace normalization.The original linguistic structure is preserved during normalization.
  • Splits: The dataset is split into 25,050 training, 3,133 validation, and 3,133 test instances.These counts define the benchmark partitions used for evaluation.
  • Leakage control: Contiguous verses and near-duplicate glossary entries are prevented from crossing train, validation, and test splits to reduce leakage.The dataset also examines overlapping gloss keys across partitions.

4 Evaluation Metrics

The evaluation measures both Sanskrit glossary-key recovery and semantic faithfulness of generated meanings. Jaccard evaluates key overlap, while Meaning Faithfulness combines key matching with embedding-based meaning similarity and penalizes spurious entries.

  • Overview: The evaluation has two dimensions: correctness of generated Sanskrit glossary keys and semantic faithfulness of generated meanings.These dimensions correspond to phrase recovery and meaning consistency.
  • Jaccard: Jaccard compares generated and reference glossary-key sets to measure recovery of reference semantic units.Each glossary key represents a Sanskrit phrase segment.
  • Meaning Faithfulness: Meaning Faithfulness matches generated keys to reference keys using normalized Levenshtein similarity and compares matched meanings with SentenceTransformer cosine similarity.A generated key is matchable only when similarity exceeds 70.
  • Aggregation: Each glossary entry contributes a product of key similarity and value similarity, and the final datapoint score averages contributions across generated entries.Unmatched keys contribute zero, while weak matches lower the overall average.
  • Segmentation behavior: Severe under-segmentation or over-segmentation can drive key similarity below threshold, whereas proper segmentation preserves high similarity.Exact key matches receive 100% similarity under the described intuition.

5 Methodology

The methodology evaluates Sanskrit glossary generation with multiple model families and compares prompting, instruction fine-tuning, and segmentation-oriented approaches.

  • The evaluated glossary-generation models are gemma-3n-E4B-it, gemma-3-12b-it, phi-4, and Qwen3.5-9B.
  • FastAlign provides a non-LLM baseline for generating glossaries from Sanskrit slokas and English translations.FastAlign is described as a lightweight statistical word-alignment model based on IBM Model 2.
  • ByT5-Sanskrit is evaluated with end-to-end glossary generation and a two-stage segmentation-plus-generation pipeline.The two-stage setup separately predicts glossary-key segments and then generates glossary entries from those segments and the reference translation.
  • In-context learning is tested with zero-shot and few-shot prompting using 1, 5, and 10 examples.The prompts and examples are provided in the paper’s appendices.
  • Instruction fine-tuning uses LoRA-based training with model-specific adaptations to rank, modules, loss, batch size, and sequence length.The shared setup includes a learning rate of 3 × 10^-5, cosine scheduling with warmup, BF16 precision, and 10 training epochs.

6 Result

Instruction fine-tuning substantially outperforms FastAlign and in-context learning across models and metrics, while errors are dominated by over-segmentation of Sanskrit compounds.

  • 6.1 Quantitative Analysis: Instruction fine-tuning substantially outperforms FastAlign and in-context learning across Jaccard and Meaning Faithfulness, with Phi-4 achieving the best overall performance.Increasing in-context examples generally improves results over zero-shot prompting, but instruction fine-tuning produces larger and more stable gains.
  • 6.1 Quantitative Analysis: Meaning Faithfulness scores are consistently higher than Jaccard scores, indicating semantically plausible meanings despite imperfect phrase segmentation.
  • 6.2 Qualitative Analysis: The qualitative analysis examines 174 low-scoring samples from the bottom 5% by Meaning Faithfulness and groups errors into six categories.
  • 6.2 Qualitative Analysis: Over-Segmentation is the dominant failure mode, accounting for 70% of errors involving incorrect fragmentation of Sandhi and Samasa compounds.
  • 6.2 Qualitative Analysis: The example trasaren.uh.(त्रसरेणुः) is incorrectly split into tra(त्र) and saren.uh.(सरेणुः).
  • 6.2 Qualitative Analysis: Concurrent over- and under-segmentation indicates inconsistent boundary detection, making morphology-aware constraints and compound boundary modeling critical.Purely semantic errors are comparatively rare.
  • 6.1 Quantitative Analysis: +66 points on seen keys and +60 points on unseen keys are reported for Phi-4 under instruction fine-tuning.The comparable gains across 12,194 seen and 8,956 unseen gloss keys are presented as evidence against simple memorization.

7 Ablations

The ablations test explicit segmentation and robustness to word-order variation. Gold segmentation improves semantic faithfulness, while shuffled word order degrades performance and exposes reliance on sequential cues.

  • Segmentation-Augmented Glossary Generation: Table 4 evaluates segmentation-augmented glossary generation with instruction-tuned models that segment the sloka before generating meanings.The table concerns the role of explicit structural decomposition in the glossary pipeline.
  • Segmentation-Augmented Glossary Generation: Direct segmentation-augmented generation performs nearly identically to direct glossary generation for the selected models.The ablation compares instruction-tuned models that first generate semantic Sanskrit segments and then grounded meanings.
  • Segmentation-Augmented Glossary Generation: Gold segmentation raises Meaning Faithfulness from 0.787 to 0.872 when models generate glossary meanings from supplied segments.This improvement shows that segmentation quality affects semantic faithfulness in addition to key recovery.
  • Robustness to Word Order Perturbation: Randomly shuffling words within each sloka line causes noticeable performance degradation, particularly for Qwen3.5-9B.The perturbation preserves original line boundaries while changing word order.
  • Robustness to Word Order Perturbation: The perturbation results indicate that adapted models still rely on sequential cues and are not fully robust to Sanskrit’s relatively free word order.This finding motivates robustness analysis beyond standard in-order evaluation.

8 Conclusion

The paper introduces grounded glossary generation as an evaluable task for recovering meaningful Sanskrit phrases and translation-grounded meanings. It contributes a benchmark and evaluation framework, while identifying morphology and compound boundary detection as primary bottlenecks.

  • Conclusion: Grounded glossary generation requires recovering semantically meaningful Sanskrit phrases and producing glossary meanings grounded in translation.The task formalizes a structured objective for Sanskrit lexical decomposition.
  • Conclusion: The paper presents a benchmark dataset, evaluation framework, and systematic study of large language models for glossary generation.The study examines instruction fine-tuning, explicit segmentation supervision, and word-ordering effects.
  • Conclusion: Morphology and compound boundary detection are identified as the primary bottlenecks for Sanskrit glossary generation.The paper points to morphology-aware modeling, retrieval-grounded generation, and multilingual lexicographic systems as future directions.

Limitations

The benchmark’s scope, metrics, adaptation setup, and error analysis impose important limits on how broadly its results should be interpreted.

  • Data scope: The dataset draws exclusively from the Vālmīki Rāmāyaṇa and Śrīmad Bhāgavatam, limiting generalization to other Sanskrit literary traditions.The paper specifically names Kāvya and Vedic prose as traditions with distinct compositional styles.
  • Evaluation: Jaccard and Meaning Faithfulness may penalize valid sandhi resolutions when they are orthographically distinct from the reference.Jaccard uses surface-level string matching, while Meaning Faithfulness uses embedding similarity.
  • Training setup: Memory-constrained instruction fine-tuning may underestimate the models’ performance ceiling.The setup required reduced batch sizes and shorter sequence lengths.
  • Training setup: Using LoRA rather than full fine-tuning leaves open whether additional parameters would further close the segmentation gap.All models were evaluated with LoRA-based adaptation.
  • Error analysis: The error analysis examines only the bottom 5% of samples by Meaning Faithfulness, which may not represent failures across the full score distribution.Its conclusions should therefore be read as characterizing the lowest-scoring cases.

A.1 Dataset Gloss Key Overlap between Train, Validation and Test

This section combines dataset-overlap documentation with evaluation and qualitative analyses of glossary-key recovery, translation quality, and model failure patterns.

  • Dataset split analysis: Table 6 reports glossary-key overlap information across the train, validation, and test splits.Contiguous verses and near-duplicate glossary entries were prevented from appearing across splits to reduce leakage.
  • Translation capability: Translation-quality rankings track Jaccard rankings, with Phi-4 leading and Qwen, Gemma-3, and Gemma-3n clustering in the middle.This comparison connects benchmark performance with translation capability and Sanskrit pre-training exposure.
  • Qualitative analysis: The qualitative analysis evaluates 174 representative samples from the bottom 5% by Meaning Faithfulness and groups errors into six mutually exclusive categories.The categories are based on mismatches between reference and generated glossary entries.
  • Qualitative analysis: Over-segmentation accounts for 70% of analyzed cases, fragmenting sandhi- and samasa-formed compounds into smaller units.The paper identifies this as the most frequent issue and links it to loss of contextual meaning.
  • Qualitative analysis: Concurrent over- and under-segmentation indicates inconsistent boundary detection, while purely semantic errors are comparatively rare.Under-segmentation remains harmful despite being less common, and some reference glossary translations were themselves incorrect.
  • Implications: Stronger morphological constraints and compound-boundary modeling are identified as priorities for reducing over-fragmentation.Jaccard was selected for set-overlap recovery because it jointly penalizes missing and spurious glossary units.
Loading 2608.25038v1…