PaperlayerView exact revision on arXiv

Source-linked AI summary

Vectorizing Classical Tamil: Representation Learning for Verse-Commentary Pairs

Amrit Gopinath, Sangeetha Sivanesan

arXiv:2609.04755v1cs.CL

TL;DR

The paper asks what representation learning can recover from a small, inconsistently registered Classical Tamil verse–urai corpus. It constructs and models 1,262 pairs with matched controls, finding weak content generation but a narrow preference for authentic word order in minimal pairs.

  • Problem

    Classical Tamil remains largely absent from contemporary NLP, and consistently aligned verse–urai pairs are difficult to construct because pair boundaries and commentary alignment require explicit decisions.

  • Method

    The study collects 1,262 verse–urai pairs spanning explanatory registers and evaluates recurrent, Transformer, Siamese, encoder–decoder, and decoder-only models against task-appropriate controls.

  • Results

    The decoder prefers authentic order in 95.5% of minimal pairs but does not reproduce commentary content; a constant string also outperforms it on generation overlap.

  • Takeaways & Limitations

    The study supports interpreting small-data representation results through matched noise, null, tokenisation, and held-out-curve controls rather than treating diagnostics as alignment or competence evidence.

  • Takeaways & Limitations

    Representation and Siamese analyses use training pairs with random mismatches, so scores may reflect source identity, length, vocabulary, or commentary register rather than verse–urai alignment.

Abstract

from arXiv · show

We construct a corpus of 1,262 verse-commentary (urai) pairs from five Classical Tamil source sections, ranging from technical grammatical prose to modern paraphrase, and ask what information representation learning can recover. We train recurrent and Transformer encoders, a Siamese-style pair-matching network, an mBART-style encoder-decoder, and a decoder-only language model. Each analysis is interpreted against an appropriate control on the same data. TF-IDF provides a strong no-training lexical retrieval baseline, alongside representation analyses and generation controls for the learned models. A fixed string containing the 25 most frequent commentary words scores higher on generation overlap than the decoder-only model. Canonical correlation reaches 1.000 on Gaussian noise at these sample sizes, token-F1 spans only about 0.02-0.20 on this corpus, and the encoder-decoder continues to lower training loss for sixteen epochs after validation loss has begun to rise. One narrow result remains: the decoder-only model prefers authentic word order in 107 of 112 minimal-pair comparisons (95.5%), but does not reproduce held-out commentary content. We release the extraction and evaluation protocol; redistribution of the source commentaries remains subject to permission.

1 Introduction

The paper asks what models trained on Classical Tamil verse–urai pairs can learn beyond lexical overlap and recurring commentary forms, using a controls-first small-data study. It introduces a corpus spanning varied explanatory registers and evaluates encoding, generation, and word-order preference.

  • Classical Tamil remains largely absent from contemporary NLP, whose Tamil-capable models mainly use Modern Tamil and web text.
  • The corpus contains 1,262 verse–urai pairs spanning modern paraphrase, gloss-and-gist material, and scholarly commentary.Here, urai denotes commentary associated with a verse.
  • The central question is whether verse–urai training recovers information beyond lexical overlap and recurring commentary forms.
  • The study examines recurrent and attention-based encoders, encoder–decoder generation, decoder-only continuation, and representation-space analyses.It compares separate verse and urai encoders with shared and Siamese-style settings.
  • A pair-choice probe tests probability preference for authentic word order, but does not demonstrate grammatical understanding.
  • The work is a pilot controls-first protocol for identifying what small-data models learn from verse–urai pairs and what they do not, rather than a deployable-generator study.

2 Related Work

The related-work discussion situates Classical Tamil within a broader commentary and classical-language NLP landscape while emphasizing the need for explicitly aligned verse–urai data and cautious interpretation of recoverability.

  • Tholkappiyam is an early Tamil grammatical work with an extensive commentary tradition, but its digital editions do not provide a consistently aligned verse–urai corpus.
  • Constructing aligned pairs requires textual extraction and explicit decisions about where verses, commentaries, and usable pairs begin and end.
  • Classical-language NLP has broader Sanskrit resources, while large Indian-language and multilingual models mainly focus on modern text.
  • The paper combines paired-representation methods and an Old Tamil-grounded word-order probe, distinguishing alignment, generation, and probability preference.Recoverability is treated as insufficient evidence that a model uses a linguistic property.

3 Dataset Construction

The dataset construction pipeline assembles and normalizes five Classical Tamil source sections from two digital libraries into a filtered corpus of 1,262 verse–urai pairs, with heterogeneous commentary provenance and style.

  • The corpus draws on five Classical Tamil source sections from Project Madurai and the Tamil Virtual University.Naaladiyar was parsed from Unicode text, while Thirukadukam and three Tholkappiyam adhikarams came from the Tamil Virtual University.
  • The sources are normalized into JSONL records after extracting verse–urai pairs, with the Tholkappiyam material collected manually.
  • 1,262 usable pairs remain from 1,277 parsed records after rows with empty verses or urai are dropped.No additional length-based filtering is applied in the v3 experiments.
  • Table 1 counts verse–urai pairs used for training after filtering, while noting unresolved Thirukadukam attribution and Naaladiyar’s status as modern paraphrase.
  • The resulting corpus contains 1,262 verse–urai pairs.
  • Urai are substantially longer than verses on average, with the largest gap in Eluttatikaram.

4 Experimental Setup and Methodology

The experiments use small-data models and multiple matched controls across representation learning, generation, retrieval, and pair-choice evaluation. Tokenization, split design, and control construction define important boundaries on interpretation.

  • With 1,262 pairs, scores are interpreted against matched controls including TF-IDF, Gaussian CCA matrices, and token-F1 null distributions.
  • LSTM, BiLSTM, and Transformer encoders use masked-word objectives, while a Siamese Transformer classifies matched pairs versus mismatches.
  • Generation uses an mBART-style encoder–decoder and a decoder-only model for verse-to-urai continuation.
  • 4.3 Tokenisation: Whitespace tokenization retains whole words, a limitation for agglutinative Classical Tamil.
  • The corpus uses a deterministic 90/10 split with 1,136 training pairs and 126 held-out pairs.
  • Generation and token-F1 use held-out verses, while the vocabulary is built on the full corpus and no out-of-vocabulary evaluation is performed.
  • The pair-choice test compares equal-length real and perturbed verses using summed token log-probabilities, with correctness defined by the original receiving the higher score.The set is small and dominated by random reorderings.

5 Results and Analysis

The controls show that lexical retrieval is a meaningful baseline, while CCA and token-F1 can look misleading without matched null comparisons. Separate encoders and pooled pair classification do not establish reliable verse–urai alignment.

  • Controls: 35.3% TF-IDF R@1 exceeds chance below 1%, making lexical retrieval the relevant baseline.Thirukadukam reaches 92.0% because its commentary repeats verse words as gloss headwords.
  • Representation diagnostics: 1.000 CCA on matched Gaussian noise shows that raw canonical correlation is not evidence of learned alignment at these sample sizes.The vectors are 256-dimensional and pools contain 100–393 pairs.
  • Generation: 0.060 token-F1 falls below TF-IDF retrieval at 0.084 and a constant 25-word string at 0.108.The usable range on this corpus is roughly 0.02–0.20, and an identical rerun scored 0.057.
  • Representation diagnostics: Matched-pair cosine similarities remain near zero across separate encoder representations.LSTM matched-pair means are 0.042, 0.021, 0.045, and 0.002 for the reported source settings.
  • Pair classification: 0.500–0.600 within-source pair-classifier accuracy is close to chance, whereas pooled accuracy of 0.842–0.845 can exploit source-specific cues.The pooled classifier result therefore does not establish verse–urai alignment.

5.5 t-SNE and Siamese matching show what the objective encourages

t-SNE groups representations more clearly by source and commentary register than by individual verse–urai pairing. Siamese matching separates observed labels on training pairs, but this does not establish generalisation.

  • 5.5 t-SNE: t-SNE plots group points by source text and commentary register more clearly than by individual verse–urai pairing.Because t-SNE can exaggerate apparent clusters, it supports rather than independently establishes the quantitative results.
  • 5.5 Siamese matching: The Siamese attention model produces training-set cosine gaps of 0.517, 0.462, and 0.500 across three source pools, with 0.345 when combined.These gaps show fit to observed pair labels, but evaluation uses training pairs and randomly sampled negatives, so generalisation is not established.
  • 5.5 Siamese matching: Figure 1 contrasts the separate and Siamese settings to show what their objectives encourage, not to provide an additional alignment test.The projection is presented as a visual contrast between the two training objectives.

5.6 The encoder–decoder overfits after epoch 4

The encoder–decoder fits the training data increasingly well after epoch 4, while held-out performance worsens. Its generations also reuse commentary material rather than reliably producing input-specific explanations.

  • Validation loss reaches its minimum at epoch 4, while training loss continues falling through epoch 20.Training loss falls from 5.78 to 1.86, whereas held-out loss rises from 9.42 at epoch 4 to 9.79 at epoch 20.
  • Four of five inspected generations begin with the same phrase despite targeting different Naaladiyar verses.The inspection also found long exact phrases from unrelated training rows.
  • The t-SNE panels show objective-induced proximity, not quantitative evidence of alignment or generalisation.Separately trained encoders occupy distinct regions, while the Siamese model brings many training pairs together because its objective rewards matching.
  • The decoder combines familiar commentary fragments rather than producing explanations specific to the input verse.This pattern also appears on out-of-domain Thirukkural couplets, where generated commentary has zero mean token overlap with the input.

5.8 Cross-attention recovers vocabulary overlap

Cross-attention is dominated by repeated vocabulary, especially from Tholkappiyam, rather than evidence of semantic alignment. The broader diagnostics also show that local form preference can coexist with weak commentary-content recovery.

  • Across 579,818 word pairs, all twenty highest attention pairs come from Tholkappiyam, and many link words to identical reused words.The strongest attention patterns therefore reflect verbatim vocabulary reuse rather than semantic alignment beyond TF-IDF.
  • The decoder-only model prefers authentic word order in 107/112 pairs (95.5%), but held-out token-F1 is 0.060 versus a 0.108 constant baseline.This captures a preference for form without recovering commentary content.
  • The pooled word-order result is driven primarily by 99 random line reorderings, while the inflection item is excluded because it contains an out-of-vocabulary token.The high pooled score should be interpreted alongside this imbalance between targeted regularities and random reorderings.
  • Urai length varies from about 1.7× to 8.7× verse length across sections, and verse input alone does not reliably identify the target explanatory register.A model may therefore produce scholarly commentary-style language when the target is a modern paraphrase.

6 Reproducibility

The release records the preprocessing, experiments, seeds, and numerical results, while noting that exact token-F1 replication is not expected.

  • All from-scratch results use v3 preprocessing with 1,262 pairs and seven special tokens.The run ledger and complete numerical record are stored in experiments_v2/RESULTS_AND_METHODOLOGY.txt.
  • The release notebooks cover the controls, encoder models, encoder–decoder, and decoder-only experiments.The final encoder–decoder result comes from the executed Kaggle notebook rather than the local copy.
  • Model notebooks use seed 3407, while the probe script uses seed 42.
  • Exact token-F1 replication is not expected because the reported value uses 30 held-out rows rather than the complete held-out split.

7 Limitations

The study’s limitations concern corpus scope, evaluation design, tokenisation, and the heterogeneity of commentary styles. These constraints limit claims about generalisation, linguistic competence, and output quality.

  • Corpus scope: 1,262 pairs span five source sections, but the corpus is small and unevenly distributed.Porulathikaram contributes 103 pairs, whereas Naaladiyar contributes 393.
  • Generalisation: The train–held-out split does not test transfer to new texts, commentators, periods, or editorial traditions.Each model uses one configuration and seed, and the vocabulary is built before splitting, so out-of-vocabulary generalisation is not evaluated.
  • Evaluation design: Representation and Siamese analyses use training pairs with random mismatches, allowing source identity, length, vocabulary, or register to substitute for pair alignment.Repeated seeds and held-out, within-source retrieval are needed before stronger representation-learning claims.
  • Preprocessing: Whole-word tokenisation can assign zero overlap to meaningful stem or suffix changes and discards verse line structure.Representations for metre or prosody would need to handle line structure consistently across original and perturbed text.
  • Grammar probe: The grammar probe contains 112 usable pairs, is dominated by random reorderings, and does not establish broad grammatical competence or use during urai generation.Some parses were machine-generated and only partly checked by hand.
  • Target heterogeneity: Urai combine modern paraphrase, lexical glosses, and scholarly commentary, so lexical overlap with a gold text does not ensure stylistic appropriateness.The paper does not include independent expert adjudication of every extracted pair or generated explanation.

8 Conclusion

The conclusion emphasizes that small-data representation results require matched controls and cautious interpretation. The decoder shows a narrow preference for authentic word order, but not recovery of commentary content.

  • Conclusion: No held-out learned verse–urai retrieval result is reported for the 1,262-pair corpus.TF-IDF is the lexical retrieval baseline, while representation and Siamese analyses are noncomparable diagnostics.
  • Conclusion: Small-data interpretation requires matched Gaussian controls for CCA, null distributions for token-F1, identical tokenisation, and held-out curves for epoch selection.These conditions address underconstrained correlation, overlap baselines, input comparability, and overfitting.
  • Conclusion: 95.5% of minimal pairs receive higher probability in authentic order, while the decoder fails to reproduce commentary content.This separation between form and content is a diagnostic, not evidence of grammatical competence.
Loading 2609.04755v1…