PaperlayerView exact revision on arXiv

Source-linked AI summary

Vagdhenu: A Vrutta (Meter) Aware Shloka-to-Chant (TTS) System for Sanskrit

Prathosh A P

arXiv:2608.26146v1cs.CL

TL;DR

Sanskrit chant synthesis must handle a low-resource domain whose metrical, phonological, and melodic requirements exceed general-purpose TTS. Vāgdhenu builds an experience-report system around a flow-matching backbone with Kannada-routed frontend processing, Sanskrit phonology, and meter-matched reference conditioning. The system reaches about 4.6 expert MOS in a five-hour clone, supports large deployments and releases, while text-side prosody control remains architecturally unavailable on the current backbone.

  • Problem

    Sanskrit parāyaṇa is a chanted register requiring precise phonology and meter, but the domain is low-resource and has specialized melodic prosody.

  • Method

    Vāgdhenu combines Kannada-routed Sanskrit frontend processing, meter-aware reference selection, a flow-matching TTS backbone, and a large-scale neural vocoder.

  • Results

    An expert single-listener mean opinion score of about 4.6 accompanied a five-hour clone that rendered the full conjunct class, including retroflex aspirates.

  • Takeaways & Limitations

    Reference-side meter conditioning works on this backbone, whereas text-side prosody conditioning is inert and designable prosody remains an open problem.

  • Takeaways & Limitations

    Evaluation relies on expert estimates rather than a controlled multi-listener study, and the released weights are single-speaker rather than voice-agnostic.

Abstract

from arXiv · show

We present Vagdhenu, a vrutta (meter) aware shloka-to-chant system for Sanskrit: a text-to-speech system that maps a metrical verse to its chanted parayana recitation at high fidelity. This is an experience report, not a new architecture. We take an off-the-shelf flow-matching TTS backbone and a large-scale neural vocoder, and add the components a faithful Sanskrit chant pipeline needs: a frontend that routes Sanskrit through Kannada orthography to avoid the Hindi-style schwa deletion that Devanagari triggers in Indic models; a frontend that obeys subtle Sanskrit phonology (visarga sandhi with its jihvamuliya and upadhmaniya allophones, the aspiration contrast of alpaprana and mahaprana, and the dental, retroflex, and palatal sibilants kept distinct); and a vrutta-aware mechanism that detects the meter and picks an exactly matched reference under a half-reference rule. We report a negative result that shaped the system: in a self-infilling flow-matching backbone, a text-side prosody conditioner is architecturally inert, because the model recovers pitch from the context mel and the embedding gets no gradient; the reference clip and a voice-steering retrain are the only working prosody levers. We also report a comparative lineage across four families (StyleTTS2, VITS2, Matcha-TTS, and the flow-matching backbone), where each earlier family hit a ceiling on conjuncts or prosody that a five-hour clone cleared at an expert MOS near 4.6. The system shipped two deployments: a 32-chapter, 5183-verse video corpus (about 17.5 hours) and an audio app covering about 18000 verses across 12 books. We release the frontend, inference and training code, weights, a single-speaker chant dataset, and an interactive demo.

1 Introduction

Vāgdhenu is an experience report on building and deploying a faithful, meter-aware Sanskrit chant synthesizer in a low-resource setting. It combines Sanskrit-specific frontend processing, meter-governed reference conditioning, comparative architecture analysis, and production quality-control findings.

  • Parāyaṇa is chanted rather than read, requiring sustained vowels, visarga, consonant distinctions, conjunct fidelity, and metrical structure.
  • The task is low-resource and demands specialized melodic prosody rather than ordinary read-speech synthesis.
  • Vāgdhenu uses detected meter to select matched references, while preserving subtle Sanskrit phonology and sandhi distinctions.
  • A comparative lineage documents conjunct or prosody ceilings across four architecture families and identifies the backbone as the domain bottleneck.
  • Controlled probes show text-side prosody conditioning is inert in the self-infilling backbone, leaving reference clips and voice-steering retraining as working levers.
  • The project reports low-signal forced-alignment quality control, two large deployments, and a public release of code, weights, data, and demonstration.

2 Related work

Prior work supplies strong neural speech-synthesis backbones, vocoders, and prosody-control methods, but Sanskrit chant remains constrained by low resources and demanding phonology. Vāgdhenu addresses these constraints with a dedicated synthesis frontend and evaluates text-side prosody control within its specific backbone.

  • Modern speech synthesis spans autoregressive, non-autoregressive, variational, style-diffusion, adversarial-vocoder, and Fourier-domain approaches.
  • Flow-matching speech models include explicit-duration systems and diffusion-transformer backbones that clone reference clips through mel infilling.
  • Sanskrit synthesis is low-resource and orthographically demanding because conjuncts, retroflex and aspirated contrasts, visarga, anusvāra, and vocalic vowels affect correctness.
  • Prior expressive-prosody methods use phoneme-level models, supervised semantic tokens, or natural-language prompts, but the paper probes their fit for chant control.
  • Alignment supports duration extraction, yet the paper reports it is low-signal as a production-scale synthesis quality gate.

3 The Sanskrit text frontend

The frontend deterministically converts Sanskrit into a backbone-compatible token stream while preserving pronunciation and metrical information. It routes text through Kannada orthography, applies Sanskrit-specific phonology and sandhi, and scans syllable weight for meter matching.

  • The frontend converts continuous saṃhitā text into the synthesis token stream and an alignment transliteration, and is released as open source.
  • Kannada routing preserves Sanskrit’s inherent vowels, avoiding the Hindi-style schwa deletion triggered by raw Devanagari in Indic backbones.
  • The frontend applies visarga sandhi, including jihvāmūlīya and upadhmānīya allophones, with context-sensitive daṇḍa-final realization.
  • It preserves aspiration contrasts, three sibilants, and the full retroflex series, including the hardest retroflex aspirate class.
  • Anusvāra becomes homorganic with the following consonant, vocalic-r cases receive explicit handling, and editorial parentheticals are removed before metrical analysis.
  • Each verse is scanned into laghu-guru sequences and matched to meter signatures that govern reference selection and duration budgets.

4 Model lineage: four eras

Vāgdhenu’s development progressed through four architecture families evaluated on the same Sanskrit task and data. Earlier systems encountered conjunct or prosody ceilings, while the production flow-matching backbone cleared the conjunct class with a short fine-tune and relies on reference conditioning for prosody.

  • Lineage overview: The four-family comparison was retained as a finding because each earlier architecture reached a hard ceiling on conjunct rendering or prosody.
  • Era 1: StyleTTS2: StyleTTS2 produced usable quality, but conjuncts remained muffled and silently loaded English priors affected an early pilot.
  • Era 2: VITS2: VITS2 made output crisper but retained the conjunct ceiling, attributed to sparse data for particular retroflex consonants.
  • Era 3: Matcha-TTS: Matcha-TTS materially improved conjuncts, while a vocoder bake-off identified reductions in hum and repha smoothing.
  • Era 4: Flow-matching backbone: An expert score of about 4.6 followed a five hour flow-matching fine-tune that cleared the full conjunct class, including retroflex aspirates.
  • Production pipeline: The inference pipeline routes text through Kannada, scans meter, selects an exact reference, infills a mel, vocodes waveform audio, and post-gates silence and breath.
  • Reference conditioning: Because the backbone lacks duration and pitch heads, the reference carries voice, melodic contour, and pace; matched selection and half-reference construction are therefore central.

6 Training and compute

Vāgdhenu combines a flow-matching backbone, speaker fine-tuning, and vocoder adaptation with controlled prosody probes that identify which conditioning paths work.

  • Backbone: About 337M parameters define the conditional-flow-matching backbone, trained for masked mel infilling with classifier-free audio and text conditioning dropout.The backbone uses dimension 1024, depth 22, 16 attention heads, text dimension 512, and a convolutional kernel of 4.
  • Voice fine-tune: A single-speaker chant voice is fine-tuned from an Indic checkpoint, then retrained on 179 paired clips to respond more strongly to references.Training uses a 1e−5 learning rate, bfloat16, distributed training across two GPUs, and 100-band mel features at 24 kHz.
  • Vocoder fine-tune: The vocoder is fine-tuned on the backbone’s mel representation and averaged across recent checkpoints, with generator-class matching required for correct loading.A mismatched generator class silently half-loads weights and produces monotone output.
  • Compute and serving: The deployed model totals about 449M parameters, runs at real-time factors of about 1.24 with 64 evaluations and 0.63 with 32, and requires a GPU.Peak inference memory is about 2.5GB; 32 evaluations are the serving setting.
  • Prosody controllability: Text-side prosody conditioning is architecturally inert because context mel already reveals pitch, leaving reference clips and voice steering as the working levers.The conditioner’s gradient norm stayed near 0.02, while high classifier-free guidance changed pitch by about zero semitones; guidance trades expressive tremor against flatter cleanliness.
  • Prosody controllability: Two alternative controllable-prosody lines were dropped after overfitting on small chant data or introducing content drift.Arbitrary designed prosody would require an explicit duration predictor and length regulator, which were designed but not built.

8 Data

The chant corpus is deliberately balanced across meters and difficult Sanskrit sounds, using controlled chant-first recording procedures to preserve phonology and prosodic boundaries.

  • Corpus design: Randomly sampled Sanskrit text would overrepresent one meter and under-cover difficult sounds, so the corpus was designed for metrical and phonetic balance.The source pools were selected for metrical diversity, including Mahābhārata Tātparya Nirṇaya, Sumadhvavijaya, and a purpose-built meter-diverse sheet.
  • Corpus design: About 12,487 half-verse segments were assembled from diverse classical sources and annotated with text, syllable weight, meter, and reference metadata.The manifest represented roughly 40 hours of recording material.
  • Coverage targets: Coverage targets required at least 300 occurrences of each retroflex consonant, 500 long vowels, 200 repha and r-clusters, 150 of each hard conjunct, and 100 of every one of 57 phonemes.Rucirā and mālinī were held out for unseen-meter testing.
  • Recording protocol: The speaker was recorded chant-first in a controlled room, with fixed microphone placement, lossless capture, and a tonic drone monitored but not recorded.The released dataset contains about 1,467 clips, roughly 5.3 hours, in two recording styles; the primary style contains about 3.0 hours over 764 verses.
  • Recording protocol: Each quarter-verse is one continuous breath group, pauses occur only at daṇḍas, and long meters use a fixed caesura breath position.Long vowels, terminal visarga, retroflex and aspirated consonants, and geminates are explicitly sustained or articulated; sessions interleave meters.
  • Recording protocol: The primary-style corpus deliberately flattens the natural meter distribution, where anuṣṭubh alone comprises 48 to 87 percent, while reserving Rucirā and Mālinī as held-out meters.This design ensures long meters are represented and supports generalization testing.

9 Production pipeline and quality control

The production pipeline scales rendering through multi-GPU processing and uses lightweight structural and duration checks after finding forced alignment unreliable at corpus scale.

  • Production pipeline: The render pipeline processes per-hemistich clips with multi-GPU fan-out and removes vocoder onset and tail artifacts using fricative-aware and stop-aware gating.The post-render gate avoids clipping leading fricatives and final stop bursts.
  • Quality control: About 1,648 of 10,220 video-corpus clips were flagged by forced alignment, but manual review found essentially none to be genuine defects.False positives were driven by conjuncts, fricative onsets, long meters, and reference priming.
  • Quality control: Only a cheap duration gate caught genuine collapses, leading production quality control to drop forced alignment in favor of lightweight gates.The same conclusion was carried into the second deployment.
  • Video deployment: About 17.5 hours of audio were rendered into 32 bilingual-card videos covering 5,183 verses in the Mahābhārata Tātparya Nirṇaya.The videos include per-hemistich highlighting and a tonic drone.
  • Deployment scale: The second deployment contained 16,017 rendering units across 12 books and about 345 chapters, with roughly 185 ardhasama verses using nearest-length reference fallbacks.The meter census separates coarse meter classes from the most frequent individual meters.
  • Video deployment: 32 of 32 chapters rendered cleanly across four GPUs, with asynchronous review and surgical single-verse rerenders for the few defects.Most defects were source-numbering or structural issues rather than synthesis failures.

11 Case study II: an audio application

The second case study turns the chant system into an audio-first application for a large scripture corpus, adapting validation and chunking to the source’s structure and prose passages.

  • Application scope: The Śrīmad Bhāgavatam application covers about 18,000 verses across 12 books and roughly 345 chapters as an audio-first mobile deployment.It drops the video half and replaces forced-alignment quality control with structural and source validation.
  • Validation: A pre-render structural audit found no empty text blocks or chapter, verse, or book-numbering mismatches against embedded colophons.The source was described as exceptionally clean before rendering.
  • Prose handling: Prose passages were handled by stripping colophons and parentheticals, splitting only at safe word boundaries, and packing whole words to a target length.The rule avoids breaks at de-sandhied junctions where spaces are not safe boundaries.
  • Application features: The application ships per-verse audio with optional drone mixing, a compact on-device database, and per-chapter karaoke timing tracks.These features adapt the chant pipeline to mobile, audio-first use.
  • Evaluation: Expert single-listener MOS was about 4.6 for the production clone, while all consonant conjuncts, including retroflex aspirates, rendered correctly.Formal evaluation was limited and was treated as a limitation rather than a result; strong commercial systems were estimated at 4.0 to 4.6 and human recitation at about 4.6 to 4.7.
  • Release: The public release includes the frontend, inference and training code, model weights, a 1,467-clip chant dataset, and an interactive demonstration.The release is single-speaker and uses the author’s own voice and data.

14 Limitations and ethics

Vāgdhenu is presented as a practical, released Sanskrit chant system whose scope and evaluation boundaries are stated explicitly. Its main limitation is expert-only evaluation, alongside unresolved constraints on controllable prosody, voice generality, repeated syllables, and rare artifacts.

  • Limitations: The main evaluation limitation is that quality figures are expert estimates rather than a controlled multi-listener study.Lineage MOS values are also not directly comparable across rows.
  • Limitations: Designable, verse-independent prosody remains out of reach on the current backbone and requires a different architecture.The conclusion identifies an architecture with explicit duration and length modeling as the needed direction.
  • Open problems: Voice-agnostic chant would require a multi-reciter chant fine-tune, while repeated-syllable depth beyond about four remains unrecoverable even with priming.Rare artifacts, including a long-vowel bare-visarga edge case, are handled per clip rather than globally.
  • Ethics and intended use: The released system is single-speaker and is intended for recitation, study recordings, and accessibility rather than general voice cloning.The production voice and dataset use the author’s consented recordings.

A Locked inference parameters

The locked system combines a flow-matching backbone, a specialized frontend, meter-aware reference conditioning, and production choices distilled from roughly eighty experiments. The lineage shows successive ceilings in conjunct rendering and prosody before the final backbone configuration reached higher expert MOS.

  • Locked inference parameters: The production configuration uses a flow-matching backbone, Euler solver, 64 function evaluations, sway sampling of about −0.7, and speed factor 0.90.It also uses the large-scale vocoder and high classifier-free guidance of about 3.0 for batch rendering.
  • Frontend: The frontend routes transliterated input through ordered normalization, sandhi, anusvāra, visarga, vocalic-r, and conjunct-handling rules.These operations encode Sanskrit-specific phonology before synthesis.
  • Experiment lineage: About eighty numbered experiments across four eras produced the final system and documented its dead ends.The ledger omits infrastructure details and anonymizes the pilot reciter.
  • Experiment lineage: The StyleTTS2, VITS2, and Matcha-TTS eras reached ceilings involving conjunct quality, vocoder artifacts, or missing meter prosody.The large-scale adversarial vocoder outperformed the Fourier-domain alternative in the Matcha-TTS era.
  • Experiment lineage: The flow-matching era reached about 4.6 MOS with all conjuncts, including retroflex aspirates, correct, and Kannada routing beat transliteration.Implicit meter conditioning was inert, while a voice-steering retrain and half-reference rule became the working prosody route.

D Glossary

The glossary defines the Sanskrit metrical, phonological, recitation, and modeling terms used throughout the paper.

  • Glossary: Vṛtta means meter.
  • Glossary: Gana means a laghu and guru pattern.
  • Glossary: Laghu and guru mean light and heavy syllable, respectively.
  • Glossary: Swara means a melodic note or contour.
  • Glossary: Daṇḍa means verse punctuation, while yati means caesura.
  • Glossary: Pāda means quarter-verse, hemistich means half-verse, and saṃhitā means sandhified continuous text.
Loading 2608.26146v1…