Source-linked AI summary
Nepali Legal Expertise through Generative and Extractive Pre-trained Transformers (NepLEGiT)
Ranjit Raut, Tishya Dhakal, Aaryan Shakya, Bhabuk Thapa, Prasiddha Koirala, Bal Krishna Bal
TL;DR
Nepali legal information is difficult to access because specialized language, fragmented sources, and limited rural legal expertise are poorly served by existing English- and Western-focused legal AI. The paper introduces NepLEGiT, a specialized GPT-2 small language model trained from scratch on Nepali legal text, alongside continual mBERT and MuRIL pre-training. NepLEGiT reaches 1.8 perplexity and 82.9% next-token accuracy, while mBERT reaches 2.35 MLM perplexity versus MuRIL’s 6.07.
Problem
Nepal’s legal accessibility is constrained by formal Nepali terminology, fragmented legal information, shortages of rural legal professionals, and legal AI’s focus on English and Western systems.
Method
NepLEGiT is a GPT-2 decoder model pre-trained from scratch on a curated Nepali legal corpus, complemented by continual masked-language-model pre-training of mBERT and MuRIL.
Results
NepLEGiT achieves 1.8 validation perplexity and 82.9% next-token accuracy; mBERT reaches 2.35 MLM perplexity versus MuRIL’s 6.07.
Takeaways & Limitations
NepLEGiT establishes a baseline for Nepali legal NLP and a documented methodology for low-resource, non-Western legal systems.
Takeaways & Limitations
GPT-2 BPE tokenization produces longer Nepali sequences, reducing contextual coverage within the 128-token window; model outputs require verification by licensed practitioners.
Abstract
from arXiv · showhide
The complexity of legal language and limited accessibility to legal information pose significant challenges to justice delivery in Nepal. Traditional legal services remain inaccessible to many citizens due to language barriers, information fragmentation, and a critical shortage of legal expertise, particularly in rural areas. We present NepLEGiT (Nepali Legal Expertise through Generative and Extractive Pre-trained Transformers), a specialized small language model (SLM) designed to democratize legal knowledge and enhance legal-service delivery in Nepal. We pre-train a decoder-based GPT-2 SLM from scratch on a curated corpus of ~4 million tokens of Nepali legal text, covering constitutional law, civil and criminal codes, and administrative regulations. The model comprises ~30 million parameters in a 6-layer, 6-head, 384-dimensional transformer trained with warmup cosine-decay scheduling, gradient accumulation, and mixed-precision arithmetic. On a held-out validation split, NepLEGiT attains a cross-entropy loss of 0.5684, a perplexity of 1.8, and a next-token prediction accuracy of 82.9%. We further evaluate continual masked-language-model pre-training of mBERT and MuRIL on the same corpus; mBERT achieves a perplexity of 2.35 (eval loss 0.8565), outperforming MuRIL (perplexity 6.07, eval loss 1.8026), providing a strong encoder baseline complementary to NepLEGiT's generative orientation.
1 Introduction
Nepal’s legal system remains difficult to access because formal, specialized Nepali legal language, fragmented information, and shortages of rural legal professionals limit assistance. NepLEGiT addresses these barriers with a curated Nepali legal corpus and specialized language models.
- AI-powered legal tools could help fulfill Nepal’s Digital Nepal Framework mandate for e-governance and digitally empowered citizenship.
- Formal Nepali legal language, Sanskrit-derived terminology, fragmented sources, and scarce rural legal expertise restrict citizens’ access to legal information.The challenges span language barriers, information fragmentation across legal sources, professional shortages, and a technological gap in existing legal AI.
- Nepal’s mixed jurisprudence and rapidly changing regulations are poorly served by legal AI focused on English and Western legal systems.
- NepLEGiT responds with a 10M-token Nepal Law Commission dataset, including a 4M-token clean training split, and specialized GPT-2, mBERT, and MuRIL models.
2 Related Work
Prior legal NLP work shows the value of domain-specific pre-training, but most systems remain concentrated in English and common-law settings. NepLEGiT follows the case for specialized, resource-efficient models tailored to Nepal’s language and legal system.
- Domain-specific pre-training has consistently outperformed general-purpose models on legal NLP tasks, including classification, NER, QA, and entailment.
- mBERT and XLM-R provide multilingual coverage but underperform in specialized legal domains where vocabulary and syntactic conventions diverge from general training data.
- English legal systems dominate prior legal AI, while Nepal’s mixed jurisprudence and formal Nepali require a bespoke low-resource solution.
- Small models trained on carefully curated domain-specific data motivate NepLEGiT’s emphasis on parameter efficiency, fast inference, and deployment under resource constraints.
- Transformer architectures provide distinct trade-offs: bidirectional encoders support understanding, autoregressive decoders support generation, and encoder–decoder models support seq2seq tasks at higher computational cost.
3 Methodology
The methodology constructs a Nepali legal corpus and evaluates decoder and encoder transformer approaches for legal language processing. NepLEGiT uses a GPT-2 causal decoder, while mBERT and MuRIL provide complementary masked-language-model baselines.
- 3.1 Dataset: The corpus is sourced from the Nepal Law Commission and covers the Constitution, civil and criminal codes, and major statutes.
- 3.2 Models: NepLEGiT adopts GPT-2 because causal decoding directly serves open-ended legal text generation, whereas bidirectional encoders target understanding tasks.
- 3.2 Models: The system uses GPT-2 BPE tokenization with NFC normalization, while mBERT and MuRIL use their dedicated Hugging Face tokenizer configurations.
- 3.2 Models: NepLEGiT is a standard causal transformer decoder with pre-layer normalization, GELU feed-forward expansion, tied input-output embeddings, and masked future attention.
- 3.3 Training: Training uses AdamW with decoupled weight decay, a two-phase linear-warmup and cosine-annealing schedule, gradient accumulation, and mixed-precision arithmetic.
4 Results
NepLEGiT converged to strong held-out validation performance, with loss near 0.57, perplexity 1.8, and next-token accuracy 82.9%. Continual MLM pre-training made mBERT the stronger encoder baseline than MuRIL, although their MLM perplexities are not directly comparable to NepLEGiT’s causal metric.
- Training dynamics: Training progressed through rapid descent, steady improvement, and convergence, ending with training loss near 0.57 and validation loss of 0.5684.The convergence phase spanned approximately 50,000–100,000 iterations.
- Training dynamics: NepLEGiT’s validation perplexity fell from approximately 1,800 at initialization to 1.8 during training.The largest reduction occurred within the first 10,000 iterations before stabilization.
- Evaluation scope: mBERT and MuRIL’s MLM perplexities are not directly comparable to NepLEGiT’s CLM perplexity because the objectives predict different token subsets.The encoders predict randomly masked tokens, whereas NepLEGiT predicts every next token autoregressively.
5 Discussion
NepLEGiT’s domain-specific pre-training produces strong language-modeling results, while encoder comparisons favor mBERT for Nepali legal understanding tasks. The discussion also identifies tokenization, overfitting, and responsible-use boundaries.
- 5.1 Domain-Specific Pre-training: 82.9% token accuracy and 1.8 perplexity show NepLEGiT outperforms GPT-2 Small despite using only 26% of its parameters.GPT-2 Small achieves 45–55% token accuracy and has four times more parameters; the comparison attributes the gap to mismatched tokenization, weights, language, and register.
- 5.2 mBERT vs. MuRIL on Nepali Legal Text: mBERT reaches 2.35 perplexity versus MuRIL’s 6.07 on the Nepali legal corpus, making mBERT the stronger encoder starting point.The difference is attributed to mBERT’s broader multilingual pre-training and MuRIL’s emphasis on transliterated and code-mixed Indic text.
- 5.3 Training Dynamics: Near-identical training and validation curves indicate no overfitting, with dropout, weight decay, and modest model capacity contributing to slight underfitting.The authors characterize this as preferable for pre-training followed by fine-tuning.
- 5.4 Perplexity Interpretation: 1.8 perplexity reflects formulaic repetition in legal Nepali but does not directly establish downstream QA or summarization performance.Instruction fine-tuning is identified as necessary for those downstream tasks.
- 5.5 Tokenization Limitations: GPT-2 BPE tokenization produces longer Nepali sequences, reducing contextual coverage within the 128-token window.The authors estimate that a custom 32K–50K Nepali BPE vocabulary could reduce token counts by 20–35%.
- 5.6 Ethical Considerations: NepLEGiT may hallucinate or reproduce historical biases, so generated legal content requires verification by licensed practitioners and cannot replace qualified professionals.The cited examples include gender-asymmetric precedents in the training corpus.
6 Conclusion
The paper presents NepLEGiT, a compact GPT-based model trained from scratch on Nepali legal text, alongside continual pre-training of encoder models. It reports strong validation and baseline-comparison results and positions the work as a foundation for broader Nepali legal NLP development.
- 6 Conclusion: NepLEGiT achieves 1.8 validation perplexity and 82.9% next-token accuracy after training a 30M-parameter GPT decoder on approximately 4M legal tokens.Training uses mixed precision, gradient accumulation, and AdamW with warmup cosine-decay scheduling.
- 6 Conclusion: mBERT reaches 2.35 MLM perplexity versus MuRIL’s 6.07 after continual pre-training, establishing a strong encoder baseline for downstream understanding tasks.This complements NepLEGiT’s generative orientation.
- 6 Conclusion: NepLEGiT establishes a baseline and replicable methodology for Nepali legal NLP and other low-resource, non-Western legal systems.The proposed ecosystem includes multilingual continual pre-training, instruction fine-tuning, retrieval-augmented reasoning, and a public legal information service.
- 6 Conclusion: The authors identify custom Nepali tokenization, decoder-model continual pre-training, supervised fine-tuning, and retrieval-augmented systems as future work.These directions target richer context, downstream task performance, and informal retrieval.