Source-linked AI summary
NepKANUN: A RAG-Based Nepali Legal Assistant
Bhabuk Thapa, Prasiddha Koirala, Ranjit Raut, Sunil Regmi, Bal Krishna Bal
TL;DR
Access to Nepali legal information is limited by complex terminology, scarce resources, and shallow existing systems. NepKANUN addresses this gap with a curated legal QA dataset, a fine-tuned LLaMA model, and RAG over reliable legal texts. It achieves strong BERTScore results on basic queries while showing reduced performance on complex legal reasoning and remaining bounded by dataset, retrieval, training, and ambiguity limitations.
Problem
Nepali legal information access is hindered by complex terminology, limited resources, scarce annotated datasets, and keyword-based systems lacking semantic depth.
Method
NepKANUN fine-tunes LLaMA 3.2 3B on 10,000 high-quality Nepali legal QA pairs and uses RAG to generate answers from retrieved legal context.
Results
BERTScore F1 reaches 0.82 for basic queries and 0.71 for complicated queries, with human evaluation reporting 5.0/5 Relevance and 4.5/5 Faithfulness for basic questions.
Takeaways & Limitations
The system provides basic Nepali legal information grounded in reliable sources, supporting broader access while not replacing expert assistance in complicated cases.
Takeaways & Limitations
Retrieval, generalization, peak performance, and intricate legal reasoning are limited by domain-specific embedding gaps, a 10,000-pair dataset, efficiency-focused fine-tuning, and ambiguous texts.
Abstract
from arXiv · showhide
Accessing legal information in Nepal is difficult due to complex terminology, limited resources, and misinformation. We introduce an AI-powered legal assistant that is tailored for Nepali legal texts and is built on a fine-tuned large language model. The technology provides precise, streamlined answers to natural language legal inquiries when integrated into a Retrieval-Augmented Generation (RAG) framework. It was trained using a custom dataset of high-quality question-answer pairs, and according to BERTScore, it obtained strong F1 scores of 0.82 (simple), 0.77 (moderate), and 0.71 (complex). Its usability is further confirmed by expert reviews. Our method shows how merging generation and retrieval can effectively democratize access to legal knowledge in Nepal by focusing on customized legal data and incorporating RAG.
1 Introduction
Nepali legal information access is constrained by complex language, limited resources, scarce annotated data, and shallow keyword-based systems. NepKANUN addresses these gaps with a curated Nepali legal QA dataset and a RAG-based assistant built on a refined LLaMA model.
- Nepali users face complex legal terminology, limited judicial resources, and low public legal knowledge when seeking rights and duties.
- Nepali legal NLP is constrained by scarce annotated datasets and keyword-based systems that lack semantic understanding of user inquiries.
- RAG grounds LLM responses in retrieved reliable sources, integrating domain knowledge to improve factual accuracy and traceability.
- The project creates a domain-specific Nepali legal question-answer dataset to fill the gap in annotated legal data.
- NepKANUN combines the refined LLaMA 3.2 3B model with RAG to generate precise, context-aware answers to legal queries.
2 Methodology
NepKANUN fine-tunes LLaMA 3.2 3B on a cleaned Nepali legal QA dataset and uses structured retrieval over authoritative legal texts. Its workflow embeds queries and chunks, retrieves diverse relevant passages, and generates Nepali answers from that context.
- 2.1 Data Preparation: The final NepKANUN dataset contains 10,000 manually validated Nepali legal QA pairs after scraping, OCR, deduplication, and correction of OCR errors.
- 2.2 Model Fine-tuning: LLaMA 3.2 3B is fine-tuned with LoRA and 4-bit QLoRA to reduce trainable parameters, memory use, and computational cost.
- 2.3 RAG Methodology: RAG grounds generated responses in reputable legal materials, including Nepal’s Constitution 2072, the Environmental Act, and Muluki Ain.
- 2.3 RAG Methodology: Queries and legal chunks share a 384-dimensional embedding space, where ChromaDB supports similarity search over indexed multilingual representations.
- 2.3 RAG Methodology: MMR retrieves k = 9 diverse relevant chunks, which the fine-tuned LLaMA model combines with the Nepali query to generate the final response.
3 Results and Discussion
NepKANUN performs strongly on simple legal questions but declines on more complex queries. Automated BERTScore and expert ratings jointly indicate accurate retrieval and response alignment, alongside difficulty with nuanced legal reasoning.
- 3.1 Automated Evaluation: BERTScore evaluates semantic alignment through contextual token embeddings rather than relying only on surface-level overlap from ROUGE or BLEU.
- 3.2 Human Evaluation: Human evaluation used five-point ratings for Faithfulness, Relevance, Logical Correctness, Completeness, and Interpretability.
- 3.3 Discussion: BERTScore F1 is 0.82 for basic queries and 0.71 for complicated queries, while complex-query Completeness and Logical Correctness are each 4.0/5.The reported decline indicates difficulty synthesizing and explaining nuanced or ambiguous legal concepts.
- 3.3 Discussion: Basic queries received a perfect Relevance score of 5.0/5 and a Faithfulness score of 4.5/5 in human evaluation.
- 3.3 Discussion: The system is useful for retrieving basic legal information but requires improvement for deeper legal analysis.
4 Conclusion
NepKANUN combines RAG with a refined LLM to provide straightforward Nepali legal answers grounded in reliable sources. It broadens access to basic legal knowledge while remaining distinct from expert assistance for complicated cases.
- NepKANUN provides precise, straightforward Nepali legal answers grounded in reliable sources through a RAG framework and refined LLM.
- The system offers people a basic grasp of their legal rights and duties and does not replace expert legal assistance in complicated instances.
5 Future Work
Future work targets greater robustness and broader impact by keeping the legal knowledge base current, expanding input modalities, and supporting regional Nepali languages.
- Future Work: Automatic legal-knowledge updates could incorporate new laws and rulings, potentially through official governmental databases.The proposal remains subject to privacy and technical constraints.
- Future Work: Speech queries are identified as a future multimodal input for the system.
- Future Work: Extending language support to regional Nepali languages is proposed to broaden the system’s impact.
6 Limitations
The system’s effectiveness and generalization are constrained by retrieval, data coverage, training efficiency, and difficult legal reasoning conditions.
- Limitations: Nepali legal retrieval may be limited because specialized embedding models are unavailable.
- Limitations: The 10,000-pair dataset may limit generalization by underrepresenting rare or extremely complicated legal issues.
- Limitations: Efficiency-focused QLoRA fine-tuning for only two epochs may have constrained peak model performance.
- Limitations: Ambiguous or interpretable legal texts may reduce reliability in intricate legal reasoning.