PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross Lingual Transfer in Tulu Legal Comprehension: Script-Dependent Improvement and RAG-Induced Knowledge Conflict

Sindhu Shetty, Spurthi Setty, Natan Vidra

arXiv:2608.28645v1cs.CLcs.AI

TL;DR

Low-resource languages may require related-language scaffolds, but rigorous evaluation is needed to identify cross-lingual comprehension failures. This study tests minimal priming and Kannada legal-corpus RAG for classifying unseen Tulu complaints, finding script-dependent comprehension and mixed RAG effects, including Llama3’s collapse and recurring fact substitution and confabulation.

  • Problem

    The study addresses whether models can interpret Tulu legal complaints when Tulu lacks substantial training resources and legal administration uses Kannada or English.

  • Method

    The authors evaluate three models on 60 Tulu legal complaints across scripts, priming conditions, and Kannada legal-corpus RAG, using classification and reasoning-trace analyses.

  • Results

    Kannada-script queries showed above-baseline comprehension, while RAG produced mixed outcomes: Llama3 collapsed, and failures involved fact substitution and confabulation.

  • Takeaways & Limitations

    Script-dependent comprehension and RAG robustness travel together, making parsing and subsequent reasoning central targets for low-resource multilingual RAG evaluation.

  • Takeaways & Limitations

    The 60-query pilot supports preliminary conclusions, while many trends would require hundreds or thousands of queries for confident significance assessment.

Abstract

from arXiv · show

Low-resource languages without an adequate training corpus often use a related, higher-resource language as a scaffold for comprehension. Still, there is a need to develop rigorous evaluation methods to identify when models fail in cross lingual low-resource environments. Using the legal domain as a backdrop, three models (Llama3, Hex-1, Sarvam) were tested on the ability to classify legal complaints written in a low resource Dravidian language (Tulu). Transliterating queries across Dravidian scripts allowed models to gain a preliminary understanding of speakers' complaints without the use of wide scale training, though the level of comprehension was heavily script dependent (with Kannada - another relatively low-resource language - producing the strongest positive trend). Retrieving from a corpus of Kannada legal papers across a RAG framework caused mixed results. Some models had a weak positive trend in comprehension under certain conditions, but when models failed, it was often across two axes: fact substitution (fixating on specific passage excerpts that skewed reasoning) and confabulation (hallucination that had no basis in either query or corpus). Within low resource domains, results identify the model's parsing of information and subsequent reasoning as the source of reasoning failure, rather than corpus contents. Script-dependent comprehension and RAG robustness also seem to travel together. This is further supported by the reasoning-trace analysis and a statistical-honesty framework deployed - techniques that are more broadly applicable to low-resource multilingual RAG evaluation.

1 Introduction

The paper evaluates whether minimal priming and Kannada legal papers can help models interpret unseen Tulu legal complaints without translation or extensive training. It focuses on script-dependent cross-lingual comprehension in a legal setting where spoken and administrative languages differ.

  • Tulu speakers may need to represent themselves in Kannada or English legal systems, creating a cross-lingual access problem.
  • Tulu is extremely low resource, while Kannada is the administrative language for legal proceedings in Karnataka.
  • The study tests whether minimal priming and Kannada legal papers can help models interpret previously unseen Tulu queries.
  • Because Tulu traditionally uses Kannada abugidas and shares vocabulary and sentence structure with Kannada, the authors propose that Kannada script may trigger stronger comprehension.
  • Three models classify Tulu complaints into LIST categories to evaluate comprehension rather than build a complete low-resource-language LLM.

2 Related Work

Prior work frames the study around knowledge conflict, retrieval-induced degradation, hallucination in legal NLP, and the special difficulty of retrieval across low-resource languages. The paper positions cross-lingual multilingual RAG in the legal domain as relatively unexplored.

  • Knowledge-conflict research shows that models can inconsistently weigh retrieved context against their internal understanding.
  • Prior studies report that retrieval can reduce downstream performance when evidence is misleading or model-retriever combinations are unfavorable.
  • Legal RAG systems can hallucinate even when processing relevant legal papers, making small knowledge conflicts consequential.
  • Low-resource language pairs rely more heavily on retrieved context and degrade more severely when that context is noisy.
  • The intersection of low-resource multilingual transfer and legal-domain RAG remains relatively unexplored.
  • Existing Tulu-Kannada research supports Kannada as a scaffold because of extensive language contact and similarity to Kannada-trained models.

3 Methodology

The study evaluates 60 informal Tulu legal complaints across scripts, priming levels, retrieval conditions, models, and comprehension checks. It combines LIST classification with reasoning-trace verification and statistical tests to distinguish comprehension from coincidental answers.

  • Task Formulation: The dataset contains 60 Tulu legal situations assigned primary and secondary LIST categories for classification evaluation.
  • Task Formulation: Queries were written in Latin, Kannada, Malayalam, and Tamil scripts, with English translations serving as a control.
  • Corpus: The retrieval corpus contains 18 Kannada Karnataka-court documents covering varied proceedings and document sizes.
  • Models: Three models were compared: Llama3 without official Kannada or Tulu support, and Sarvam and Hex-1 trained with Indic-language emphasis but not Tulu.
  • Experimental Conditions: Conditions varied query script, linguistic priming, Kannada-corpus RAG, and sparse versus extensive priming tiers.
  • Retrieval: RAG retrieved the top three chunks by cosine similarity after chunking legal passages into 800-character units.
  • Evaluation: Comprehension was assessed through LIST Hits@1/Hits@3, reasoning-trace verification, and Fisher or binomial tests with significance threshold 0.05.
  • Mitigation: A CRAG-style judge filtered passages for relevance before Llama3’s prompt in a 20-complaint mitigation test.

4 Findings

Kannada-script Tulu was understood above baseline by all three models, with further directional gains under full priming. RAG effects were mixed: Llama3 degraded sharply, while query language produced the only consistent cross-model RAG difference.

  • Bare comprehension: Kannada-script bare comprehension exceeded chance and majority-class baselines for Llama3, Hex-1, and Sarvam.Hits@1 was 46.7% for Llama3, 41.7% for Hex-1, and 48.3% for Sarvam; p < .001 in every case.
  • Priming: Full-tier priming showed a further comprehension increase for every model using Kannada script.The passage describes this as a trend rather than reporting significance for each comparison.
  • RAG effects: Llama3’s RAG performance fell below its no-RAG ceiling, unlike Hex-1 and Sarvam.Llama3 scored 23.3% with RAG and full priming versus 60.0% for its no-RAG full-priming ceiling, p < .001; Hex-1 and Sarvam showed no significant degradation.
  • Query language: English-gloss queries under RAG significantly outperformed Kannada-script Tulu queries under identical retrieval for all three models.Fisher’s exact tests gave p < .001 in every comparison.

5 Discussion

Models showed preliminary Tulu comprehension without extensive training, but performance depended on script and model. RAG produced mixed outcomes, with failures linked to knowledge conflict expressed as fact substitution or confabulation.

  • Comprehension Transfers Without RAG: Condition E slightly improved Llama3 and Hex-1 over Condition H, but neither change was significant, while Sarvam declined slightly.Llama3 changed from 46.7% to 51.7% and Hex-1 from 41.7% to 45.0%; Sarvam changed from 48.3% to 46.7%.
  • Comprehension Transfers Without RAG: Sparse structural priming produced modest, non-significant accuracy changes across Kannada, Tamil, and Malayalam scripts.Kannada showed small gains for all models, while Tamil and Malayalam were flat or only weakly positive.
  • Comprehension Transfers Without RAG: Matching script, especially Kannada, appears to support Tulu parsing through related-language or broader Dravidian cues, though the mechanism remains unclear.The evidence favors script-specific assimilation as suggestive rather than statistically established.
  • RAG Causes a Large, Reproducible Collapse or No Significant Improvement: RAG reduced Llama3’s comprehension, whereas Hex-1 and Sarvam maintained scores similar to priming-only conditions.Llama3 scored 25.0% with RAG versus 26.7% or 23.3% in comparison conditions; Hex-1’s RAG score was 38.3% versus 45.0% with sparse priming.
  • RAG Causes a Large, Reproducible Collapse or No Significant Improvement: RAG failures reflected fact substitution or confabulation, including unrelated classifications despite retrieval of a relevant cheque-bounce dispute.Fact substitution treated retrieved details as query content, while confabulation introduced information absent from both query and passage.
  • Relation of RAG Inclusion and Script Across Model Performance: Script advantage and RAG robustness were correlated across models: Hex-1 and Sarvam retained Kannada benefits, while Llama3 did not significantly outperform Latin queries.The relationship is reported as a correlation, not a demonstrated causal mechanism.
  • Relation of RAG Inclusion and Script Across Model Performance: CRAG did not improve Kannada-script queries over unmitigated RAG, but English-translated complaints reached 60.0% Hits@1 without standard-threshold significance.Kannada-script queries achieved 25.0% Hits@1 versus 23.3% for full RAG; English-translated complaints reached 60.0%, with p = .054.

6 Conclusion

The study evaluates whether minimal priming and Kannada legal retrieval can scaffold Tulu legal complaint classification without heavy training. Models showed latent comprehension, but RAG affected models differently and could trigger content-level reasoning failures.

  • 6 Conclusion: The Kannada–Tulu setting tests cross-lingual legal access where the spoken language differs from the administrative language and may lack training resources.The study frames Kannada as the administrative language and Tulu as the spoken, extremely low-resource language.
  • 6 Conclusion: All three models showed relatively high comprehension without priming or RAG, while Kannada-script conditions generally improved with priming.The Kannada improvement was not statistically significant for any model.
  • 6 Conclusion: RAG confused Llama3 through fact substitution and confabulation, while Hex-1 and Sarvam were not significantly affected by retrieval.The conclusion links script effects and RAG effects as interrelated phenomena rather than independent ones.
  • 6 Conclusion: CRAG-style relevance verification helped Llama3 discern irrelevant passages for English queries, but this effect did not transfer to Kannada script.The mitigation was studied only on Llama3.
  • 6 Conclusion: Future work targets larger datasets, cross-Dravidian transfer, broader hallucination mitigation, and additional language pairs.These directions aim to investigate mechanisms and generalizability of low-training multilingual legal transfer.

Limitations

The pilot’s conclusions are constrained by a small query set, uncertain corpus provenance and review, and unequal model reasoning configurations.

  • Limitations: The final pilot used 60 queries, but many findings remained trends rather than statistically significant behavior requiring much larger samples.The authors identify expansion to hundreds or thousands of queries as a priority, while noting that large effects could still reach significance.
  • Limitations: Corpus documents were not thoroughly reviewed by a native speaker, leaving residual risk that hallucinations entered during corpus creation.The documents were scanned, but native-speaker semantic review was absent.
  • Limitations: Public corpus documents may have appeared in model training, potentially confounding comparisons between RAG and non-RAG conditions.The authors state that prior exposure cannot be verified.
  • Limitations: Sarvam and Hex-1 had reasoning modes disabled, unlike Llama3, so their responses were not produced under equivalent conditions.Cross-model reasoning comparisons therefore require caution.

Ethics Statement

The study uses hypothetical complaints rather than real individuals’ case-intake data, while its public court-order corpus was not anonymized. The authors acknowledge privacy-related limitations and recommend named-entity redaction for future work.

  • The 60 complaints describe realistic hypothetical scenarios but are not real individuals’ complaints or case-intake data.
  • Court orders were retrieved from ecourtsindia.com and used for document content and LIST-category information, not personal information.
  • The retrieved corpus was not anonymized, and the authors recommend named-entity redaction before future use.
  • The paper cautions that the pipeline is not deployment ready because hallucinations could misclassify clients and produce confidently incorrect reasoning.
  • The authors state that scripts and the 60 authored sentences will be accessible, while redistribution rights for legal-corpus papers remain unclear.

A.1 The 60-Sentence Query Set

The study evaluates 60 authored Tulu legal-complaint scenarios spanning LIST categories, presented in English, Romanized Tulu, and Indic-script forms. The query materials pair complaints with human-selected labels and multilingual renderings used for model classification.

  • The query set contains 60 Tulu legal-complaint sentences paired with human-selected primary and secondary LIST categories.
  • The examples cover domains including housing, consumer or money disputes, land and property, employment, torts, courts, estates, and identity.
  • The set includes complaint scenarios such as inheritance disputes, rent conflicts, broken services, unpaid wages, property encroachment, injuries, and court-access problems.
  • The worked prompt examples show models receiving a complaint, a human-selected label, a prediction, and a reasoning trace for evaluation.

C.3 RAG Content and Correct Classification Aligning

RAG sometimes aligned with correct classification when retrieved content matched the complaint, but relevant retrieval alone did not guarantee success. The examples contrast accurate use of on-topic passages with failures despite relevant evidence.

  • Sarvam correctly classified a dog-bite complaint as Torts/Individuals when retrieval supplied on-topic personal-injury content.
  • The same query and condition produced an unrelated Estates/Wills prediction from Llama3, showing model-level divergence despite shared retrieval.
  • The examples therefore distinguish content alignment from reliable downstream reasoning under RAG.
  • A retrieved ancestral-property inheritance passage aligned with Hex-1’s correct Estates/Wills prediction for a family-land complaint.
  • Retrieval relevance did not ensure accurate classification: Sarvam predicted Land/Property despite an on-topic financial or debt passage.

D Full Condition-Level Results

The full results are organized by script, priming tier, model, and RAG condition, with Hits@1 and Hits@3 reported across 19 conditions. Additional analyses test script effects, disagreement patterns, significance, and relevance verification.

  • Figure 3 and Table 7 report no-RAG comprehension accuracy by script and priming tier for all three models, with n = 60 per cell.
  • Figure 4 and Table 8 summarize Hits@1 and Hits@3 across all 19 model conditions, with n = 60 for each condition.
  • Table 9 groups Fisher’s exact and binomial tests into baseline, priming-effect, and RAG-effect comparisons.
  • Table 10 compares Kannada-script and Romanized-script Hits@1 across six priming and RAG tiers, using Fisher’s exact tests.
  • Table 11 measures which condition was uniquely correct when full or sparse no-RAG and RAG predictions diverged.
  • For 20 Llama3 Kannada-script D-condition queries, CRAG relevance verification yielded p = .054, compared with D_full = 23.3% and E_full = 60.0%.

E Court Cases Cited

Table 13 identifies the eight court cases named in Appendix C from the 18-document Kannada legal corpus, including their court, case, CNR, and order-date details.

  • Table 13 lists the eight court cases named in Appendix C from the 18-document Kannada legal corpus.
  • The cases are public lower-court filings retrieved from ecourtsindia.com.
  • The table records each case number, deciding court, eCourts CNR identifier, and order date.
Loading 2608.28645v1…