PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark

Moniruzzaman Mahadi, Abrar Mohammed Tanzim Alam, Sayma Siddika Monalisa, Mir Mohammad Asif Abdullah, Swakkhar Shatabda, Md Adnan Arefeen

arXiv:2608.30327v1cs.CL

TL;DR

The paper asks whether fine-tuning improves small models’ use of supplied law or merely changes measured answer accuracy. It separates scoring, retrieval, and model effects using constrained option scoring, cyclic rotations, and a supplied-law control, finding no detected increase in reliance on the governing provision after fine-tuning.

  • Problem

    Legal QA accuracy conflates retrieval, use of present law, and evaluation interpretation, making fine-tuning’s effect on supplied-law use unclear.

  • Method

    The study evaluates six instruction-tuned models and LoRA adapters with constrained option-letter scoring, four-order rotation, and controlled governing-provision removal.

  • Results

    Five of six reference models use supplied law under rotation-invariant control, but fine-tuning shows no detected increase in governing-provision reliance.

  • Takeaways & Limitations

    Legal adaptation claims require separating scoring method, retriever, and model failure before interpreting accuracy gains.

  • Takeaways & Limitations

    The benchmark covers 199 questions from two examination years in one jurisdiction, and five-candidate training remains untested.

Abstract

from arXiv · show

Fine-tuning can improve legal question-answering accuracy without improving how models use law supplied in context. We study this distinction in bilingual Bangladeshi legal QA, where observed errors can arise from answer scoring, retrieval, or failure to use relevant law. We construct a hierarchy-preserving statutory corpus, 2,165 reviewed bilingual fine-tuning examples, and a 150-item supplied-law control. We evaluate six instruction-tuned models: Llama-3.2-1B, Llama-3.2-3B, Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, and Gemma-4-E2B, with three LoRA seeds per model. To separate effects, we combine constrained option-letter scoring, cyclic option rotation, and controlled removal of the governing provision. On 398 Bar Council outputs, an exact-line parser attributes an accuracy gain of 50.0\% to the Qwen3.5-2B seed-42 adapter, whereas option scoring yields only $3.0\%$. For Gemma-4-E2B, the two scoring methods favor different systems. When the governing provision is guaranteed to be present, five of six reference models improve by $14.7\%-19.3\%$ under the four-order criterion. Removing that provision reduces accuracy by $8.0\%-15.3\%$ for models and by $13.8\%-14.9\%$ points for their adapters. However, difference-in differences estimates show no increase in reliance on the governing provision after fine-tuning. Results show that legal adaptation claims require separating scorer, retriever, and model effects. Our Code and data are available at https://anonymous.4open.science/r/bangladesh-legal-qa-11E3

1 Introduction

The study separates scoring, retrieval, and model-use failures in bilingual Bangladeshi legal QA to test whether fine-tuning improves actual use of supplied law.

  • Motivation: Wrong answers can reflect retrieval failure, failure to use present law, or evaluation misreading the model’s output.The study focuses on multiple-choice questions with Bangladeshi legal sections supplied as context.
  • Study design: The evaluation covers six instruction-tuned models, three LoRA adapters per model, and 2,165 reviewed bilingual training examples.Training examples pair questions with the Bangladeshi law section needed to answer them, and evaluation uses 2022–2023 Bar Council examinations in Bangla and machine-translated English.
  • Scoring diagnostic: 50.0% versus 3.0%: exact-line parsing reports a much larger Qwen3.5-2B seed-42 gain than next-token option scoring on 398 Bar Council rows.For Gemma-4-E2B, the two scoring methods favor different systems.
  • Evaluation controls: The primary scorer reads option-letter logits and requires correctness across all four cyclic option orders.This avoids generated-answer parsing and tests position-stable answer selection.
  • Supplied-law control: 14.7–19.3%: five of six reference models improve when the governing provision is guaranteed present under the four-order criterion.Removing that provision reduces accuracy for both reference models and adapters, but difference-in-differences detects no increased reliance after fine-tuning.

2 Related Work

Prior legal QA work establishes relevant benchmarks, retrieval challenges, and evaluation instability; this paper narrows the question to whether small models use supplied Bangladeshi law.

  • Bangladesh legal QA: MINA and a legal-help-forum study establish bilingual Bangladeshi legal QA settings but do not isolate the source of errors.The forum study also reports fabricated citations and unsafe advice identified by licensed legal professionals.
  • Legal benchmarks: LexGLUE and LegalBench provide broad legal reasoning benchmarks, whereas this study examines supplied statutory sections in Bangla and English.The setting is narrower than seven-dataset LexGLUE and 162-task LegalBench.
  • Retrieval: Retrieval research shows that legal-span precision, complex queries, and context coherence affect performance, motivating separation of context availability from context use.The supplied-law control guarantees the governing section is present without retrieval failure.
  • MCQ evaluation: Option order, option-ID priors, prompt format, and minor evaluation changes can substantially alter legal and general MCQ comparisons.Reported effects include 13%–85% shifts across option orders and leaderboard movement of up to eight places.
  • Evaluation controls: CircularEval and option-ID scoring jointly separate option-position and generated-text effects from model answer selection.The paper adopts both controls unchanged and combines them with supplied-law control.

3 Data and Evaluation Sets

The paper builds a hierarchy-preserving bilingual statutory corpus, reviews fine-tuning records, and evaluates original and translated Bar Council questions alongside a separately reviewed supplied-law control.

  • Corpus: The statutory corpus covers six Bangladesh Bar Council acts, selected schedules, and English text from the official law portal.Bangla text comes from published act collections, with hierarchy preserved from act to clause.
  • Fine-tuning data: 3,425 candidate records became 2,165 retained examples after two author review passes.The retained set contains 1,211 Bangla and 954 English records across single-hop, advanced-selection, and bar-exam-style types.
  • Training format: Final fine-tuning prompts provide one governing section, while evaluation prompts provide five sections including distractors.The training format therefore differs from the distractor-rich supplied-law evaluation setting.
  • Bar Council evaluation: The Bar Council set contains 199 unique questions evaluated in Bangla and English, yielding 398 question–language pairs without additional independent items from rotations or seeds.Cancelled item 2022-005 is excluded in both languages.
  • Translation limitation: English translations received no legal-expert review, and model-judge audits reduce concern about gross errors without establishing legal correctness.Removing flagged items changes estimates by at most 0.8% or 2.1% and does not change result signs.
  • Overlap checks: Training records exclude the evaluation questions, but overlap checks do not rule out shared legal content or pretraining exposure.The closest wording similarities are 0.717 character-level and 0.463 by shared five-character spans.
  • Supplied-law control: The supplied-law control uses 150 newly constructed MCQs with a governing section and four same-act distractors.Simple surface heuristics score 25%–37%, while independent legal reviewers confirm every answer key.

4 Method

The method controls scoring and option-position instability, compares matched reference and LoRA systems, and contrasts retrieved, absent-law, and supplied-law contexts.

  • Design goals: The evaluation separates scoring effects, use of a known-present legal section, and whether fine-tuning increases that section’s benefit.These are the study’s three targeted effects.
  • Context conditions: Training supplies one governing section, whereas evaluation supplies five candidate sections with four distractors.This tests transfer from clean-law training to distractor-rich evaluation rather than direct five-section training.
  • Scope: The benchmark measures whether statutory text changes answer selection, not legal reasoning or legal practice.Retrieval methods describe how sections are retrieved but do not guarantee that the governing section is present.
  • Scoring: Constrained option-letter scoring selects the highest-scoring A–D next token after FINAL ANSWER: (, avoiding generated-text parsing.The scorer follows option-ID scoring.
  • Option rotation: Strict consistency requires the correct answer under every cyclic option rotation.Random guessing scores (1/4)^4 = 0.39%, while always choosing one letter scores zero.
  • Model pairs: Each fine-tuned system adds an unmerged LoRA adapter to its matched reference checkpoint, with seeds 17, 42, and 73.Model revision, template, prompt, context, precision, runtime, and scorer remain identical within each pair.
  • Evaluation conditions: Evaluation compares no legal context, BM25 or dense retrieval, and a supplied-law control with a matched withheld governing section.Retrieved contexts are capped at 2,000 characters.
  • Scorer diagnostic: The scorer diagnostic compares exact, lenient, and constrained readings of free-form answers on the same 398 Bar Council rows.This isolates how scoring changes the measured fine-tuning effect.

5 Experimental Setup

The experiment separates general fine-tuning gains from context-specific law use by comparing controlled context conditions before and after LoRA adaptation. Evaluation uses strict, option-order-robust scoring on held-out Bar Council and supplied-law control questions.

  • Data and evaluation: 2,165 fine-tuning records are split into 1,728 training, 220 validation, and 217 internal test examples, while examination questions remain separate.Records sharing an act and section stay within the same split.
  • Evaluation design: Strict consistency requires the correct answer across all four option orders, while direct option-letter scoring avoids parsing generated answer lines.The study compares context benefits before and after fine-tuning using RefC and FTC accuracies.
  • Statistical analysis: Bootstrap intervals are descriptive comparisons, while Holm correction applies only to the paired McNemar tests.Confidence intervals use 10,000 question-grouped bootstrap samples.
  • Primary comparison: Context specificity is the difference between the context benefit after fine-tuning and the corresponding benefit before fine-tuning.Positive values mean context provides a larger benefit after fine-tuning.
  • Controlled conditions: The supplied-law control guarantees the governing section among five same-act sections, whereas Bar Council retrieval conditions do not guarantee that section.This design distinguishes context use from retrieval failure in the controlled setting.

6 Results

Results show that measured fine-tuning gains depend strongly on scoring, while supplied law helps most models when the governing provision is guaranteed present. Removing that provision lowers accuracy, but fine-tuning does not increase context specificity, and examination evidence remains retrieval-limited.

  • Scoring effects: Gemma-4-E2B changes from a +10.1% adapter advantage under exact-line parsing to a −2.5% disadvantage under direct scoring.For Qwen3.5-4B, the two gains differ by only 0.8%.
  • Supplied-law control: Five of six reference models gain 14.7–19.3% when the governing section is guaranteed present under strict consistency.All five confidence intervals exclude zero, and Qwen3.5-4B reaches 94.0%.
  • Supplied-law control: 12.0–22.8% gains persist on 92 high-overlap questions, although Llama-3.2-1B remains the exception at −2.2%.This post-hoc check tests whether lexical overlap explains the supplied-law effect.
  • Supplied-law control: 8.0–15.3% accuracy losses follow removal of the governing section, while adapter means fall by 13.8–14.9%.The pattern supports dependence on the governing section rather than merely other text from the same act.
  • Strict consistency: At one fixed order, supplied law helps all six models, but strict-consistency effects are 1.3–8.0% larger for the five responsive models.Because fixed-order accuracy also rises, position stability alone cannot explain the strict gain.
  • Strict consistency: Llama-3.2-1B shifts from a +6.7% fixed-order effect to −1.3% under strict consistency, showing that its gain does not survive option movement.Its strict interval is [−7.3%, +4.7%].
  • Fine-tuning and context specificity: Fine-tuning gains without law range from 13.3% and 16.2% for the two weakest reference models to losses of 7.6% and 11.1% for the strongest two.The pattern is descriptive because only six models are evaluated.

7 Discussion

The discussion separates scoring, retrieval, and model-use effects, showing that apparent fine-tuning gains can change under stricter evaluation while supplied law helps models without increased reliance after fine-tuning.

  • The apparent fine-tuning effect changes once scorer, retriever, and model failure are separated.
  • Constrained option scoring removes formatting effects, while four-way option rotation tests whether gains depend on answer position.These controls measure answer selection rather than explanation quality.
  • Five reference models improve when the governing provision is present, and both models and adapters decline when it is removed.The supplied-law control eliminates retrieval uncertainty; Llama-3.2-1B is the exception because its fixed-order gain disappears under rotation.
  • Fine-tuning changes answer selection and preserves dependence on the governing provision, but does not detectably increase that dependence.The transferable measurement sequence separates scorer, retriever, and model failure before interpreting legal adaptation.

8 Conclusion

The conclusion finds that supplied Bangladeshi law improves rotation-invariant answer selection, while fine-tuning changes answers without a detected increase in reliance on the governing provision.

  • 14.7–19.3%: the governing provision improves accuracy for five of six instruction-tuned checkpoints under a rotation-invariant control.Removing it reduces accuracy by 8.0–15.3% for responsive reference models and 13.8–14.9% for their adapters.
  • +50.0% versus +3.0%: exact-line parsing and matched constrained scoring produce sharply different Qwen3.5-2B seed-42 gains.The scoring methods also reverse one model comparison.
  • Single-provision fine-tuning changes answers without a detected increase in provision dependence.Five-candidate training remains untested, so legal adaptation claims require separating scoring, retriever, and model failure.

Limitations

The study’s claims are bounded by a small, repeated benchmark, untested training conditions, non-scaling model comparisons, possible contamination, and incomplete language and legal-expert validation.

  • 199 questions from two examination years in one jurisdiction bound the benchmark’s scope.Repeated Bangla, English, context, rotation, and seed conditions do not add independent test items; MCQ selection proxies statutory knowledge rather than legal reasoning or practice.
  • Five-candidate training remains untested because training supplies one governing provision while evaluation supplies five candidate sections.The null result does not extend to that intervention.
  • The six checkpoints do not form a scaling study because they differ in family, size, release, and precision.Relations between starting accuracy and fine-tuning effects are descriptive.
  • Contamination cannot be ruled out, including public pretraining exposure, overlap with a retained training item, and repeated development on the same questions.
  • English examination translations received no legal-expert review, and no lawyer exhaustively reviewed the 2,165 fine-tuning records.Two legally trained reviewers confirmed retained control-item answer keys but validated rather than authored those items.
  • Fourteen of 75 Bangla control items contain a majority-English passage because schedules are available only in English.This limits language-specific interpretation despite identical contexts for matched systems.
  • Bar Council questions lack expert governing-provision labels, preventing oracle retrieval, Recall@k, and retrieval-versus-context-use separation on those questions.The supplied-law control removes this uncertainty only for its 150 items.
  • The Gemma-4-E2B sign reversal is less general because evaluation-method effects and over-adaptation cannot be separated for that model.A format example at inference time was not tested.

Ethics Statement

The paper frames these systems as research artifacts rather than legal advisors and identifies practical risks from evaluation design, corpus construction, language coverage, and automated assistance.

  • Bar-exam performance is not evidence of competence to advise a person.The task omits drafting, citation verification, client facts, procedural judgment, and professional responsibility.
  • The release contains public statutes and examinations but no client records or personal data.Rights are claimed only for generated materials, annotations, translations, metadata, and arrangement where held.
  • A parser that rewards format compliance can over-state legal-QA progress.Separating answer selection from formatting and model failure from retrieval failure limits inference from the reported results.
  • The authors used ChatGPT and Claude for manuscript fluency and clarity while retaining human responsibility for the paper’s ideas, analyses, claims, and accuracy.
  • Schedules are available only in English, so some Bangla control prompts contain majority-English passages.This limits language-specific interpretation.
  • Hierarchy-preserving statutory conversion keeps sections, subsections, clauses, explanations, and illustrations distinct for supplying provisions as context.Section 81 illustrates how this avoids treating illustration numbering as clause noise.
  • The evaluation varies governing-provision position across candidates, but the observed distribution is not approximately uniform.Across 712 advanced-selection records, positions range from 5.1% to 50.4%.
  • Human review retained 2,165 records after checking answers against cited law and removing repetitive, cross-act, or incomplete records.The earliest preserved pool contained 3,425 records, with 1,260 removed.

C Training Details

The study fixes a bilingual LoRA training and evaluation recipe across six models, with controls for scoring, option order, retrieval, and Gemma training defects.

  • Training recipe: Every model uses the same LoRA recipe, including rank and alpha 16, effective batch 16, at most two epochs, and validation-loss early stopping.All models select learning rate 2e−4 from the prespecified grid, without widening it.
  • Evaluation protocol: Evaluation uses constrained next-token scoring over A, B, C, and D rather than generated-text parsing, with a fixed FINAL ANSWER protocol for option questions.Direct questions retain prose targets, while MCQs are scored from the partial answer stem.
  • Data and runs: The training data are split by normalized act and section groups, yielding 1,728 training, 220 validation, and 217 internal-test records without statutory overlap.Each model uses seeds 17, 42, and 73 on one 16 GB T4.
  • Gemma corrections: Gemma-4-E2B required seven training attempts, but only the repaired second attempt was reported after removing multimodal LoRA targets and masking prompt tokens from target loss.The reported run passed checks for language-only adaptation, target-loss masking, and language-matched shuffled controls.
  • Training diagnostics: The pre-repair Gemma run is not a replication because it used a different denominator, predates the audit, and lacks retained per-item outputs for recomputation.A seed-42 crash was repaired by retraining to 216 steps; the strict-consistency score remained unchanged across executions.
  • Evaluation controls: Cyclic option rotation requires correctness in all four positions, while the supplied-law control guarantees the governing section is present and contrasts it with withheld sections.The withheld control uses sections from the question’s language pool that appear in neither BM25 nor dense retrieval hits.

G Additional Results

Additional analyses show that parser choices, retriever choice, translation checks, and quoting behavior materially affect interpretation of legal QA results.

  • Statistical results: 12/12, 3/12, 7/12, 0/12, 2/12, and 0/12 tests are Holm-significant for Llama3.2-1B, Llama-3.2-3B, Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, and Gemma-4-E2B.The released JSON provides per-seed differences, adjusted p values, and bootstrap intervals for all 72 comparisons.
  • Parser robustness: +11.3%, +10.1%, and +10.8% are Gemma-4-E2B parser gains for seeds 17, 42, and 73, while constrained gains are −1.5%, −2.5%, and −1.8%.The parser effect therefore persists across seeds but differs sharply from constrained scoring.
  • Parser robustness: +52.8%, +50.0%, and +50.8% are Qwen3.5-2B exact-line gains across seeds, compared with constrained gains of +5.5%, +3.0%, and +3.8%.These values show stable inflation under exact-line parsing across the three seeds.
  • Quoted-text analysis: 56/150 (37.3%) control items are answered by choosing the option that most quotes the supplied law, while reference law effects remain varied after removing ties.The post-hoc subset contains 92 items where a distractor has strictly greater 5-gram coverage than the correct option.
  • Translation audit: 14.1% and 1.7% of never-seeded items are flagged by judges A and B, respectively, with answer preservation of 94.9% and 99.4%.After restoring seeded items, judge A flags 29 of 199 items and judge B flags 4 of 198.
  • Translation robustness: Removing 8 answer-changing items shifts every English difference in differences by at most 0.8%, while removing all 25 flagged items shifts them by at most 2.1%; neither changes a sign.All 29 flags occur in 2022, yet 2022 is the smaller year for five of six models.

J Reproducibility

The release supports numerical reproduction through structured artifacts, hashes, scripts, and provenance checks, while documenting two remaining provenance gaps.

  • Reproduction command: python 07-analysis/scripts/reproduce.py runs 21 bundled analyses in about one minute without a GPU or network.The list option prints the claim map connecting each script to its claim and regenerated folder.
  • Diagnostic tables: Table 12 decomposes accuracy into provision = law minus withheld and same-act = withheld minus none, while Table 13 reports item counts for Table 3 cells at or below 5%.Strict-consistency counts require correctness under all four rotations, and fine-tuned counts are three-seed means.
  • Release contents: The release includes the bilingual corpus, QA set, benchmark, retrieval bundle, experiment contract, adapter manifests, per-item outputs, notebooks, run manifests, and analysis scripts.Benchmark directories record revisions, prompt and contract hashes, scorer, package versions, hardware, memory, and result-file SHA-256 values.
  • Manifest verification: The experiment manifests make the frozen contract and prompt checkable across eight checkpoint executions, including six reported and two superseded runs.Individual weight-file SHA-256 values provide durable provenance for model weights and manifest-hash-verified benchmark outputs.
  • Limitations: Two provenance gaps remain: missing RUN_COMPLETE markers for two Qwen benchmark directories and unreproduced directory-tree hashes for corrected Gemma adapters despite matching file inventories.The Qwen terminal manifests and result hashes are verified.
Loading 2608.30327v1…