PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models

Antoine Louis, Gijs van Dijck, Gerasimos Spanakis

arXiv:2309.17050v1cs.CL

TL;DR

Many people lack knowledge of legal rights and processes, while existing legal QA often targets narrow domains or gives brief answers. The paper pairs a retrieve-then-read approach with LLeQA, a French dataset of expert-annotated statutory questions. The framework performs promisingly on automatic metrics, but qualitative analysis finds frequent inaccuracies and hallucinations.

  • Problem

    Existing legal question answering often focuses on specialized domains or brief answers, despite the need for accessible legal guidance among people with limited knowledge of rights and processes.

    Source: §1 · p. 1

  • Method

    The paper evaluates a retrieve-then-read system that retrieves statutory provisions and uses an adapted instruction-tuned LLM to generate long-form answers, supported by the LLeQA dataset.

    Source: §1 · pp. 1–2

  • Results

    The framework shows promising automatic-metric performance, yet qualitative analysis finds that answers frequently contain inaccuracies and can fabricate facts.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    LLeQA offers a foundation for research on interpretable long-form legal QA, while the findings point to a need for better modeling and evaluation.

    Source: §6 · p. 7

    The framework remains vulnerable to hallucinations in generated answers and rationales, and conventional automatic metrics may not accurately reflect answer quality.

    Source: §6 · p. 7

Abstract

from arXiv · show

Many individuals are likely to face a legal dispute at some point in their lives, but their lack of understanding of how to navigate these complex issues often renders them vulnerable. The advancement of natural language processing opens new avenues for bridging this legal literacy gap through the development of automated legal aid systems. However, existing legal question answering (LQA) approaches often suffer from a narrow scope, being either confined to specific legal domains or limited to brief, uninformative responses. In this work, we propose an end-to-end methodology designed to generate long-form answers to any statutory law questions, utilizing a "retrieve-then-read" pipeline. To support this approach, we introduce and release the Long-form Legal Question Answering (LLeQA) dataset, comprising 1,868 expert-annotated legal questions in the French language, complete with detailed answers rooted in pertinent legal provisions. Our experimental results demonstrate promising performance on automatic evaluation metrics, but a qualitative analysis uncovers areas for refinement. As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains. We publicly release our code, data, and models.

1 Introduction

Limited public understanding of legal rights and costly expert assistance leave many people vulnerable, while existing legal QA often covers narrow domains or gives only brief answers. The paper presents a retrieve-then-read approach and the LLeQA dataset, reporting strong automatic-metric performance alongside qualitative concerns about factual accuracy.

1 Introduction

  • Limited knowledge of rights and legal processes, together with the high cost of expert help, can leave people unable to access useful legal guidance.The introduction cites estimates of 1.5 billion people with unresolved legal challenges and 4.5 billion potentially excluded from legal protections.Source: §1 · p. 1
  • Prior legal QA work often targets specialized domains or produces brief answers, leaving long-form answers across statutory subjects comparatively under-served.The paper frames legal QA as a possible source of affordable, expert-like assistance.Source: §1 · p. 1
  • The proposed end-to-end system retrieves relevant legislative articles with a lightweight bi-encoder, then uses an adapted instruction-tuned LLM to produce interpretable long-form answers.Reader adaptation uses in-context learning or parameter-efficient fine-tuning.Source: §1 · p. 1
  • Qualitative review found that answers can be syntactically correct and topically relevant yet frequently contain inaccuracies, despite promising automatic-metric performance.The authors identify substantial room to improve both modeling and evaluation.Source: §1 · p. 2
  • LLeQA provides 1,868 French legal questions with expert-written answers and references, drawn from a corpus of 27,942 statutory articles.The dataset adds 760 questions and 5,308 articles to BSARD, with question taxonomy, jurisdiction, and paragraph-level annotations.Source: §1 · pp. 1–2

2 Related Work

Prior legal QA datasets address varied tasks, but many focus on a specific domain, document type, or short-answer format. The related work situates this research within knowledge-grounded QA and approaches to generating or extracting rationales.

Legal question answering

Knowledge-grounded question answering

  • Knowledge-grounded QA commonly retrieves documents, web pages, or knowledge-base entries before a reader predicts an answer from supporting materials.The passage likens this open-book paradigm to query-based multi-document summarization, which can be extractive or abstractive.Source: §2 · p. 2

Rationale generation

  • Rationale research includes abstractive explanations, predefined evidence markers, extraction of rationales from inputs, and saliency maps that highlight influential tokens.Examples span commonsense reasoning and natural-language inference.Source: §2 · p. 2

3 The LLeQA Dataset

LLeQA combines expert-authored French legal question-answer pairs with a broad Belgian statutory corpus and paragraph-level relevance annotations. Its analysis describes the dataset’s breadth and evidence demands, while noting that synthetic paragraph annotations are imperfect.

3.2 Dataset Analysis

  • LLeQA is presented as the only dataset covering any statutory-law subject with expert-derived long-form answers, which are an order of magnitude longer than those in other public legal QA datasets.Its detailed answers often discuss scenarios contingent on circumstances such as age, employment, or marital status.Source: Table 1 · p. 3, §3.2 · p. 3

3.1 Dataset Construction

  • From about 2,550 collected questions, filtering and deduplication yield 1,868 question-answer pairs with legal references.Questions come from a Belgian nonprofit’s website and include answers written by experienced jurists.Source: §3.1 · pp. 2–3
  • The corpus contains 27,942 Belgian statutory articles, assembled from 23,759 articles across 35 codes and 4,183 additional articles from 34 legal acts.Articles are cleaned, assigned identifiers, matched to question references, and split into paragraphs for rationale extraction.Source: §3.1 · p. 3
  • Because only 10.4% of questions initially had paragraph-level references, an LLM generates paragraph relevance signals for the remaining samples.The model considers paragraphs from relevant articles and returns identifiers it judges to contribute to the answer, or “None.”Source: §3.1 · p. 3
  • The test set uses expert-annotated paragraph references, while remaining samples are randomly divided into training and development sets at a 10/90/10 split, respectively.Source: §3.1 · p. 3

3.2 Dataset Analysis

  • Housing and healthcare are the largest question topics, together making up almost half of LLeQA; family, work, and immigration account for over a third.The question-word analysis identifies “can,” “how,” “what,” and “must” as the most frequent, corresponding mainly to permissions, procedures, definitions, and obligations.Source: §3.2 · pp. 3–4, Table 2 · p. 4
  • About 80% of questions link to fewer than five articles, with a median of two, while combined evidence averages 1,857 words per question.Only 8% of corpus articles are referenced, and nearly half of referenced articles come from five statutes.Source: §3.2 · p. 4
  • Synthetic paragraph annotation reaches 47.5% F1, compared with 15.3% for random selection and 27.2% for always selecting the first paragraph.The authors characterize its quality as below expert annotation, while considering it potentially useful for in-context learning.Source: §3.2 · p. 4

4 Method

The framework retrieves a small set of statutory provisions before generating an answer conditioned on them. Its retriever ranks provisions for a question, while its generator supports extractive rationales, in-context learning, and parameter-efficient fine-tuning.

4 Method

  • The retrieve-then-read pipeline selects a small subset of statutory articles, then conditions answer generation on the articles returned.The retriever orders provisions by relevance; the generator produces a comprehensive answer using the question and supporting articles as context.Source: §4 · p. 4, §4.1 · p. 4

4.1 Retriever

  • The retriever uses a shared-encoder bi-encoder to embed questions and provisions, score their similarity, and rank relevant provisions above irrelevant ones.Training uses contrastive optimization with irrelevant provisions sampled both in-batch and as BM25 hard negatives.Source: §4.1 · pp. 4–5

4.2 Generator

  • The generator conditions an autoregressive language model on the question and retrieved legal articles to produce a comprehensive answer.For interpretability, it generates evidence paragraph markers rather than free-form rationales.Source: §4.2 · pp. 5–6
  • In-context learning ranges from zero-shot instructions to one- or few-shot prompts with demonstrations selected dynamically by question similarity.The retrieved provisions are the top five for the test question; few-shot prompts include as many demonstrations as fit in the context window.Source: §4.2 · p. 5
  • The fine-tuned generator uses QLoRA, quantizing the pretrained model to 4-bit, freezing its parameters, and training low-rank adapters.The target answer is optimized under a language-modeling objective conditioned on the question and relevant provisions.Source: §4.2 · pp. 5–6
  • Dynamic NTK-aware scaling extends the context window by nonlinearly rescaling positions while retaining their original values within the pretrained window.The paper reports preliminary results suggesting this mitigates perplexity degradation beyond the maximum window size without additional fine-tuning.Source: §4.2 · p. 6

5 Experiments

The experiments compare retrieval and generation setups, then assess both automatic metrics and qualitative answer quality. The results show retrieval gains and stronger generation after task-specific fine-tuning, but also expose retrieval bottlenecks and hallucinations.

5.1 Experimental Setup

  • The setup compares a CamemBERT dense retriever with BM25 and mE5, and tests four LLaMA-derived instruction-tuned readers in 7B variants.The models use context windows up to 8192 tokens for in-context learning and 4096 for fine-tuning.Source: §5.1 · p. 6

5.2 Automatic Evaluation

  • Evaluation covers retrieval recall at small cutoffs and mean reciprocal rank, answer quality with METEOR, and rationale extraction with paragraph-marker F1.The authors defer human evaluation because it is expensive and noisy, and note that lexical-overlap metrics such as ROUGE inadequately assess open-ended answers.Source: §5.2 · pp. 6–7
  • Fine-tuning consistently improves performance across generators, while fewer than half of relevant articles appear in the top five and rationale extraction is near zero for most LLMs.WizardLM and Vicuna show high overlap with ground-truth answers; demonstrations generally help, but adding more does not significantly change performance.Source: §5.2 · p. 7, Table 4 · p. 7

5.3 Qualitative Analysis

  • WizardLM and Vicuna produce polished French answers but can fabricate facts, dates, sources, and conditions; T ¨ULU tends toward concise answers and Guanaco toward repetitive or convoluted output.T ¨ULU’s brevity is associated with its instruction-tuning on short completions, while repetition penalties may be ineffective when specialized topics require repeated terminology.Source: §5.3 · p. 7, Table 5 · p. 7

6 Conclusion

The paper introduces LLeQA and evaluates a retrieve-then-read approach for long-form legal answers with interpretable justifications. It identifies hallucinations and imperfect automatic metrics as continuing limitations, and restricts the dataset to research use because premature deployment risks harm to lay users.

6 Conclusion

  • LLeQA is an expert-annotated dataset for developing systems that generate comprehensive legal answers with interpretable justifications.The authors present it as a foundation for interpretable long-form legal question answering and improved access to legal help.Source: §6 · p. 7
  • The framework remains vulnerable to hallucinations in answers and rationales, while conventional automatic metrics may misrepresent answer quality.The authors identify both challenges as avenues for future work.Source: §6 · p. 7
  • The dataset is restricted to research use because premature deployment could lead laypeople to rely on guidance and worsen their circumstances.Source: §6 · p. 7

A Details of LLeQA

The appendix documents LLeQA with a dataset nutrition label and sample questions. It also describes synthetic rationale annotation experiments and the budget-driven choice of ChatGPT for training-set annotations.

Documentation

  • The dataset nutrition label provides a standardized, concise overview of LLeQA characteristics to help assess its suitability for AI development.Source: A Details of LLeQA · pp. 11–12

Dataset examples

  • Dataset examples show systems generating long-form answers from short legal questions and supplied statutory articles, with rationales identifying supporting paragraphs.Source: A Details of LLeQA · p. 12

Automatic annotation

  • GPT-4 achieved 65.5% F1 on synthetic rationale annotation, but its estimated training-set annotation cost was about €750—ten times ChatGPT’s—so the authors used ChatGPT.Source: A Details of LLeQA · p. 12

B Evaluation Metrics

The section defines retrieval, generation-quality, and rationale-accuracy metrics. Retrieval uses cutoff-based recall and reciprocal rank; generation quality uses METEOR, while rationale accuracy compares predicted paragraph markers with gold markers.

B Evaluation Metrics

  • Retrieval is assessed with macro-averaged recall at cutoff k and mean reciprocal rank over questions.Recall measures the proportion of relevant provisions in the retrieved filter set; reciprocal rank is the inverse rank of the first relevant provision.Source: B Evaluation Metrics · p. 12
  • METEOR evaluates generated answers against references through word alignment and a recall-weighted harmonic mean of unigram precision and recall.It aligns exact words, then stemmed words, synonyms, and paraphrases; an n-gram penalty can reduce the F-score by up to 0.5 when there are no n-gram matches.Source: B Evaluation Metrics · pp. 12–13
  • Rationale accuracy uses F1 to compare predicted paragraph markers with ground-truth markers.Precision is the share of predicted markers that are correct, and recall is the share of ground-truth markers that are correctly predicted.Source: B Evaluation Metrics · p. 13

C Qualitative Results

The qualitative-results materials provide an example of WizardLM answers across configurations and a template for paragraph-level rationale extraction. They also identify a table of zero-shot rationale-extraction results and annotation costs.

C Qualitative Results

  • Table 8 presents an example of WizardLM long-form answers across different settings.The example is drawn from a question in the LLeQA test set.Source: C Qualitative Results · p. 13, Table 8 · p. 14
  • Figure 2 gives a template for extracting paragraph-level rationales.Source: Figure 2 · p. 14
  • Table 9 reports zero-shot rationale-extraction results on the LLeQA test set and average annotation cost per 10K samples for API-accessible proprietary models.Source: Table 9 · p. 14

Related papers

Loading 2309.17050v1…