Source-linked AI summary
CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering
Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, Bruno Fleisch
TL;DR
Legal QA needs evidence to validate generated answers, yet legal QA datasets have often been small and manually curated. CBR-RAG retrieves relevant cases to enrich LLM prompts and evaluates embedding representations and similarity methods; its generated responses outperform baseline models in similarity to ground truth.
Problem
Method
CBR-RAG uses CBR retrieval, indexing, and similarity knowledge to add relevant cases to LLM prompts and compares embedding representations and similarity methods.
Source: §1 · p. 2Results
CBR-RAG responses outperform baseline models in similarity to ground truth.
Source: §7 · p. 14Takeaways & Limitations
Case retrieval brings clear performance benefits, but choosing an appropriate embedding for case representation is key.
Source: §7 · p. 15None of the evaluated embedding methods, including LegalBERT, was fine-tuned to the ALQA-specific legal corpus.
Source: §7 · p. 15
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM. We introduce CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases. This integration augments the original LLM query, providing a richer prompt. We present an evaluation of CBR-RAG, and examine different representations (i.e. general and domain-specific embeddings) and methods of comparison (i.e. inter, intra and hybrid similarity) on the task of legal question-answering. Our results indicate that the context provided by CBR's case reuse enforces similarity between relevant components of the questions and the evidence base leading to significant improvements in the quality of generated answers.
1 Introduction
RAG supplies external knowledge to LLMs for knowledge-intensive tasks, but legal QA requires evidence that can validate generated answers. This paper applies CBR to retrieval for legal QA and compares embedding representations and similarity methods.
1 Introduction
- Legal QA outputs need validation against evidence, while LLM hallucination and factual-faithfulness concerns complicate deployment.RAG can present factual data to guide responses, but moderation of generated content remains a concern.Source: §1 · p. 2
- CBR can organize prior solutions as a knowledge base and help match queries with relevant cases, drawing on practices or regulations.Prior ensemble CBR-neural systems reported better factual correctness than purely neural methods.Source: §1 · p. 2
- The paper formalizes CBR's role in forming context for RAG and compares retrieval representations and similarity methods for legal QA.The comparison covers general and domain-specific embeddings and inter, intra, and hybrid similarity.Source: §1 · p. 2
- The authors report results for a generative legal QA application and identify CBR-RAG as promising for knowledge-reliant generation.The paper is organized around related work, CBR-RAG, retrieval methods, encoders, evaluation, and conclusions.Source: §1 · p. 2
2 Related Work
Prior legal NLP work includes CBR case representations and transformer-based legal models, while legal question answering remains relatively under-served. The paper situates CBR-RAG at the intersection of these lines of work.
2 Related Work
- Legal CBR systems have used factors, decision trees, and association rules to index or compare cases, with later work adopting transformer embeddings.Examples include HYPO, CATO, and SMILE.Source: §2 · pp. 2–3
- Legal LLMs face black-box and factual-faithfulness concerns, while RAG supplies factual data but does not use CBR's varying matching strategies across content segments.The paper draws on CBR-IR integration that retrieves on-point cases to guide search and browsing.Source: §2 · p. 3
- Existing legal QA datasets were mostly small and manually curated; ALQA offers over 2,100 question-answer-snippet triplets synthesized from an Australian legal corpus.The LegalBench rule-qa task, for example, contains 50 question-answer pairs.Source: §2 · p. 3
3 CBR-RAG: Using CBR to form context for LLMs
CBR-RAG retrieves prior cases and adds their content to the LLM query as context. Its casebase uses legal QA pairs with supporting snippets, and the casebase's sparse legal-act references motivate neural embeddings for indexing and weighted similarity.
3 CBR-RAG: Using CBR to form context for LLMs
- CBR-RAG combines CBR's retrieve-only stage, indexing vocabulary, and similarity knowledge containers to retrieve cases as context for an LLM query.The retrieved content augments the original query into a contextually enriched prompt.Source: §3 · pp. 3–4
- The system represents a prompt as p = (Q, C), where Q is the user's question and C is contextual text that guides answer generation.The generated response is denoted A, and the generative model G.Source: §3 · p. 4
3.1 Casebase
- The casebase contains 2,124 LLM-generated Australian legal QA pairs, each corroborated by a supporting snippet from a legal document.The dataset is based on the Australian Open Legal Corpus.Source: §3.1 · p. 4
- An example case pairs a legal question and answer with support text that identifies the case, action, location, outcome, and legal concept.The example concerns the interpretation of reasonable grounds for a warrantless search.Source: §3.1 · p. 5
- The casebase's 785 unique legal acts are sparse: 1,183 cases (57%) have no act reference, and only 44 acts occur more than once.The Federal Court Rules is the most frequently mentioned act.Source: §3.1 · p. 5
- Because act-only indexing is unsuitable for this casebase, neural embeddings provide its indexing vocabulary, with weighted similarity contributing to similarity knowledge.Figure 2 lists the ten most frequent acts and shows the act-frequency distribution.Source: §3.1 · p. 5, Figure 2 · p. 5
4 Representation and Similarity
Section 4 represents each legal case with its question, answer, evidence support, and entities, then describes dual embeddings and intra-, inter-, and hybrid strategies for retrieving relevant context.
4 Representation and Similarity
- Each case is represented as a question, answer, evidence support, and entities extracted from that support.The support supplies context for factually accurate answer generation, while the original corpus retains a link to the full document.Source: §4 · p. 6
4.1 Representation
- Dual embeddings distinguish intra-attribute matching, such as question-to-question comparison, from inter-attribute retrieval across questions, support text, or entities.Intra embeddings target semantic similarity between attributes; inter embeddings support finding relevant content without requiring like attributes.Source: §4.1 · pp. 6–7
4.2 Case Retrieval
- Intra retrieval matches a query embedding to case questions, while inter retrieval searches for relevant cases using support snippets or entities without requiring like-attribute matches.The retrieved cases are used to augment the prompt context.Source: §4.2 · p. 7
- Hybrid retrieval combines intra- and inter-embedding comparisons of the query in a weighted approach to retrieve cases.The hybrid approach uses both query representations when selecting the top-k cases.Source: §4.2 · pp. 7–8
5 Embedding models
The paper explores general-purpose BERT and AnglEBERT alongside legal-pretrained LegalBERT, and discusses using these models to generate f and g representations.
5.1 BERT
- BERT learns generalisable text embeddings through bidirectional context and self-supervised masked-language modeling and next sentence prediction.Its pre-training uses a combination of large general-purpose text datasets totaling 3.3B words.Source: §5.1 · pp. 8–9
5.2 LegalBERT Trained on General Legal Data
- LegalBERT is further pre-trained on 12 GB of diverse English legal documents to embed domain knowledge in its representations.The documents include UK, EU, and US legislation and case law.Source: §5.2 · p. 9
5.3 BERT with AnglE Embeddings
- AnglE learns embeddings by contrastively matching positive and negative text pairs, but domain-specific AnglEBERT requires supervised paired examples that expert labeling may make prohibitive.AnglE’s loss functions are designed to address vanishing gradients in cosine-similarity comparisons, particularly at similarity extremes.Source: §5.3 · p. 9
5.4 Dual-embedding Case Representation with AnglE
- Inter retrieval uses Cue(Q) to orient query embeddings toward attributes beyond questions; intra retrieval uses an empty prompt and processes Q directly as f(Q).The cue is “Represent this sentence for searching relevant passages: ” followed by Q.Source: §5.4 · pp. 9–10
6 Evaluation
The evaluation tested retrieval representations and weighting schemes on ALQA, then measured answer similarity against reference answers. Hybrid AnglEBERT with three retrieved cases performed best in generation, while retrieval quality and dataset construction informed the setup.
6 Evaluation
- The evaluation compared BERT, LegalBERT, and AnglEBERT embeddings across question-only [1,0,0], support-only [0,1,0], entity-only [0,0,1], and hybrid [0.25,0.40,0.35] retrieval, using an LLM without case retrieval (No-RAG) as baseline.Source: §6 · pp. 10–11
6.1 Legal QA Dataset Analysis
- Random manual checks screened dataset questions and answers for anomalies; similarities were mostly 0.7–0.8, with denser clustering for LegalBERT.The similarity distributions were presented for intra- and inter-embeddings in Figure 4; the authors considered them unlike sparse feature-engineering distributions.Source: §6.1 · pp. 11–12, Figure 4 · p. 11
- The case base contained 2,084 cases, and 32 test questions were retained after generating 35 new pairs that required information from two parent cases.Forty cases were removed from the original 2,124 for offensive content; generated test cases were manually reviewed and pruned.Source: §6.1 · pp. 12–13
6.2 Retrieval Analysis
- Hybrid AnglEBERT with weights [0.25, 0.40, 0.35] performed best for retrieval at k = 3, which was selected for generation experiments alongside k = 1.Retrieval used F1-score for Retrieval@k, shown in Figure 5.Source: §6.2 · p. 13, Figure 5 · p. 13
6.3 Generation Results
- Generation quality was measured by cosine similarity between Mistral answer embeddings and reference answers, across six retrieval algorithms, k = 1 and k = 3, and No-RAG.Higher cosine similarity was treated as better agreement with the expected answer.Source: §6.3 · pp. 13–14
- 1.94% average performance increase: Hybrid AnglEBERT with k = 3 achieved the best semantic similarity, and all hybrid variants outperformed No-RAG.The comparison used full cases as RAG context; Table 4 reports cosine scores for hybrid algorithms.Source: Table 4 · p. 14, §6.3 · p. 14
- Hybrid AnglEBERT k = 3 significantly outperformed No-RAG and hybrid LegalBERT k = 3 at 95% confidence, and hybrid BERT k = 1 at 90%.Significance was assessed using one-tailed paired t-tests.Source: §6.3 · p. 14
7 Conclusions
The paper concludes that CBR-RAG improves LLM output by adding supporting information from prior cases, while the choice of embedding matters. It identifies expert evaluation, fine-tuning, and improved case aggregation as further work.
7 Conclusions
- CBR-RAG responses were more similar to ground truth than baseline responses, and the authors conclude case retrieval offers performance benefits.They also emphasize selecting an appropriate embedding for case representation.Source: §7 · pp. 14–15
- AnglEBERT performed best, which the authors suggest reflects the importance of contrastive embedding optimization relative to standard masked-language-model training.None of the evaluated embeddings was fine-tuned on the ALQA-specific legal corpus.Source: §7 · p. 15
- Future work includes expert qualitative validation, contrastive fine-tuning and supervision-burden analysis, alternative embeddings, and strategies for aggregating multiple cases into coherent prompts.The authors note that combining multiple neighbors while maintaining a coherent prompt is challenging.Source: §7 · p. 15