PaperlayerView exact revision on arXiv

Source-linked AI summary

Case law retrieval: problems, methods, challenges and evaluations in the last 20 years

Daniel Locke, Guido Zuccon

arXiv:2202.07209v1cs.IRcs.AI

TL;DR

The paper surveys case law retrieval research and identifies persistent evaluation challenges, including limited published work and missing large-scale baselines. It also highlights commercial advances in query expansion and related legal search systems.

  • Problem

    Case law retrieval research remains limited despite the importance of helping lawyers find cases and the existence of large commercial systems.

    Source: §1 · pp. 2–3, §7 · p. 31

  • Method

    The paper surveys and summarizes the case law retrieval literature and its key observations.

    Source: §7 · p. 31

  • Results

    Commercial legal search systems are advancing, while query expansion is adopted commercially despite limited published research.

    Source: §7 · p. 32

  • Takeaways & Limitations

    Large-scale collections and baselines for state-of-the-art retrieval methods are required to evaluate progress in case law retrieval.

    Source: §7 · p. 32

    The effectiveness of citation networks for aiding retrieval remains unexplored.

    Source: §3 · p. 7

Abstract

from arXiv · show

Case law retrieval is the retrieval of judicial decisions relevant to a legal question. Case law retrieval comprises a significant amount of a lawyer's time, and is important to ensure accurate advice and reduce workload. We survey methods for case law retrieval from the past 20 years and outline the problems and challenges facing evaluation of case law retrieval systems going forward. Limited published work has focused on improving ranking in ad-hoc case law retrieval. But there has been significant work in other areas of case law retrieval, and legal information retrieval generally. This is likely due to legal search providers being unwilling to give up the secrets of their success to competitors. Most evaluations of case law retrieval have been undertaken on small collections and focus on related tasks such as question-answer systems or recommender systems. Work has not focused on Cranfield style evaluations and baselines of methods for case law retrieval on publicly available test collections are not present. This presents a major challenge going forward. But there are reasons to question the extent of this problem, at least in a commercial setting. Without test collections to baseline approaches it cannot be known whether methods are promising. Works by commercial legal search providers show the effectiveness of natural language systems as well as query expansion for case law retrieval. Machine learning is being applied to more and more legal search tasks, and undoubtedly this represents the future of case law retrieval.

1 INTRODUCTION

Case law retrieval is central to legal practice but increasingly difficult as case collections grow, while published research on ad-hoc ranking remains limited. This survey reviews retrieval methods, related legal-information-retrieval work, evaluation gaps, and future directions.

Motivation

  • Lawyers spend roughly 15 hours per week finding case law, which supports advising clients and fulfilling duties to the court and administration of justice.The task is especially important in common-law jurisdictions because stare decisis requires identifying binding legal principles.Source: §1 · p. 2
  • Legal collections have expanded from millions of published decisions to search systems containing hundreds of millions or billions of documents.The scale makes manual indexing increasingly infeasible and complicates retrieval across broad collections.Source: §1 · p. 2

Methods

  • Standard retrieval has progressed from Boolean, vector-space, probabilistic, and knowledge-engineering methods to separate retrieval and ranking phases using weighted features and deep learning.Features include court, jurisdiction, citations, and past query logs.Source: §1 · pp. 2–3

Research gap

  • Published research on improving or comparing ad-hoc case-law ranking remains limited relative to general information retrieval, likely because commercial providers withhold their methods.The survey distinguishes this gap from substantial work on argumentation retrieval, ontologies, case-based reasoning, discovery, and other legal search tasks.Source: §1 · p. 3

Contribution

  • The article surveys case law retrieval over the past 20 years, covering case structure, retrieval history, Cranfield collections, methods, literature gaps, and future research.Its scope is ad-hoc querying for case law, although lawyers also search legislation, codes, litigation documents, patents, dockets, and internal firm materials.Source: §1 · p. 3

2 OVERVIEW OF A CASE

A case contains both decision text and associated metadata, and different fields support different retrieval tasks. Headnotes and catchwords can aid filtering and searching even though the headnote is not part of the decision.

Case components

  • An example case headnote contains metadata rather than forming part of the decision, while its information remains relevant for filtering and searching cases.Source: §2 · p. 4

Retrieval fields

  • Known-item retrieval can use citations and file numbers, while ad-hoc search can filter by division, proceeding, delivery date, or judges.These fields are distinct from searching the judgment text itself.Source: §2 · p. 4
  • Catchwords are manually identified descriptions that typically include cited cases and legislation, and are usually the principal non-text field searched for ad-hoc retrieval.Decisions may also be classified by topic to aid retrieval.Source: §2 · p. 4

3 THE TASK OF CASE LAW RETRIEVAL, AND AN HISTORY

This section examines the scope and history of case law retrieval as a multifaceted, interactive task shaped by legal collections and research practices. It focuses on how retrieval objectives and evaluation balance quality, cost, recall, precision, and efficient access to relevant authorities.

3 THE TASK OF CASE LAW RETRIEVAL, AND AN HISTORY

  • Case law retrieval includes ad-hoc querying, question answering, routing, known-item search, navigation, and intra-document retrieval, with querying posing the greatest challenge for lawyers.Lawyers must articulate their information need, understand document storage, and know how the retrieval mechanism operates.Source: §3 · pp. 4–5
  • Case law retrieval is difficult because collections are large, decisions vary from six paragraphs to 961 pages, and full text or catchwords are more important than titles or headnotes.Headnotes may be absent or vary in quality, although increased publishing control has made them more available.Source: §3 · p. 5
  • Retrieval also depends on editorial indexing, summaries, classification, court and jurisdiction, judge identity, legally specific language, enduring historical relevance, and citation networks.Citations are central to precedent, but their effectiveness for retrieval remains unexplored.Source: §3 · pp. 6–7

3.1 What is the nature of case law retrieval?

  • The task’s nature is contested between total recall and precision, because lawyers must be fully informed yet face time, cost, and information-burden constraints.Dabney’s total-recall position reflects professional responsibility, while Gerson emphasized precision and the need to avoid overburdening lawyers.Source: §3.1 · p. 7
  • A competing view judges legal information systems by how quickly they find one relevant decision, after which citation-tracing tools can locate additional authorities.This approach is supported by systems enabling citation tracing by paragraph and by Gerson’s account of finding an initial relevant decision quickly.Source: §3.1 · p. 7
  • Case law retrieval should reduce legal-research and service costs while maintaining quality.Source: §3.1 · p. 8
  • Unlike prior-art search and systematic-review searching, case law retrieval is not necessarily a total-recall task.Source: §3.1 · p. 8
  • Case law retrieval is better understood as an interactive task, with searchers willing to examine only around the top 20 results.Lu and Conrad instead appear to view case law search as a high-recall, cost-constrained task.Source: §3.1 · p. 8

4 AVAILABLE COLLECTIONS FOR CASE LAW RETRIEVAL

Available case-law collections vary substantially in scale, query realism, and assessment quality, limiting reliable comparison of retrieval methods. Larger and more realistic collections remain costly and difficult to construct.

Collection limitations

  • Earlier collections lack general utility because they contain few assessments, unsuitable domains, automatically assigned relevance labels, or queries representing broad topics rather than specific legal needs.Koniaris et al.’s assessments came from top-ranked LDA documents, while its queries were derived from Westlaw Digest topics.Source: §4 · p. 9
  • Locke and Zuccon’s collection improves assessment depth with at least 200 judgments per topic, but its small number of queries remains problematic.The collection contains more than 3.5 million documents and uses standard and expert-issued queries.Source: §4 · p. 9

Collection scale

  • Existing collections range from small sets to more than 3.5 million documents, while public legal systems contain roughly 850,000 to over 4 million cases and commercial systems far more.Collection scale is often too small to represent broad case-law searching, though it may represent a single court or jurisdiction.Source: §4 · pp. 8–9

Assessment challenges

  • Citation-based relevance judgments are unreliable because cases cite decisions for different reasons and may omit relevant authorities, making expert assessments preferable but difficult to obtain.The authors note that citing every decision addressing the same question is discouraged in jurisdictions such as Australia and the UK.Source: §4 · p. 10

Commercial evaluation

  • Commercial providers can exploit large query logs to serve cases for similar queries, potentially prioritizing user-data effectiveness over Cranfield-style metrics.Westlaw is described as using query logs to generate surrogate documents for learning ranking features.Source: §4 · p. 10

Future evaluation

  • Cranfield evaluations remain important when user data is unavailable, but legal query logs are sensitive and realistic collections require costly lawyer relevance assessments.Abstract topics avoid client confidentiality, whereas long cases and complex legal issues make expert assessment necessary.Source: §4 · p. 10

Related tasks

  • Related collections support tasks such as retrieving cases relevant to a seed case and identifying how cited cases are treated, rather than general ad-hoc retrieval.These tasks appear in more recent COLIEE competitions.Source: §4 · p. 11

5 METHODS FOR CASE LAW RETRIEVAL

The survey organizes case law retrieval methods and reviews how they have been evaluated, emphasizing limited empirical comparability across systems and collections.

5 METHODS FOR CASE LAW RETRIEVAL

  • The survey summarizes major case law retrieval methods and the principal works examining them.These methods and works are compiled in Table 3.Source: §5 · pp. 11–12, Table 3 · p. 12
  • Published research has not empirically compared the main case law retrieval methods, leaving a significant gap in the literature.Much comparative evaluation instead concerns commercial search systems.Source: §5 · p. 12
  • Evaluations use heterogeneous collections, queries, relevance assessments, and measures, making effectiveness comparisons difficult.Some studies use lawyer-annotated areas of law, while others judge only the first retrieved results subjectively.Source: §5 · pp. 13–14
  • External evaluations reported collections of unknown size and one exceeding 1 million documents; one study used 22 queries, measuring the greater of precision or recall at 20 documents, while another judged first 10 results.The latter used the same collection, whose size was not detailed.Source: §5 · p. 13
  • Related FIRE IrLed and AILA tasks examine citation identification and factual-scenario case retrieval, but neither exactly matches ad-hoc case law retrieval.IrLed becomes a ranking problem through MAP, P@10, Recall@100, and MRR; AILA retrieves cases from natural-language factual scenarios.Source: §5 · p. 14

5.1 Boolean and natural language systems

Boolean and natural-language systems represent contrasting approaches to case law search, with the reported commercial comparison favoring natural-language queries but evaluation conditions limiting comparability.

5.1 Boolean and natural language systems

  • The comparison used Boolean and natural-language queries over collections of 12,000 and 410,000 documents.The evaluation measured effectiveness at P@20, reflecting the view that lawyers may inspect only roughly the first 20 results.Source: §5.1 · p. 15
  • Natural-language queries achieved higher reported effectiveness than Boolean queries on both evaluated collections.On the smaller collection, recall was 0.423 and 0.244 for Boolean queries versus 0.57 and 0.329 for natural-language queries; on the larger collection, P@20 and recall were 0.611 and 0.217 versus 0.759 and 0.269.Source: §5.1 · p. 15
  • The reported study remains the principal comparative evidence for Boolean and natural-language case law retrieval systems.Its Boolean query construction has been criticized because searchers could spend as much time as needed formulating those queries, whereas natural-language queries were fixed.Source: §5.1 · pp. 15–16
  • Short queries can poorly specify information needs, while legal language and long documents make vocabulary mismatch and term-frequency assumptions difficult.Relevant material may consist of only a paragraph or sentence, and the case-law-specific qualitative evidence on query quality remains limited.Source: §5.1 · pp. 16–17
  • Because lawyers may not browse all retrieved results, ranking and explainability are important even when retrieval favors recall.Date-based Boolean ranking was easy to explain, whereas more effective ranking models were harder to explain; ongoing case updates can also reduce reproducibility.Source: §5.1 · p. 17

5.2 Conceptual search, ontologies, case-based retrieval and argument retrieval

Conceptual, ontological, case-based, and argument-retrieval methods address the gap between legal information needs and document language, but their coverage and annotation requirements constrain ad-hoc use.

5.2 Conceptual search, ontologies, case-based retrieval and argument retrieval

  • Case-based retrieval can supply structured seed cases to an information-retrieval system as relevance feedback for searching the broader case-law collection.The approach combines deeply structured but small CBR collections with large but shallow IR collections.Source: §5.2 · pp. 17, 19
  • Conceptual and ontological methods seek to reduce vocabulary mismatch by representing legal concepts rather than relying solely on surface text.Some systems map documents and queries to ontologies, concept fingerprints, or task-based index structures.Source: §5.2 · p. 18
  • Argument and scenario-based systems broaden retrieval beyond entire cases by helping lawyers discover arguments or group outcomes across large collections.One system presents legal arguments rather than entire appellate cases, while another groups outcomes from 500,000 cases by features.Source: §5.2 · pp. 18–19
  • Ontology-based retrieval depends on expert knowledge, and manually classifying the growing body of case law is not feasible.Consequently, automatic methods are needed to determine ontological classifications.Source: §5.2 · p. 19
  • The described CBR approach is limited because it requires a seed case and remains constrained by the scope of the initial CBR collection.It therefore does not directly address ad-hoc query-based retrieval.Source: §5.2 · p. 19
  • Machine-learning-based approaches may reduce the conceptual and representation problems identified in earlier legal retrieval methods.The survey notes that these problems may become less significant as machine-learning approaches are adopted more widely.Source: §5.2 · p. 19

5.3 Question Answering

Question-answering research extends case law retrieval through natural-language interfaces, answer cards, and snippet-level retrieval, while query expansion addresses vocabulary mismatch through statistical, log-based, and ontology-based methods. Evidence is strongest for commercial systems and selected expansion approaches, but many studies lack detailed or quantitative evaluation.

5.3 Question Answering

  • Commercial question-answering systems use natural-language interfaces, machine learning, and large user-query logs to return candidate answers alongside search results.WestSearch PLUS provides access to over 22 million single-sentence case-law summaries and returns high-confidence candidate answers with ad-hoc results.Source: §5.3 · pp. 19–20
  • Question-answering research is relevant to ad-hoc case-law ranking because commercial systems employ learning-to-rank methods and return paragraph- or sentence-level results.The literature also includes legal-code and multilingual retrieval tasks whose methods may inform case-law retrieval despite being outside this article’s scope.Source: §5.3 · p. 20

5.4 Query expansion

  • Implicit expansion in a proprietary system greatly improved effectiveness, whereas BM25 with pseudo-relevance feedback performed worse than the BM25 baseline.The proprietary approach used either pseudo-relevance feedback or a language-modeling engine with a subject-appropriate external corpus.Source: §5.4 · pp. 20–21
  • Query expansion improved both MAP and recall at 1000, and surrounding-word translation probabilities performed well relative to query-log estimates.The evaluation covered over 20,000 cases and 335 queries.Source: §5.4 · p. 21
  • Ontology-based expansion commonly adds superclass and subclass terms, but several studies report only qualitative improvements or omit automated expansion comparisons.Some evaluations used small collections, limiting the extent to which their findings generalise.Source: §5.4 · pp. 22–23

5.5 Query reduction

Query reduction remains sparsely studied in case law retrieval and has mainly used unigram term-selection methods, leaving multi-word and syntactic phrase selection as suggested next steps. Search-result diversification can enrich legal result sets, but its evaluation depends on task assumptions, relevance assessments, and difficult-to-measure search-time effects.

5.5 Query reduction

  • Only a small body of work examines keyword extraction or query reduction for case law, with prior studies comparing automatically scored terms against manually generated queries.Reported term-scoring approaches include IDF, parsimonious language models, and Kullback-Leibler informativeness.Source: §5.5 · pp. 23–24
  • Existing automatic query-reduction research selects unigrams, making multi-word terms and syntactic phrases a natural next step for investigation.This limitation is stated for methods that automatically generate queries.Source: §5.5 · p. 24

5.6 Search Result Diversification

  • The value of diversification depends on whether case-law retrieval prioritises recall or precision, because ranking changes can affect search time without changing the retrieved set.The time required to find all relevant decisions is important but has not been easily quantified.Source: §5.6 · p. 24
  • Diversification is intended to expose hidden aspects of the legal query space, but its reported results may not generalise because relevance assessments came from an LDA topic model rather than manual judgments.The methods traded off relevance and diversity when constructing ranked result lists.Source: §5.6 · p. 25
  • Web-search diversification methods were the best evaluated approaches across a-nDCG, ERR-IA, and S-recall.The study sought to produce 30 diversified results from an original ranking of 100 over a collection of 3890 cases.Source: §5.6 · p. 25

5.7 Use of citation networks

Citation networks have been used to trace and rank related case law, but their application to ad-hoc retrieval remains limited and methodologically constrained. Effective ranking must account for citation topicality and how cited decisions are treated, not merely citation counts.

Use of citation networks

  • Citation networks have been applied to ranking case-law documents in only two reported instances, contrasting with their broader use in web-result ranking.The literature identifies limited public application despite citation indexes, visual tools, and proposals to use citations as ranking features.Source: §5.7 · p. 26
  • FLEXICON combined citations with a vector-space ranking model on a 1,000-document collection, but its commercial implementation details were not disclosed.The system’s exact use of citations was not specified.Source: §5.7 · p. 26
  • DivRank and other web-based diversification methods were reported effective for diversifying case-law search results.Koniaris et al. considered diversification rather than standard ad-hoc query reranking.Source: §5.7 · p. 26
  • Citation-vector ranking can exclude uncited or newly decided cases, including decisions addressing uncommon legal questions.This limitation arises when citation vectors are used as ranking features rather than alongside other retrieval signals.Source: §5.7 · p. 26
  • Simple citation-network analysis treats citations alike despite differing purposes, including approval, following, disagreement, or procedural relevance.Case-law documents also cover multiple topics, so citation topicality and treatment of the cited case remain important unexplored ranking dimensions.Source: §5.7 · p. 27
  • Existing approaches include topic-linked citation browsing, paragraph-level citation analysis, and seed-judgment ranking using BM25 plus bibliographic coupling.These methods generally address tracing or similarity from a seed case rather than ad-hoc query ranking, and some do not evaluate ranking directly.Source: §5.7 · p. 27

5.8 Deeper understanding of legal text and AI applications to case law retrieval

AI and deeper legal-text understanding have produced semantic, rhetorical, and machine-learning approaches to case-law retrieval, but most evaluations remain small or address tasks adjacent to ad-hoc ranking. Reported systems nevertheless demonstrate applications ranging from fact-focused search to embedding-based similarity and automated query understanding.

Deeper understanding of legal text and AI applications to case law retrieval

  • Commercial legal-search systems apply machine learning to query reformulation, intent understanding, semantic search, case analysis, and litigation analytics.The surveyed literature also notes automatic overruling detection, parsing, and recommendations based on similar user profiles.Source: §5.8 · pp. 28, 30
  • Semantic search and embedding-based methods have been explored for legal text, but no work specifically evaluated semantic similarity for ad-hoc case-law ranking.Related applications include seed-document retrieval, statutory-provision matching, similar legal provisions, and e-Discovery.Source: §5.8 · p. 28
  • Rhetorical-role retrieval has been evaluated on a 188-case vaccine-related collection, where documents were ranked by the number of relevant sentences they contained.The approach retrieved relevant sentences before aggregating them to the document level, but the small collection limits the evaluation’s scale.Source: §5.8 · p. 29
  • Nejadghole et al. searched only factual portions of cases by combining classifier-based annotation with word embeddings over a 46,000-decision collection.They manually annotated 150 decisions and used the annotations to train a classifier for the remaining collection.Source: §5.8 · pp. 29–30
  • An evaluation of fact-focused retrieval reported MAP 0.78 using 15 queries paired with five candidate sentences each.Results were ordered by similarity to the query after automatically annotating the collection.Source: §5.8 · p. 30
  • Rhetorical-role classification faces sparse training examples and inconsistent tagging, and one study found better performance when ranking ignored sentence classifications.Automatic summarisation research is identified as a possible source of methods for addressing these difficulties.Source: §5.8 · p. 30

6 OPEN ISSUES, CHALLENGES AND DIRECTIONS

Case-law retrieval faces unresolved evaluation, document, task-definition, and methodological challenges. Future work should establish scalable baselines and test collections while examining domain-specific citation features, semantic representations, and text roles.

OPEN ISSUES, CHALLENGES AND DIRECTIONS

  • There is no standard collection for ad-hoc case-law ranking, making methods difficult to compare and limiting the validity of evaluations beyond small collections.Semantic-understanding studies often use hundreds of documents rather than the hundreds of thousands or millions typical of IR evaluations.Source: §6 · p. 30
  • Case-law documents are unusually difficult to retrieve because they are long, cover diverse topics, and use complex language and structure.These intrinsic characteristics create difficulties not commonly present in other retrieval domains.Source: §6 · p. 31
  • Whether retrieval should prioritize total recall or high recall with acceptable precision changes the importance of ranking, replicability, explainability, and user trust.Precision-oriented tasks make ranking more important when lawyers must find relevant documents through few interactions.Source: §6 · p. 31
  • Future evaluations should establish robust test collections and baselines, then examine citations, topicality, citation treatment, court relationships, and elapsed time.These domain-specific features have not been empirically evaluated for case-law retrieval ranking.Source: §6 · p. 31
  • Semantic search and segmentation by a subsection’s rhetorical role are identified as promising directions for future case-law retrieval research.Prior semantic and ontological work has generally been small-scale, with recent commercial systems as exceptions.Source: §6 · p. 31

7 CONCLUSION

The survey finds limited published comparison of ad-hoc case-law ranking despite substantial commercial activity and broader legal-IR research. It highlights natural-language retrieval, query expansion, large-scale evaluation, and machine learning as central directions for the field.

CONCLUSION

  • The literature has shifted toward natural-language queries as Boolean retrieval becomes less common in legal education and practice.Boolean search remains present, but the survey reports a turn toward natural-language querying.Source: §7 · p. 31
  • Commercial legal-search systems and broader research increasingly apply machine learning across information retrieval and legal tasks.The survey presents machine learning as central to the future of case-law retrieval alongside natural-language and question-answering systems.Source: §7 · p. 32
  • Published research on ad-hoc case-law ranking remains sparse compared with work on conceptual search, question answering, and other legal-search tasks.Commercial providers may withhold successful ranking factors because of the size and competitiveness of the legal-search market.Source: §7 · p. 32
  • Query expansion is widely adopted by commercial search systems, although little published research empirically evaluates its effectiveness.The absence of standardized testing is identified as a major evaluation problem.Source: §7 · p. 32
  • Large-scale public collections and baselines are needed to compare ad-hoc case-law retrieval methods outside commercial settings.Creating such collections is time-consuming and would most likely require lawyers, whereas commercial providers can leverage vast query logs.Source: §7 · p. 32

Related papers

Loading 2202.07209v1…