PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges

Farid Ariai, Joel Mackenzie, Gianluca Demartini

arXiv:2410.21306v3cs.CLcs.AI

TL;DR

Legal NLP must contend with lengthy, complex documents and concerns about fairness, bias, and explainability. This survey follows the PRISMA framework to examine the field. It reports greater research activity in LJP, LQA, LTC, and LLMs, while identifying statutory law as a significant focus and highlighting open challenges.

  • Problem

    Legal NLP faces lengthy documents and complex language, as well as fairness, bias, and explainability concerns.

    Source: §1 · p. 2

  • Method

    The survey follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework.

    Source: §4 · p. 9

  • Results

    Research is more extensive in LJP, LQA, LTC, and LLMs, and the survey identifies a significant focus on statutory law and challenges including bias, fairness, interpretability, and explainability.

    Source: Summary · p. 29, §14 · p. 29

  • Takeaways & Limitations

    The survey highlights domain-specific fine-tuning, bias and fairness, interpretability, explainability, robustness, and multilingual capabilities as open legal-NLP challenges.

    Source: §14 · p. 29

    Integrating RLHF into legal NLP pipelines is constrained by the need for accurate legal knowledge during human feedback.

    Source: ORC9: Reinforcement Learning from… · p. 27

Abstract

from arXiv · show

Natural Language Processing (NLP) is revolutionising the way both professionals and laypersons operate in the legal field. The considerable potential for NLP in the legal sector, especially in developing computational assistance tools for various legal processes, has captured the interest of researchers for years. This survey follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, reviewing 154 studies, with a final selection of 131 after manual filtering. It explores foundational concepts related to NLP in the legal domain, illustrating the unique aspects and challenges of processing legal texts, such as extensive document lengths, complex language, and limited open legal datasets. We provide an overview of NLP tasks specific to legal text, such as Document Summarisation, Named Entity Recognition, Question Answering, Argument Mining, Text Classification, and Judgement Prediction. Furthermore, we analyse both developed legal-oriented language models, and approaches for adapting general-purpose language models to the legal domain. Additionally, we identify sixteen open research challenges, including the detection and mitigation of bias in artificial intelligence applications, the need for more robust and interpretable models, and improving explainability to handle the complexities of legal language and reasoning.

1 Introduction

Legal NLP offers assistance across legal processes, but lengthy documents, complex language, and fairness and explainability concerns make the domain challenging. This survey maps the field’s tasks and related resources, while identifying uneven research activity and gaps in prior surveys.

1 Introduction

  • Legal NLP faces challenges from lengthy documents, complex legal language, and complicated document structures, alongside fairness, bias, and explainability concerns.The introduction notes that legal AI must meet accuracy, fairness, and transparency standards because it can affect clients’ lives and rights.Source: §1 · p. 2
  • The survey spans legal NLP tasks, datasets, corpora, and language models, presenting a broad view of the field and its practical challenges.It covers areas including question answering, judgement prediction, text classification, summarisation, named entity recognition, and argument mining.Source: §1 · p. 2
  • LDS and LQA have extensive research activity, while NER and legal corpora receive comparatively less attention.The survey also notes substantial datasets and research contributions for LDS and LQA.Source: §1 · p. 2

2 Related Work

  • The survey aims to provide a general view of legal NLP rather than focus solely on specific applications, covering tasks, datasets, corpora, and language models.It positions this breadth against prior surveys that focused on selected tasks or omitted resources such as legal corpora and pre-training methods.Source: §2 · pp. 3–4
  • Its review includes datasets and large corpora in 24 languages, popular legal language models, and detailed discussions of challenges and technical solutions.The authors present the survey as a resource for legal NLP research and a roadmap for areas needing further exploration.Source: §2 · p. 4

3 Background and Foundational Concepts

This section introduces the specialized language, structure, and reasoning of legally consequential texts and the challenges they pose for conventional NLP. It surveys task-tailored methods, retrieval-augmented generation, legal applications and limitations of LLMs, and publication venues in legal NLP.

3.1 Legal Documents

  • Legal documents encode authoritative rules, rights, duties, procedural requirements, and reasoning, supporting analysis, drafting, compliance, and teaching.Examples include court filings, judgments, statutes, treaties, contracts, and formal correspondence.Source: §3.1 · p. 5

3.1.1 Legal language and its characteristics.

  • Legal language’s formal, specialized wording, complex syntax, punctuation, and intertextual references demand contextual interpretation because small textual details can change rights and liabilities.The section also notes that legal texts can be less accessible to non-specialists and that disambiguation, nested entities, non-machine-readable PDFs, and varied reasoning challenge conventional NLP pipelines.Source: §3.1.1 · p. 5

3.1.2 Domains with Shared Characteristics.

  • Medical texts and software documentation share legal texts’ specialized vocabularies, large corpora, and cross-references, alongside case-based analysis in medicine.These shared features include domain-specific terminology and extensive references to related material.Source: §3.1.2 · p. 6, §3.1.1 · p. 6

3.2 Legal NLP

  • Legal NLP supports drafting and analyzing documents, compliance and due-diligence checks, legal education, and complex decision-making.The field aims to make textual legal information more accessible, interpretable, and actionable.Source: §3.2 · pp. 6–7

3.2.1 Basic foundations and concepts of NLP.

  • Tokenization segments lengthy legal texts, embeddings represent semantic relationships, and transformers use self-attention to contextualize input representations.Pre-trained language models learn patterns from large corpora and can be fine-tuned for legal tasks.Source: §3.2.1 · p. 7
  • Multi-task learning shares knowledge across related tasks, while parameter-efficient fine-tuning updates a small parameter subset to reduce compute and time in resource-limited settings.The survey describes these approaches as useful for data-scarce legal NLP and notes competitive performance for PEFT in tasks such as text generation.Source: §3.2.1 · p. 7
  • Task-tailored ML paradigms enhance core NLP methods, particularly pretrained language models, while retrieval-augmented generation combines information retrieval with generation to add external context.RAG retrieves external knowledge before generating responses, complementing NLP methods with context that can enhance text generation and refinement.Source: §3.2.1 · p. 7

3.2.2 LLMs as an application of NLP in the legal sector.

  • LLMs are deep-learning NLP models, typically Transformer-based and trained on extensive text corpora, that perform tasks including summarisation, document analysis, and question answering.Their release into widespread attention followed ChatGPT’s November 2022 launch.Source: §3.2.2 · p. 8
  • GPT-4 was reported near the 90th percentile on the Uniform Bar Exam, but Martínez estimated 48th overall and 15th on essays.The differing estimates illustrate uncertainty about its performance.Source: §3.2.2 · p. 8
  • Smaller fine-tuned models outperform ChatGPT by about 30 percentage points on some legal classification tasks [16], while legal-content hallucinations reached 58% in some cases.The survey notes that hallucinations raise reliability concerns for complex tasks.Source: §3.2.2 · p. 8
  • A LexisNexis survey found about half of lawyers expect LLMs to transform legal practice, 92% anticipate some impact, 77% foresee efficiency gains, and 63% predict changes in legal education.Source: §3.2.2 · p. 8

3.2.3 Key publications and conferences in legal NLP.

  • Legal NLP research appears in dedicated journals, conferences, and workshops, as well as broader AI, NLP, and information-retrieval venues.Examples include Artificial Intelligence and Law, ICAIL, JURIX, NLLP, XAILA, and COLIEE; the TREC Legal Track evaluated legal-document retrieval for e-discovery from 2006 to 2011.Source: §3.2.3 · p. 8

4 Methodology

The survey uses a PRISMA-based review to identify and screen legal NLP studies, then applies eligibility criteria to select research for synthesis.

4 Methodology

  • The review follows PRISMA to support a transparent and comprehensive assessment of legal NLP research.The methodology describes database searching, manual screening, and eligibility criteria.Source: §4 · p. 9

4.1 Search Strategy

  • Searches across Google Scholar and IEEE Xplore used legal-NLP queries and publication windows tailored to tasks, corpora, and legal language models.Peer-reviewed journal articles and high-quality conference papers were prioritized, with relevant non-peer-reviewed sources also considered.Source: §4.1 · p. 9

4.2 Study Selection

  • Manual title-and-abstract screening and full-text review reduced 154 identified studies to 131 retained for synthesis.Full-text review checked relevance, quality, and inclusion criteria; literature reviews were also examined for engagement with existing legal NLP research.Source: §4.2 · p. 9

4.3 Eligibility Criteria

  • Eligibility required focus on a target legal NLP task, legal language models, or legal corpora, plus empirical research or a significant methodological contribution.Editorials, opinion pieces, papers below basic methodological standards, and general or unrelated NLP studies were excluded.Source: §4.3 · pp. 9–10

5 Legal Question Answering

Legal question answering requires interpreting laws and applying legal principles to facts, making both reasoning demands and dataset design central concerns. The surveyed approaches combine legal knowledge and retrieval with answer generation, and some report improvements in retrieval or performance without domain-specific training.

5 Legal Question Answering

  • Legal question answering requires reviewing laws, interpreting statutes and regulations, and applying principles and precedent to facts; both knowledge-driven and case-analysis questions demand advanced reasoning.The survey notes a gap between machine performance and human expertise in legal reasoning.Source: §5 · p. 10, §5.1 · p. 10

5.1 Datasets

  • JEC-QA contains 26,365 Chinese multiple-choice questions annotated for reasoning type, while the survey distinguishes concept-focused knowledge questions from case-analysis questions.The categories both demand deep text comprehension; the survey describes this as a challenge for NLP.Source: §5.1 · p. 10
  • LQA datasets range from Q4PIL’s 17 EU-regulation questions to EQUALS’s 6,914 Chinese-law question–answer pairs and GerLayQA’s 21,538 layperson questions with lawyer answers.EQUALS annotations identify the reasoning type, while GerLayQA answers are grounded in German legal-code paragraphs.Source: §5.1 · p. 10

5.2 Approaches

  • Approaches include Chinese legal QA with a knowledge graph for query comprehension and answer ranking, BM25 and dense-vector retrieval with BERT reranking, and retrieve-then-generate answers grounded in retrieved text.Source: §5.2 · p. 11
  • Discourse-based DiscoLQA improved LQA performance on Q4EU even without domain-specific training.Q4EU contains over 70 questions and 200 answers on six European norms; DiscoLQA focuses on discourse elements to guide retrieval [104][105][106].Source: §5.2 · p. 12
  • Machine-generated questions are reported to improve scalability and access to legal information for non-experts, while incorporating question tags can improve legal answer retrieval.The question-tag approach uses a two-stage pipeline with BM25 retrieval followed by a reranker [116][117][118].Source: §5.2 · p. 12

6 Legal Judgement Prediction

Legal judgement prediction (LJP) predicts outcomes from case descriptions and applicable legislation, but historical data may be biased. The survey reviews datasets and approaches, including models that capture dependencies among legal subtasks and decision elements.

6 Legal Judgement Prediction

  • LJP predicts legal outcomes from case descriptions and applicable legislation, while historical-data bias can create feedback loops and amplify discrimination if unmanaged.Legal experts still chiefly perform LJP and require specialised training to identify relevant statutes, charge ranges, and penalty terms.Source: §6 · p. 12

6.1 Datasets

  • LJP datasets comprise court-view generation, law-article, charge-prediction, and prison-term-prediction types, each presenting unique challenges that require text comprehension and the ability to apply complex legal reasoning.Source: §6.1 · p. 12
  • 171,981 Chinese cases in Court View Gen cover 51 charge categories, while Ye et al. generate court views from charge labels to provide explicit justifications for charge decisions.The dataset contains cases with a single defendant and a corresponding charge.Source: §6.1 · p. 12, §6.2 · p. 13

6.2 Approaches

  • Models predicting outcomes for Convention articles outperformed previous feature-based approaches; trained on textual proceedings, a model achieved an average 75% accuracy, dropping to 58–68% on future cases, while judge-surname-only prediction reached 65%.QAjudge iteratively selects relevant yes-no questions about case facts; its Answer Net provides answers and its Predict Net generates the final judgement.Source: §6.2 · p. 13
  • LADAN distinguishes confusing law articles with graph learning and attention, improving law-article, charge, and penalty-term prediction over previous methods.ML-LJP incorporates charge- and term-related articles into multi-label law-article prediction and outperformed SOTA models, particularly for prison-term prediction.Source: §6.2 · p. 14
  • MSJudge models interactions among claims, facts, and debates using multi-stage judicial data, including pre-trial claims and court debates, and more accurately characterises those interactions for judgement prediction, improving over SOTA baselines.GJudge outperforms other models, including SOTA RLJP, achieving higher F1 scores.Source: §6.2 · p. 14
  • TopJudge orders subtasks using a directed acyclic graph reflecting legal decision-making and outperformed previous models on large-scale Chinese criminal cases.CL4LJP uses supervised contrastive learning to distinguish similar law articles and charges.Source: §6.2 · pp. 13–14

7 Legal Text Classification

Legal text classification assigns predefined categories to legal documents, often across large multi-label spaces. The survey covers specialised datasets and methods addressing long inputs, limited labelled data, cross-lingual transfer, and document structure.

7 Legal Text Classification

  • Legal document classification can be a large multi-label task, with thousands of potential categories, while locating relevant rulings in similar cases for argumentation is increasingly challenging for legal professionals.EURLEX57K contains 57,000 EU legislative documents annotated with EuroVoc concepts.Source: §7 · pp. 14–15, §7.1 · p. 15

7.1 Datasets

  • Datasets cover diverse legal classification targets, including Greek legislation, US legal procedural postures, contract modalities, and Terms of Service unfairness.POSTURE50K supports supervised and zero-shot evaluations with infrequent categories represented; Claudette6 analysis found language differences, especially in German, including structure and terminology.Source: §7.1 · p. 15

7.2 Approaches

  • DistilBERT with long and task-motivated attention processes inputs up to eight times longer than standard BERT and outperformed fine-tuned BERT and other transformers in speed and performance.The architecture is optimised for legal document classification.Source: §7.2 · p. 16
  • An active-learning pipeline combining legal-text pre-training, knowledge distillation, and clustered sampling reduces labelling actions and overall legal text classification cost.Multi-task learning is also described as a way to mitigate data scarcity [132].Source: §7.2 · pp. 15–16
  • For unfair-clause detection, translating query documents into English matched or beat separate-language models when translation quality was high; with low-quality translation, separate models performed better.Annotation projection and translation of English training documents performed slightly below separate-language models but remained viable alternatives.Source: §7.2 · p. 16

8 Legal Document Summarisation

Legal document summarisation must handle lengthy, structured texts whose statutory language and citations matter. The survey reviews datasets and extractive, abstractive, and combined approaches for producing useful legal summaries.

8 Legal Document Summarisation

  • Legal document summarisation must account for long documents, internal structure, statutory language, citations, and differences in interpretation across court levels.Extractive methods select existing sentences or phrases, whereas abstractive methods generate new sentences that paraphrase key information.Source: §8 · p. 17

8.1 Datasets

  • Datasets include Multi-LexSum, with 40,000 source documents and 9,000 expert-written summaries for US federal civil-rights lawsuits, and CLSum for multi-jurisdictional common-law judgements.CLSum uses LLMs for data augmentation and incorporates legal knowledge for summary generation and evaluation.Source: §8.1 · p. 17

8.2 Approaches

  • Legal summarisation systems combine document-specific processing with different selection and generation strategies; FLEXICON combined extractive and abstractive methods to retain core facts while producing concise summaries.Casesummarizer scores sentences using legal-case TF-IDF and entities, dates, and section headings, and offers scalable summaries.Source: §8.2 · pp. 17–18
  • Reinforcement-learning methods tune legal summarisation with lexical and semantic rewards, while graph-based extractive ranking reduces redundancy and includes argumentative sentences from underrepresented sections [156].A Dutch-court study also applied reinforcement learning with a Bi-LSTM and abstractive summarisation using BART.Source: §8.2 · p. 18
  • Transfer learning that combines extractive and abstractive summarisation addressed scarce labelled legal data, outperformed previous results on Australian Legal Case Reports, and established a new abstractive baseline.An earlier FLEXICON system often produced documents longer than desired.Source: §8.2 · p. 18

9 Legal Named Entity Recognition

Legal named entity recognition identifies legal-specific entities in complex, structured texts. The survey describes datasets with specialised entity types and approaches adapting sequence-tagging models and annotation tools to legal contexts.

9 Legal Named Entity Recognition

  • Legal NER extracts entities such as laws, legal norms, and procedural terms, requiring methods tailored to legal language and document structure.These entities can structure legal documents and enhance legal information-retrieval systems.Source: §9 · p. 18

9.1 Datasets

  • Legal NER resources include E-NER, an annotated collection derived from US Securities and Exchange Commission EDGAR filings, and an Indian High and Supreme Court judgment corpus containing 46,545 entities across 14 types.Its development and test sets cover cases from 2018 to 2022, ensuring no training leakage.Source: §9.1 · p. 19

9.2 Approaches

  • A Romanian-German domain-adaptation model combining BERT, Bi-LSTM, CRF, and adversarial learning produced marginal gains for German but lower performance on Romanian legal data.The results indicate that adaptation benefits vary by language and domain.Source: §9.2 · p. 20
  • LeDA lets annotators define new legal concepts dynamically and includes meta-annotation to adjudicate conflicting annotations.Păis et al. combine Bi-LSTM and CRF layers with word and character embeddings, gazetteers, and legal affixes.Source: §9.2 · pp. 19–20

10 Legal Argument Mining

Legal argument mining identifies claims and premises, models their relations, and can expose reasoning chains relevant to judicial decisions and legal tasks. The section surveys annotation resources and approaches for detecting and structuring these arguments.

10 Legal Argument Mining

  • Legal argument mining identifies claims and premises and their interrelations, supporting legal reasoning, conflict resolution, and why-question answering.Effective mining involves identifying argumentative units, modelling rhetorical relations, and testing whether those structures can be derived automatically.Source: §10 · p. 20

10.1 Datasets

  • Habernal et al. annotated 373 ECHR decisions, comprising about 2.3 million tokens and over 15,000 argument spans, with Krippendorff’s alpha close to 0.80.Six law students annotated the corpus under expert supervision.Source: §10.1 · p. 20
  • The Demosthenes corpus covers 40 CJEU fiscal state-aid decisions, with sentence-level annotations distinguishing premises, conclusions, and non-argumentative text.Its tasks include argument-clause recognition, clause-relation prediction, and premise/conclusion classification; the annotation process achieved 80% inter-annotator agreement.Source: §10.1 · p. 20

10.2 Approaches

  • Statistical classifiers distinguished argumentative sentences with approximately 80% accuracy on the ECHR corpus.The surveyed approaches also investigate semantic relatedness and rhetorical patterns for argument-boundary detection.Source: §10.2 · p. 21
  • ResGCN outperformed baseline methods on both ECHR and CJEU datasets when classifying premises, conclusions, and non-argumentative text.The graph-based framework models text segments as nodes and their sequential or semantic relationships as edges, with virtual-node augmentation and iterative label refinement.Source: §10.2 · p. 21
  • ResAttArg outperformed DistilRoBERTa on link prediction with lower computational demand in the surveyed comparison.The study extends Demosthenes with direct and indirect support, rebuttal, undercut, and rephrase relations.Source: §10.2 · p. 21
  • Habernal et al.’s multi-task transformer identifies arguments and classifies propositions as premises or conclusions, outperforming previous legal NLP models in expert evaluations.The model also distinguishes non-argumentative sentences.Source: §10.2 · p. 21

11 Large Legal Datasets

This section surveys legal datasets and benchmarks for task evaluation, alongside large corpora used for pre-training. The resources span legal tasks, jurisdictions, languages, and corpus scales.

11.1 Evaluation benchmarks

  • LexGLUE brings together legal datasets for varied tasks, including classification and multiple-choice questions, with chronological train, development, and test splits.Its datasets include ECHR, SCOTUS, EUR-Lex, LEDGAR, UNFAIR-ToS, and CaseHOLD.Source: §11.1 · p. 22
  • FairLex provides four legal datasets across Europe, the United States, Switzerland, and China in five languages, with sensitive attributes for demographic, regional, and topical fairness analysis.The datasets align with ECHR violation prediction, SCOTUS issue-area classification, FSCS case-approval prediction, and CAIL crime-severity prediction.Source: §11.1 · p. 22
  • AsyLex contains 59,112 Canadian refugee-status determination documents from 1996–2022, including human-annotated and inferred labels for entity extraction and judgement prediction.The dataset includes 19,115 gold-standard human-annotated and 30,944 inferred labels.Source: §11.1 · p. 23
  • LEXTREME aggregates 11 human-annotated datasets across 24 languages and multiple legal domains, covering single-label and multi-label classification and NER.Its dataset and language aggregate scores show performance rising with model size.Source: §11.1 · p. 23
  • Park and James develop an NLI dataset from Korean criminal verdicts using adversarial hypothesis generation; they report annotator domain-knowledge and long-context challenges.The study describes targeted data collection and gamification as responses to annotator engagement challenges.Source: §11.1 · p. 23
  • COLIEE competitions cover case retrieval and entailment alongside statute retrieval and question answering; the 2023 edition reports over 5,700 case-law files.The 2021 competition featured five tasks across case and statute law.Source: §11.1 · pp. 22–23
  • The Cambridge Law Corpus contains 258,146 UK court cases spanning from the 16th century to the present, with outcome annotations for a subset of 638 cases.It includes raw text and metadata in XML and has a supporting Python library.Source: §11.1 · p. 23

11.2 Large corpora for pre-training

  • The surveyed pre-training corpora include the 256 GB Pile of Law across 17 jurisdictions and 24 languages, and the 689 GB MultiLegalPile [98].Pile of Law includes case law, legislation, and contracts; its legal-content selection uses filtering and manual review.Source: §11.2 · p. 24

12 Legal Language Models and Methods for Legal Domain Adaptation

The section reviews legal-oriented language models and methods for adapting general-purpose models to legal documents. The approaches address domain-specific language and long inputs through specialized training, architectures, and adaptation techniques.

12.1 Language Models

  • Further pre-training BERT on legal corpora and pre-training from scratch generally outperformed using BERT directly; legal-BERT variants showed competitive performance with lower environmental impact.The surveyed comparison considered direct BERT use, further pre-training, and legal-specific pre-training from scratch.Source: §12.1 · p. 24
  • Lawformer adapts Longformer for Chinese legal texts, using sliding-window, dilated, and global attention to process long documents for tasks such as LJP and LQA.It is pre-trained on Chinese legal documents segmented into criminal and civil cases.Source: §12.1 · pp. 24–25
  • AraLegal-BERT adapts BERT to Arabic jurisprudence by pre-training from scratch on legal documents, including legislative materials and contracts.Source: §12.1 · p. 25
  • SaulLM-7B combines a 30-billion-token legal continued-pretraining corpus with instruction fine-tuning using generic and legal-specific instructions [27].It is built on the 7-billion-parameter Mistral architecture.Source: §12.1 · p. 25
  • Legal-LM combines legal-corpus pre-training and a legal knowledge graph with preference optimization and external knowledge retrieval and validation for legal consulting.The surveyed framework is presented as generating precise, user-aligned legal advice.Source: §12.1 · p. 25

12.2 Methods for Improving In-Domain Adaptability of Legal Language Models

  • Prefix domain adaptation trains about 0.1% of model parameters and improves few-shot performance over full-model tuning while reducing computational overhead.The method tunes a deep prompt using a domain-adapted prefix from legal forums and reports maintained or exceeded performance across multiple legal tasks.Source: §12.2 · p. 25
  • Long-document adaptations include extending Longformer input capacity to 8,192 subwords and adding TF-IDF-based variants of LegalBERT, which outperform traditional linear classifiers while maintaining computational efficiency [74].The adaptations include sorting and deduplicating subwords by TF-IDF scores or adding a TF-IDF embedding layer.Source: §12.2 · p. 25

13 Open Research Challenges

The survey identifies open research challenges in legal NLP and offers directions for future work. These include fairness, privacy, interpretability, annotation practice, data scarcity, and multilingual support.

13 Open Research Challenges

  • Historical legal decisions may reflect human biases, and models trained on such data can produce unfair outcomes in downstream classification and prediction.The section identifies bias and fairness as a central challenge because legal decisions can deeply affect individuals’ lives.Source: ORC1: Bias and Fairness · p. 26

ORC2: Privacy Concerns.

  • Privacy-preserving methods require careful calibration because excessive privacy constraints may degrade model accuracy.The section calls for privacy-preserving NLP methods tailored to legal-domain demands.Source: ORC2: Privacy Concerns · p. 26
  • Interpretability and explainability are underexplored in many legal NLP contexts, although tracing AI decisions is described as essential for identifying and mitigating bias.The section links transparency and understandability with trust and responsible use in legal settings.Source: ORC3: Interpretability and… · p. 26
  • Legal NLP studies often omit annotator backgrounds and procedures, making dataset-quality assessment, fairness, and trust harder to evaluate.The section calls for explicit choices about descriptive versus prescriptive annotation and transparent documentation.Source: ORC4: Annotation Process and… · p. 26
  • Annotated-data scarcity limits robust model development because labelling complex legal texts requires substantial cost and expertise.Augmentation also faces challenges in preserving legal-text integrity, tailoring approaches to models, scaling computation, and avoiding overfitting.Source: ORC5: Scarcity of Reliable Annotated… · p. 27
  • The section identifies multilingual capability as underdeveloped and notes that English-focused augmentation limits applicability in multilingual and extremely low-resource legal contexts.It calls for augmentation approaches that support diverse languages.Source: ORC6: Multilingual Capabilities · p. 27, ORC5: Scarcity of Reliable Annotated… · p. 27

ORC6: Multilingual Capabilities.

The identified open challenges span multilingual access, legal-text processing and representation, model efficiency, and legal reasoning. Across these areas, the survey notes research gaps as well as constraints arising from legal language and practice.

ORC6: Multilingual Capabilities.

  • Distinct linguistic structures, legal terminology, and document standards make multilingual legal NLP difficult; research remains limited for languages including Persian and Arabic.The survey calls for expanded research in underserved languages to support broader applicability.Source: ORC6: Multilingual Capabilities · p. 27

ORC7: Ontologies and Knowledge Graphs.

  • Static ontologies are difficult to build for legal texts because open-texture terms evolve, and models must accommodate both legal and real-world concepts.The survey describes legal ontologies as needing to change with law and its interpretation over time.Source: ORC7: Ontologies and Knowledge Graphs · p. 27

ORC8: Pre-processing Legal Text.

  • Nested clauses and cross-references make legal documents hard to segment, complicating preprocessing and making fine-tuning on raw legal data impractical.The survey links these preprocessing difficulties to limits on legal NLP application performance.Source: ORC8: Pre-processing Legal Text · p. 27

ORC9: Reinforcement Learning from Human Feedback (RLHF).

  • RLHF research in law is scarce, and its use requires legal experts to provide guidance on complex concepts, with the cost of employing them a barrier.The survey says the investment may be worthwhile if downstream savings from improved worker efficiency are demonstrated.Source: ORC9: Reinforcement Learning from… · pp. 27–28

ORC11: Small Language Models (SLMs).

ORC10: Expanding Legal Domain Coverage.

  • Intellectual Property, Criminal, Banking, Family, and Human Rights Law have seen limited exploration across legal NLP tasks, including legal question answering.The survey identifies tailored solutions and insights for these sectors as a motivation for expanding research.Source: ORC10: Expanding Legal Domain Coverage · p. 28

ORC13: Legal Logical Reasoning.

ORC15: Stochastic Parrots.

The survey cautions that language models may mimic patterns rather than understand language, with risks in critical legal applications. It discusses retrieval-augmented generation as a response to long-document challenges, while noting unresolved legal-setting constraints and broader research gaps.

ORC15: Stochastic Parrots.

  • Mimicked language patterns can produce unreliable outcomes in critical legal settings, especially when models lack high-quality, unbiased training data.The survey highlights particular risk in legal judgment prediction, where biased or unfair data could have irreversible outcomes, and cites Bender et al..Source: ORC15: Stochastic Parrots · p. 28

ORC16: Retrieval-Augmented Generation.

  • Lengthy documents, cross-references, and complex linguistic structures can lead LLMs to hallucinate precise answers; RAG incorporates relevant passages into generation to address input-length limits.Legal RAG still faces challenges involving jurisdictions, temporal currency, multilingual texts, and retrieval bias.Source: ORC16: Retrieval-Augmented Generation · p. 29

Summary.

  • Table 5 links open research challenges to legal NLP areas, and the survey reports that most are related to LJP, LQA, LTC, and LLMs.The survey says significant work remains across all areas of focus.Source: Summary · p. 29, Table 5 · p. 29

14 Conclusion

  • The survey reviews legal NLP tasks, datasets, and domain-tailored language models, while identifying bias, fairness, interpretability, explainability, preprocessing, and multilingual capability as open challenges.It also identifies ontology-based methods for more accurate legal reasoning as an open challenge.Source: §14 · p. 29
  • The surveyed approaches vary across tasks: legal NER uses methods from rule-based to BERT-based, LDS ranges from TF-IDF to transformers, and LAM uses domain-specific schemes and graph-based or residual-network approaches.The survey also reports multi-class classification as dominant in LTC, with CNNs and Bi-LSTMs widely used.Source: §14 · p. 29

Related papers

Loading 2410.21306v3…