PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law

Shounak Paul, Arpan Mandal, Pawan Goyal, Saptarshi Ghosh

arXiv:2209.06049v5cs.CLcs.AIcs.LG

TL;DR

Indian legal texts have distinguishing characteristics, and existing legal PLMs had not been compared in a common setting or explored for other countries’ legal text. This study adapts legal PLMs using Indian data and evaluates them across legal tasks and jurisdictions; InLegalBERT performs best on almost all tested end-tasks.

  • Problem

    Indian legal texts differ from EU and US legal texts, and prior work had not compared legal PLMs in a common setting or explored adapting them to other countries.

    Source: §1 · p. 1

  • Method

    The study continues pre-training LegalBERT and CaseLawBERT on Indian legal data, trains a BERT-base model from scratch with an Indian-specific vocabulary, and evaluates them across three tasks and Indian and non-Indian datasets.

    Source: §1 · p. 2

  • Results

    InLegalBERT attains the best F1 scores across all three tasks and five datasets, and outperforms the original-paper results on four of five datasets.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    The authors conclude that pre-training on Indian legal text generally improves performance across the tested legal tasks and datasets.

    Source: §6 · p. 9

Abstract

from arXiv · show

NLP in the legal domain has seen increasing success with the emergence of Transformer-based Pre-trained Language Models (PLMs) pre-trained on legal text. PLMs trained over European and US legal text are available publicly; however, legal text from other domains (countries), such as India, have a lot of distinguishing characteristics. With the rapidly increasing volume of Legal NLP applications in various countries, it has become necessary to pre-train such LMs over legal text of other countries as well. In this work, we attempt to investigate pre-training in the Indian legal domain. We re-train (continue pre-training) two popular legal PLMs, LegalBERT and CaseLawBERT, on Indian legal data, as well as train a model from scratch with a vocabulary based on Indian legal text. We apply these PLMs over three benchmark legal NLP tasks -- Legal Statute Identification from facts, Semantic Segmentation of Court Judgment Documents, and Court Appeal Judgment Prediction -- over both Indian and non-Indian (EU, UK) datasets. We observe that our approach not only enhances performance on the new domain (Indian texts) but also over the original domain (European and UK texts). We also conduct explainability experiments for a qualitative comparison of all these different PLMs.

1 INTRODUCTION

Indian legal texts differ from EU and US legal texts and may contain digitization-related errors, while existing legal PLMs have room to improve on Indian tasks. This work pre-trains legal PLMs on Indian data and evaluates them across Indian and other-country legal tasks.

1 INTRODUCTION

  • Indian legal texts differ from EU and US legal texts and can contain missing spaces and punctuation caused by typographical or scanning errors.The authors motivate pre-training on Indian legal text so models can observe these patterns.Source: §1 · p. 1
  • The study continues pre-training LegalBERT and CaseLawBERT on Indian legal data and trains CustomInLawBERT from scratch with an Indian-specific vocabulary.It evaluates the models on statute identification, judgment segmentation, and judgment prediction using Indian and non-Indian datasets.Source: §1 · p. 2
  • CustomInLawBERT comes close to the continued-pretraining models on multiple datasets despite using fewer training steps.The reported counts are 700k steps for CustomInLawBERT, 1.3M for InLegalBERT, and 2.3M for InCaseLawBERT.Source: §1 · p. 2
  • InLegalBERT outperforms the other tested PLMs in almost all end-tasks and achieves state-of-the-art results on four of five datasets.The authors report that Indian-data pre-training improves results on legal datasets from India and other countries.Source: §1 · p. 2

2 RELATED WORK

Prior work developed legal PLMs for several jurisdictions and legal document types. This study addresses the lack of common-setting comparisons and the unexplored use of existing models for other countries’ legal text.

2 RELATED WORK

  • Legal PLMs described include LegalBERT for EU, UK, and US materials, CaseLawBERT for US case law and contracts, PoLBERT for a multi-jurisdictional corpus, and Lawformer for Chinese legal text.The paper points readers to Table 1 for details of the pre-training datasets.Source: §2 · p. 2
  • Earlier work also applied transformer-based models such as BERT to legal datasets across multiple countries.The paper notes scope to improve on general-domain BERT for distinctive legal text.Source: §2 · p. 2
  • Prior work had not compared these PLMs in a common setting or explored how to adapt them for legal text from other countries.The paper frames its Indian legal text case study as addressing these gaps.Source: §2 · p. 2

3 DATA COLLECTION

The study assembled Indian court judgments and Central Government Acts as its pre-training corpus. The judgments span multiple courts and decades, and the text underwent artifact removal during preprocessing.

3 DATA COLLECTION

  • Court judgments were collected from 29 Indian courts: the Supreme Court, 26 High Courts, and two District Courts.Sources included court websites and legal repositories.Source: §3 · p. 2
  • The collection includes 1,113 Central Government Acts containing 32,021 Sections.The paper describes Acts as documents codifying the country's laws, with each Act comprising related laws called Sections.Source: §3 · p. 3
  • The corpus contains around 5.4 million documents and 27GB of raw text.Table 1 compares this pre-training dataset with datasets used in prior legal-domain work.Source: §3 · p. 3, Table 1 · p. 3
  • The cases range primarily from 1950 to 2019, cover multiple legal domains, and are heavily concentrated in recent decades.A few cases date to the 1800s.Source: §3 · p. 3
  • Preprocessing removes text artifacts such as sequences of dashes, page numbers, and metadata appearing in random positions.The authors attribute many anomalies to digitization of handwritten documents.Source: §3 · p. 3

4 PRE-TRAINING SETUP

The section develops Indian-legal PLMs through continued pre-training and a from-scratch model with a custom vocabulary, then evaluates pre-training quality on an Indian legal test set. Its setup reflects vocabulary differences in Indian legal text and uses chunked documents with MLM and NSP objectives.

4 PRE-TRAINING SETUP

  • The models start from LegalBERT or CaseLawBERT for continued pre-training, or train BERT-base from scratch with the custom Indian-legal vocabulary.The resulting models are named InLegalBERT, InCaseLawBERT, and CustomInLawBERT, respectively.Source: §4 · p. 3

4.1 Custom Vocabulary for Indian Legal Data

  • Indian legal text uses local terms and salutations, motivating a custom vocabulary; all three vocabularies share many tokens, yet each has substantial unique tokens.The custom tokenizer uses WordPiece, trained on 10% of the training data with a 30,522-token limit.Source: §4.1 · p. 3

4.2 Creating training and testing examples for pre-training

  • Documents are randomly split 9:1 while preserving sentence-count length distributions, then divided into equal-sized contiguous chunks regardless of sentence boundaries.This yields 21.6M training examples and 2.4M test examples; training chunks are limited to 254 tokens.Source: §4.2 · pp. 3–4, §4.3 · p. 4

4.3 Training Objectives

  • Pre-training uses MLM and NSP with dynamic masking and sampling; whole-word masking selects 15% of tokens and applies the 80/10/10 replacement policy.Continued pre-training runs for 300K steps and training from scratch for 700K steps, with the paper noting that limited compute constrained the training effort.Source: §4.3 · p. 4

4.5 Evaluation of pre-training

  • InLegalBERT and InCaseLawBERT have lower perplexity than their original models on the Indian test set; CustomInLawBERT also outperforms BERT.InLegalBERT is reported as more adaptable to the Indian setting than InCaseLawBERT, while CustomInLawBERT does not reach the other legal PLMs’ perplexities.Source: §4.5 · p. 5, Table 2 · p. 5

5 APPLICATION ON END-TASKS

The paper applies the PLMs to benchmark legal tasks on Indian and other-country legal text. It uses the models as BERT-based encoders for fair comparison and includes standard language-model baselines.

5 APPLICATION ON END-TASKS

  • The models are fine-tuned on practically important benchmark tasks over Indian legal text and text from other countries.The authors note that the tasks may ultimately require more complex architectures, but those architectures contain a BERT-based text encoder.Source: §5 · p. 5
  • Different PLMs are compared as the encoder module, keeping the encoder role consistent across the end-task evaluations.The models encode text either in parts or in full, depending on the task architecture.Source: §5 · p. 5
  • BERT-base, the original LegalBERT, and the original CaseLawBERT serve as baselines.The two legal baselines use the same architecture as BERT-base.Source: §5 · p. 5

5.1 The fine-tuning architecture used for all end-tasks

All end-task variants use HierBERT with the PLM as a lower-level encoder, while the remaining architecture stays fixed for comparison. Documents are processed in sentence or chunk units, with task-specific output requirements and implementation choices.

5.1 The fine-tuning architecture used for all end-tasks

  • Documents longer than BERT’s 512-token limit are split into sentences or chunks, and tasks use either one document-level representation or representations for every sentence or chunk.Some datasets already provide sentence boundaries; others are chunked.Source: §5.1 · pp. 5–6
  • HierBERT encodes each sentence or chunk with a lower-level BERT, then an LSTM contextualizes the resulting [CLS] embeddings and attention produces a document embedding.The rest of the architecture is held constant across PLM variants for a fair comparison.Source: §5.1 · pp. 5–6

5.2.2 Implementation.

  • For ILSI and ECtHR-B, the vanilla HierBERT model classifies document embeddings with a sigmoid feed-forward network, using at most 128 tokens per sentence and 128 sentences per document.The implementation feeds the first 128 sentences to HierBERT after truncating each sentence to 128 tokens.Source: §5.2.2 · p. 6
  • The study compares its results with those reported in the original dataset papers and uses the Wilcoxon Signed Rank Test to assess significance.Table 3 covers macro-averaged percentage performance on ILSI and ECtHR-B for Legal Statute Identification.Source: §5.2.2 · p. 6, Table 3 · p. 6

5.2 Legal Statute Identification (LSI)

The section evaluates legal statute identification on Indian ILSI and European ECtHR-B cases. InLegalBERT improves results on both, while dataset-specific pretraining and vocabulary affect which models perform best.

5.2.1 Datasets.

  • The datasets comprise 65K Indian criminal-case examples labeled with 100 Indian Penal Code statutes and 11K European Court of Human Rights facts labeled with 10 convention articles.Both datasets have train, development, and test splits.Source: §5.2.1 · p. 6

5.2.3 Results.

  • The study compares macro-precision, macro-recall, and macro-F1 across models that share the same architecture apart from their PLM encoders.Training used 25 epochs with early stopping and weighted binary cross-entropy; hyperparameters were selected using validation performance.Source: §5.2.3 · p. 6, §5.2.1 · p. 6
  • LegalBERT has the highest ILSI precision, while CaseLawBERT exceeds LegalBERT on ILSI F1; on ECtHR-B, LegalBERT outperforms CaseLawBERT.The authors suggest LegalBERT's ECtHR training may explain its ECtHR-B advantage.Source: §5.2.3 · p. 6
  • InLegalBERT improves ILSI recall and F1, and leads all metrics on ECtHR-B; it reaches 75.88% macro-F1 versus the prior 74.7% best.On ILSI, its precision is slightly below vanilla LegalBERT; the reported models also surpass the prior ILSI macro-F1 of 28%.Source: §5.2.3 · pp. 6–7

5.3 Semantic Segmentation of case judgements

The section evaluates sentence-level semantic segmentation of Indian and UK judgments. InLegalBERT achieves the best macro-F1 on both datasets, though the strongest reported ISS result comes from a Sent2vec-based method.

5.3.1 Datasets.

  • ISS contains 50 Indian Supreme Court judgments and 9,308 labeled sentences; UKSS contains 50 UK Supreme Court judgments and 18,155 labeled sentences, each using the same seven segment labels.The labels include Facts, Arguments, Statutes, Precedents, Ratio Decidendi, and lower- and present-court rulings.Source: §5.3.1 · p. 7

5.3.2 Implementation.

  • The task uses HierBERT with a CRF over contextualized sentence embeddings, limits sentences to 128 tokens, and evaluates both datasets with five-fold cross-validation.The reported setup uses 25 epochs with early stopping and negative log-likelihood loss.Source: §5.3.2 · p. 7

5.3.3 Results.

  • InLegalBERT achieves the highest macro-F1 among the tested PLMs on both ISS and UKSS; on UKSS it reaches 61.54%, above the prior 60% LegalBERT result.The InLegalBERT differences from other variants are statistically significant; on ISS, the reported Sent2vec method exceeds 80% macro-F1 and outperforms the transformer models.Source: §5.3.3 · p. 8
  • LegalBERT and CaseLawBERT improve over BERT, with the stronger off-the-shelf model varying by dataset; further Indian-data pretraining improves both, while CustomInLawBERT slightly trails the other legal PLMs.The authors suspect the small ISS dataset may limit effective transformer fine-tuning.Source: §5.3.3 · p. 8

5.4.2 Implementation.

  • For court judgment prediction, the ILDC text is chunked into non-overlapping 128-token groups, and the model uses the last 128 chunks.This is the implementation choice described for the separate ILDC experiment.Source: §5.4.2 · p. 8

5.4 Court Judgment Prediction

The section studies predicting whether an Indian court accepts or rejects an appeal. InLegalBERT achieves a statistically significant macro-F1 advantage over the other tested models, and all five legal PLMs surpass the prior reported result.

5.4 Court Judgment Prediction.

  • Court judgment prediction uses the judgment without its final decision to predict acceptance or rejection of the appellant's claims as a binary classification task.The ILDC-multi dataset consists of approximately 35K Supreme Court of India documents, each with multiple claims or appeals.Source: §5.4 · p. 8, §5.4.1 · p. 8

5.4.3 Results.

  • InLegalBERT's macro-F1 is statistically significantly higher than every other tested model and the original-paper results; all five legal PLMs outperform Malik et al..The prior work used the same broad architecture with XLNet as its lower encoder, while the authors suggest legal knowledge in the tested PLMs may explain the gains.Source: §5.4.3 · p. 8
  • The experiment uses HierBERT with a feed-forward sigmoid layer and a single output unit for classification.The model comparison reports macro-precision, macro-recall, and macro-F1.Source: §5.4.3 · p. 8

5.5 Analysis of Explainabilty of Models

The explainability analysis compares models’ attention to spans marked important by legal experts on a 56-document test subset. InLegalBERT and InCaseLawBERT generally show closer agreement with expert annotations than their original counterparts, though agreement does not consistently track macro-F1.

5.5.1 Dataset.

  • The ILDC-expert set contains 56 test documents with character-level spans annotated by five legal experts.Each document has five annotation sets, one per expert.Source: §5.5.1 · p. 9

5.5.2 Method.

  • Models assign attention weights to document chunks, which are compared with expert importance distributions derived from annotated character spans.Chunks depend on each model’s tokenizer; lower KL divergence indicates a closer match between model and expert distributions.Source: §5.5.2 · p. 9, §5.5.1 · p. 9

5.5.3 Results.

  • InLegalBERT has lower KL divergence than LegalBERT for 4 of 5 experts, while InCaseLawBERT is lower than CaseLawBERT for all 5.Both re-trained models also score lower than CustomInLawBERT for a majority of experts.Source: §5.5.3 · p. 9, Table 6 · p. 9
  • Expert agreement and macro-F1 do not rank models identically: InCaseLawBERT agrees more with experts than InLegalBERT despite lower macro-F1, and CaseLawBERT has poor KL scores despite decent macro-F1.Macro-F1 is measured on ILDC-multi’s test set, whereas KL divergence uses the smaller ILDC-expert subset.Source: §5.5.3 · p. 9

6 CONCLUSION AND FUTURE WORK

The paper concludes that continuing pre-training on Indian legal text generally improves existing legal PLMs across three tasks and datasets from India and other countries. InLegalBERT shows the strongest reported results, while the comparison of approaches depends partly on training budget.

6 CONCLUSION AND FUTURE WORK

  • Indian legal-domain pre-training generally improves LegalBERT and CaseLawBERT across three legal NLP tasks and datasets from India and other countries.The tasks are Legal Statute Identification, Semantic Segmentation, and Court Judgement Prediction.Source: §6 · p. 9
  • InLegalBERT improves significantly over LegalBERT on Indian data, while InCaseLawBERT’s gains over CaseLawBERT are much smaller.InLegalBERT achieves the best F1 scores across all three tasks and five datasets.Source: §6 · pp. 9–10
  • InLegalBERT also improves performance on old EU data and outperforms original-paper results on 4 of 5 datasets.Source: §6 · p. 10
  • CustomInLawBERT comes close to the re-trained models in macro-F1 despite using 700k training steps, versus 1.3M and 2.3M steps.Source: §6 · p. 10
  • The choice between re-training and training from scratch depends on factors such as pre-training data size and computing resources; re-training may be promising when steps are limited.Training from scratch can underperform with a limited pre-training budget, while re-trained models retain legal data from multiple countries.Source: §6 · p. 10

Related papers

Loading 2209.06049v5…