PaperlayerView exact revision on arXiv

Source-linked AI summary

SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval

Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen, Qi Tian

arXiv:2304.11370v1cs.IRcs.CL

TL;DR

Legal case retrieval needs models that handle long, structured documents and key legal elements. SAILER uses structure-aware pre-training to represent legal cases. Across four legal benchmarks, it significantly improves over baselines in low- and full-resource settings.

  • Problem

    Existing PLMs have limited long-document capacity or neglect legal case structure, while retrieval systems are important for finding relevant precedents.

    Source: §1 · pp. 1–2

  • Method

    SAILER uses an encoder-decoder to encode case facts and reconstruct masked reasoning and decision text, leveraging document structure and key legal elements.

    Source: §1 · p. 2

  • Results

    Across four legal benchmarks, SAILER significantly improves over baselines in low- and full-resource settings.

    Source: §6 · p. 9

  • Takeaways & Limitations

    SAILER generates legal case representations with stronger discriminative ability by reconstructing key legal elements and judgment results.

    Source: §6 · p. 9

    On COLIEE2020, SAILER's improvement over SEED and RetroMAE is not very significant, and the limited test cases constrain significance testing.

    Source: §5.4.2 · p. 8

Abstract

from arXiv · show

Legal case retrieval, which aims to find relevant cases for a query case, plays a core role in the intelligent legal system. Despite the success that pre-training has achieved in ad-hoc retrieval tasks, effective pre-training strategies for legal case retrieval remain to be explored. Compared with general documents, legal case documents are typically long text sequences with intrinsic logical structures. However, most existing language models have difficulty understanding the long-distance dependencies between different structures. Moreover, in contrast to the general retrieval, the relevance in the legal domain is sensitive to key legal elements. Even subtle differences in key legal elements can significantly affect the judgement of relevance. However, existing pre-trained language models designed for general purposes have not been equipped to handle legal elements. To address these issues, in this paper, we propose SAILER, a new Structure-Aware pre-traIned language model for LEgal case Retrieval. It is highlighted in the following three aspects: (1) SAILER fully utilizes the structural information contained in legal case documents and pays more attention to key legal elements, similar to how legal experts browse legal case documents. (2) SAILER employs an asymmetric encoder-decoder architecture to integrate several different pre-training objectives. In this way, rich semantic information across tasks is encoded into dense vectors. (3) SAILER has powerful discriminative ability, even without any legal annotation data. It can distinguish legal cases with different charges accurately. Extensive experiments over publicly available legal benchmarks demonstrate that our approach can significantly outperform previous state-of-the-art methods in legal case retrieval.

1 INTRODUCTION

SAILER addresses legal case retrieval’s challenges of long, structured documents and relevance that depends on key legal elements, using structure-aware pre-training; experiments report improvements over state-of-the-art baselines.

1 INTRODUCTION

  • Growing legal-case corpora make finding relevant precedents increasingly challenging, while high-quality annotations are labor-intensive and require domain expertise.These constraints motivate retrieval methods that can work without large-scale supervised annotations.Source: §1 · p. 1
  • Legal cases contain long, correlated sections whose writing logic vanilla PLMs may miss because of limited text capacity or neglected structure.The described sections are Procedure, Fact, Reasoning, Decision, and Tail, each ranging from hundreds to thousands of words.Source: §1 · p. 2
  • Legal relevance depends on key circumstances and their legal abstractions, so cases with similar wording may still warrant different judgments.Without guidance, open-domain PLMs may fail to understand these elements and retrieve suboptimally.Source: §1 · p. 2, Table 1 · p. 2
  • SAILER encodes each case’s Fact into a dense vector, then uses two shallow decoders to reconstruct masked Reasoning and Decision text.This encoder-decoder framework uses those structures to capture dependencies and legal knowledge; the paper presents it as the first pre-training work to exploit legal-case structure.Source: §1 · p. 2
  • SAILER significantly improves over state-of-the-art baselines on Chinese and English legal benchmarks in zero-shot and fine-tuning settings.The experiments also demonstrate benefits from modeling long-distance dependencies and using structural knowledge.Source: §1 · p. 2

2 RELATED WORK

Prior work has advanced dense retrieval through representation-focused pre-training and developed a range of legal case retrieval models. The paper identifies gaps in how these approaches capture legal document structure and retrieval-specific legal relevance.

2.1 Pre-trained Language Models for Dense Retrieval

  • Dense-retrieval pre-training improves [CLS] representations through encoder skip connections and autoencoder reconstruction, including Condenser [14], coCondenser [15], SEED-Encoder [21], SIMIM [32], and RetroMAE [19].Autoencoder approaches reconstruct input sentences, with methods varying their decoder design and information bottleneck.Source: §2.1 · p. 2
  • These autoencoder models may miss logical relationships among legal case structures and weaken discrimination by reconstructing facts irrelevant to case relevance.Source: §2.1 · p. 2

2.2 Legal Case Retrieval

  • Legal case retrieval includes expert-knowledge models, such as Zeng et al.’s sub-elements, and NLP models, including paragraph-level BERT ranking and text-network similarity [4].Source: §2.2 · p. 3
  • LEGAL-BERT [5] and Lawformer [34] address legal text broadly or at length, but neither uses pre-training objectives tailored to legal case retrieval.Lawformer can process documents with thousands of tokens; the authors argue pre-trained models’ potential for legal case retrieval remains underexploited.Source: §2.2 · p. 3

3 BACKGROUND AND PRELIMINARIES

The section defines legal case retrieval as ranking relevant candidate cases for a query and explains why legal documents’ structure and the links among Fact, Reasoning, and Decision matter for modeling.

3.1 Problem Formulation

  • Legal case retrieval ranks the top-k relevant cases from a candidate set for a given query case.The paper assumes the query contains only the case’s basic Fact, while candidates are complete case documents; a dual encoder maps query and document text to embeddings and scores them by dot product.Source: §3.1 · p. 3
  • Legal case retrieval uses a dual encoder whose query and document embeddings are compared by dot product.The encoders are typically implemented or initialized with a pre-trained language model that produces an embedding such as the [CLS] representation.Source: §3.1 · p. 3

3.2 Preliminary

  • Legal cases have structured components such as Procedure, Fact, Reasoning, Decision, and Tail, though Chinese cases may convey these sections implicitly through formatting.Fact describes arguments, evidence, and events; Reasoning applies rules to facts and repeatedly mentions key legal elements; Decision responds to the dispute, while Procedure and Tail provide procedural and court information.Source: §3.2 · pp. 3–4
  • Judges use the Fact to identify relevant law and cases, then explain rule application in Reasoning before making a Decision.Important facts are analyzed in Reasoning and affect the final decision, making the logical connection among Fact, Reasoning, and Decision important to modeling legal cases.Source: §3.2 · p. 4, Figure 2 · p. 4

4 METHOD

SAILER uses a fact encoder and two decoders to reconstruct legal-case reasoning and decisions, encouraging fact representations to capture key legal elements and support discrimination between cases.

4 METHOD

  • SAILER pairs a BERT-like Fact Encoder with shallow Reasoning and Decision Decoders that reconstruct the corresponding sections from the encoded facts.This asymmetric encoder-decoder design simulates the writing process of legal case documents.Source: §4 · p. 4, Figure 3 · p. 4

4.1 Fact Encoder

  • The Fact Encoder masks a small share of fact tokens, encodes the masked facts as a [CLS] vector, and learns to recover masked tokens with MLM.Source: §4.1 · p. 4

4.2 Reasoning Decoder

  • The Reasoning Decoder uses the fact vector to reconstruct masked reasoning, directing it toward relevant legal elements and long-distance dependencies between Facts and Reasoning.It masks 30%-60% of the Reasoning section for reconstruction.Source: §4.2 · p. 5

4.3 Decision Decoder

  • The Decision Decoder models legal judgment prediction as text-to-text reconstruction, helping produce vectors that discriminate between cases with different judgments.For Chinese cases, it masks formatted law articles, charges, and penalty terms; where decisions lack a specific format, it masks high-TF-IDF words.Source: §4.3 · p. 5

4.4 Learning

  • After pre-training, SAILER drops both decoders and fine-tunes the encoder to bring query cases closer to relevant cases than irrelevant ones.Training uses BM25 hard negatives and in-batch negatives for contrastive learning.Source: §4.4 · p. 5

5 EXPERIMENT · 5.1 Datasets and Metrics

The experiments evaluate SAILER on four legal case retrieval benchmarks using task-specific training protocols and competition evaluation metrics.

5 EXPERIMENT

  • The experiment section introduces the setup and reports results to demonstrate SAILER’s effectiveness.Source: §5 · p. 5

5.1 Datasets and Metrics

  • Four legal case retrieval benchmarks are used, with dataset statistics summarized in Table 2.Source: §5.1 · p. 5, Table 2 · p. 6
  • LeCaRD uses fact-paragraph queries, entire-case candidates, and 100 candidates per query; CAIL2022-LCR follows the same format.Source: §5.1 · p. 5
  • COLIEE2020 provides 200 candidates per query for reranking, whereas COLIEE2021 requires retrieval from the entire corpus without a candidate pool.The paper characterizes COLIEE2021 as more difficult than COLIEE2020.Source: §5.1 · p. 5
  • LeCaRD and CAIL2022-LCR use zero-shot evaluation; COLIEE2020 and COLIEE2021 models are fine-tuned on their combined training sets and evaluated on their test sets.The zero-shot choice reflects the highly limited number of annotated queries for LeCaRD and CAIL2022-LCR.Source: §5.1 · p. 5
  • LeCaRD and CAIL2022-LCR are evaluated with NDCG, Precision@5, Recall@5, and F1; COLIEE tasks use MRR, Precision@5, Recall@5, and F1, with R@100 also reported for COLIEE2021.Source: §5.1 · p. 5

5.2 Baselines

The comparison uses traditional retrieval, generic pre-trained, and retrieval-oriented pre-trained baselines, with different initialization and pre-training procedures for each group.

5.2 Baselines

  • BM25 and QL represent traditional retrieval, using exact word matching and Dirichlet smoothing, respectively.Source: §5.2 · p. 6
  • Generic baselines include BERT [9], BERT_xs, and RoBERTa [18], spanning ad-hoc retrieval, criminal-law pre-training, and enlarged-data MLM-only pre-training.BERT_xs was pre-trained with 663 million Chinese criminal judgments.Source: §5.2 · p. 6
  • Legal-domain models include Lawformer [34], the first Chinese pre-trained language model in the legal domain focused on processing long documents, and LEGAL-BERT [5], described as a pre-trained language model.Source: §5.2 · p. 6
  • Retrieval-oriented baselines comprise Condenser [14], coCondenser [15], SEED [21], CoT-MAE [33], and RetroMAE [19], with dense-vector, decoder, and contrastive-learning designs.Condenser aggregates text information into dense vectors; coCondenser adds corpus-level contrastive loss, while SEED and CoT-MAE use asymmetric encoder-decoder approaches and RetroMAE strengthens decoding.Source: §5.2 · p. 6
  • Traditional models use pyserini defaults, generic models use open-source checkpoints, and retrieval-oriented models are BERT-initialized and pre-trained on SAILER’s legal corpus with their papers’ best parameters.Source: §5.2 · p. 6

5.3 Implementation Details

SAILER’s implementation uses case-only Chinese and English pre-training corpora, structure-specific masking, and BM25-based fine-tuning with hard negatives.

5.3 Implementation Details

  • The Chinese corpus contains tens of millions of cases divided into five sections, while the English corpus uses U.S. federal and state court cases.Chinese cases with facts under 50 words are filtered; unlike LEGAL-BERT, the pre-training corpus excludes legislation and contracts.Source: §5.3.1 · p. 6

5.3.1 Pre-training Procedures.

  • SAILER initializes its encoders from Chinese-BERT-WWM and BERT-base-uncased, uses randomly initialized transformer decoders, and applies different decision masking by language.Pre-training runs up to 10 epochs with AdamW, a 1e-5 learning rate, batch size 72, and encoder/decoder mask ratios of 0.15/0.45; Chinese decision masking targets law articles, charges, and penalty terms, while English masks high-TF-IDF words. The English Decision Decoder uses the Reasoning Decoder’s mask ratio.Source: §5.3.1 · p. 6
  • Fine-tuning recalls 100 candidates per query with BM25, treats irrelevant documents as hard negatives, and trains on batches with a 1:15 positive-to-negative ratio.Fine-tuning runs up to 20 epochs with AdamW, a 5e-6 learning rate, batch size 4, and linear scheduling with 0.1 warmup; experiments use eight NVIDIA Tesla A100 GPUs.Source: §5.3.2 · p. 6

5.4 Experiment Result

In zero-shot evaluation, SAILER leads on most metrics across both datasets and is the only model reported to outperform traditional retrieval methods. With fine-tuning, it improves over all baselines, though traditional methods remain competitive and the COLIEE2020 comparison has limited significance evidence.

5.4.1 Zero-Shot Evaluation.

  • Retrieval-oriented pre-trained models generally outperform generic models, while BM25 and QL remain competitive and generic models usually trail lexical matching.The authors suggest generic pre-training does not adequately capture legal retrieval relevance.Source: §5.4.1 · p. 7
  • Without supervised data, SAILER achieves the best performance on most metrics across both datasets, significantly surpasses other pre-trained models, and is the only model to beat traditional retrieval methods.The authors attribute this to incorporating knowledge from case structure during pre-training.Source: §5.4.1 · pp. 7–8, Table 3 · p. 7
  • Chinese RoBERTa outperforms models trained on extensive criminal-law data; the authors suggest omitting NSP may improve [CLS] robustness and retrieval performance.They present this as a possible explanation, not a demonstrated mechanism.Source: §5.4.1 · p. 7

5.4.2 Fine-tuning Evaluation.

  • With fine-tuning, SAILER improves over all baselines, outperforms RetroMAE without enhanced decoding, and is the only pre-trained model above QL at Recall@100 on COLIEE2021.The authors attribute its gains to using case structure to infer expert knowledge through contextual tokens.Source: Table 4 · p. 7, §5.4.2 · p. 8
  • On COLIEE2020, SAILER achieves state-of-the-art metric values, but its gains over SEED and RetroMAE are not very significant; prior work did not test significance because test cases were limited [2].The authors note that non-significant improvements can still indicate effectiveness when similarly sized fine-tuned models perform closely.Source: §5.4.2 · p. 8

5.5 Ablation Studies · 5.6 Hyperparameter Analysis

Ablations support SAILER’s multi-decoder architecture and pre-training objectives, while hyperparameter tests show that performance depends on masking choices but remains robust across decoder depths.

5.5 Ablation Studies

  • Removing both decoders dramatically reduces performance, while removing either the Reasoning Decoder or Decision Decoder significantly reduces it.The Reasoning Decoder’s removal causes less degradation than the Decision Decoder’s removal; the authors suggest random masking may miss key legal elements.Source: §5.5 · p. 8
  • Sharing a decoder for reasoning and decision reconstruction slightly degrades performance, supporting SAILER’s multi-decoder architecture.The shared-decoder variant is SAILER_share.Source: §5.5 · p. 8

5.6 Hyperparameter Analysis

  • Tests on LeCaRD varied the encoder masking ratio from 0 to 0.30 and the decoder masking ratio from 0.15 to 0.60.Source: §5.6.1 · p. 8

5.6.1 Impact of Mask Ratio.

  • Decoder masking improves performance to some extent, but becomes unstable above 0.45; excessive encoder masking slightly reduces performance by discarding useful input information.An appropriate encoder mask ratio helps performance, and SAILER performs well across a wide range of mask ratios.Source: §5.6.1 · p. 8
  • At encoder and decoder masking ratios of 0.15 and 0.45, respectively, SAILER achieves its best performance.Source: §5.6.1 · p. 8

5.6.2 Impact of Decoder Layer Number.

  • Increasing the number of decoder layers decreases performance, but SAILER remains robust across decoder-layer settings.The authors note a similar finding for SEED [21].Source: §5.6.2 · pp. 8–9

5.7 Case Study · 5.8 Visual Analysis

The visual analyses suggest that SAILER attends to legally decisive elements and separates cases with confusing charges without supervision. Compared with SEED, it focuses more on judgment-relevant words, including “allow,” a keyword in the illustrated charge.

5.7 Case Study

  • SAILER emphasizes words affecting the judgment, such as “sold,” “drug,” and “allowed,” unlike SEED’s focus on names and places.The comparison uses SEED because it reconstructs the original text with the same decoding mechanism as SAILER; darker words indicate higher attention.Source: §5.7 · p. 9
  • SAILER emphasizes “allow,” a key term for the charge of providing a venue where drug users can take drugs.The analysis interprets this attention as evidence that SAILER incorporates legal knowledge and focuses on key legal elements.Source: §5.7 · p. 9

5.8 Visual Analysis

  • Chinese BERT mixes cases from different charges, while Chinese RoBERTa separates Theft, Robbery, and Defraud but confuses three other charges.The confused charges are Intentional Injury, Provocation, and Affray.Source: §5.8 · p. 9, Figure 5 · p. 9
  • SEED produces a uniform vector distribution, but its boundaries between similar charges are too close for legal case retrieval.SEED reconstructs the original text, learning more information than Chinese BERT in this analysis.Source: §5.8 · p. 9, Figure 5 · p. 9
  • Without supervision, SAILER more accurately separates vectors for six easily confused charges than the compared models.The t-SNE visualization uses randomly selected cases from the pre-training corpus and vectors from off-the-shelf pre-trained models in a zero-shot setting.Source: §5.8 · p. 9, Figure 5 · p. 9

6 CONCLUSION

SAILER uses legal-document structure, key legal elements, and judgment results in pre-training to improve legal case representations. Experiments on four benchmark datasets show significant improvements over baselines in low- and full-resource settings.

6 CONCLUSION

  • SAILER significantly improves over baselines on four legal benchmarks in both low-resource and full-resource settings.The paper proposes exploring legal knowledge graphs and law articles as additional expert knowledge for future pre-training work.Source: §6 · p. 9
  • SAILER uses legal-document structure in pre-training, reconstructing key legal elements and judgment results to generate better case representations and stronger discrimination.Source: §6 · p. 9
Loading 2304.11370v1…