Assessing the Performance Gap Between Lexical and Semantic Models for Information Retrieval With Formulaic Legal Language

1 Introduction

• Passage retrieval improves specificity for legal topics, saves practitioners time, and avoids processing entire decisions that may exceed model context windows.Case-level retrieval can mix procedural and substantive issues, while long documents create computational difficulties for BERT-based models.Source: §1 · p. 1
• CJEU citations point to specific paragraphs, making paragraph retrieval a natural task rather than whole-case retrieval.Citing paragraphs often quote or closely paraphrase the cited paragraph.Source: §1 · pp. 1–2
• Formulaic judicial phrases and repeated citation patterns make CJEU language especially routinized compared with several other courts.These formulas support uniformity, faster drafting, and searching for similar cases.Source: §1 · p. 2
• The study asks when lexical methods such as BM25 outperform dense models in formulaic legal language, balancing retrieval effectiveness against computational cost.BM25 relies on term frequency, whereas dense models use large parameterized models and substantial computation.Source: §1 · p. 2
• BM25 appears better when query–target overlap is lower, while dense models may miss relevance because contextual information introduces bias or noise beyond direct quotes.The comparison uses qualitative and quantitative analysis with three complementary metrics.Source: §1 · p. 2

2.1 Legal passage retrieval

• Legal passage retrieval has gained attention because increasing numbers of decisions make efficient legal research more necessary.Source: §2.1 · p. 2
• Existing datasets vary between retrieving passages, retrieving case citations, and retrieving entire cases from a factual case description.LePaRD retrieves passages, CLERC retrieves citations, and the European Court of Human Rights dataset retrieves whole cases.Source: §2.1 · p. 2
• The CJEU dataset was the first known European Union Law resource for legal paragraph retrieval, and this study expands its baselines from three methods and one metric.The extension evaluates BM25, additional models, and more evaluation metrics while analyzing repetitive CJEU citation language.Source: §2.1 · p. 2

3.1 Dataset

• The dataset contains manually extracted CJEU paragraph-to-paragraph citations from February 1978 to October 2021, with missing-text and French paragraphs removed.The preprocessing removed 269 paragraphs missing text and 4,851 French paragraphs.Source: §3.1 · p. 2
• Training, validation, and test sets are split temporally, with candidate paragraphs drawn from the training and validation periods for each test query.The split targets approximately 80%, 10%, and 10% of citations.Source: §3.1 · pp. 2–3
• Relevant passages are defined as paragraphs cited by each query, but excluding surrounding context makes reported performance a lower bound.Additional context around citing and cited paragraphs might enable more effective retrieval.Source: §3.1 · p. 3

3.2.1 Lexical models.

• BM25 combines query-term inverse document frequency with term frequency and document-length adjustment, whereas TF-IDF omits term-frequency scaling and length normalization.Dense methods score candidates using cosine similarity between query and candidate embeddings.Source: §3.2.1 · p. 3, §3.2.2 · p. 3

3.2 Methods

• The comparison includes BM25, TF-IDF, five zero-shot dense models, and two CJEU-fine-tuned dense models.LegalSBERT-ft fine-tunes a 110M-parameter LegalBERT model, while SBERT-ft fine-tunes a 33M-parameter all-MiniLM-L6-v2 model.Source: §3.2.2 · p. 3, §3.2.1 · p. 3

3.3 Evaluation metrics

• Evaluation uses Recall@1, @5, @10, and @20, nDCG@10, MAP, and MRR to measure top-result recall, ranking quality, and global ranking.This expands the original dataset paper’s evaluation from one metric to seven.Source: §3.3 · pp. 3–4

4 Performance comparison of lexical and dense models

• BM25 outperforms TF-IDF across all metrics, while TF-IDF remains stronger than zero-shot dense models except Ada-v2 on most comparisons.For Recall@20, TF-IDF-1gram scores 76.74% versus 75.47% for Ada-v2.Source: §4 · p. 4
• BM25 outperforms zero-shot dense models on Recall@5, @10, and @20 and nDCG@10, while Ada-v2 surpasses it on Recall@1, MAP, and MRR.For Recall@5, BM25 reaches 63.63% versus 61.28% for the best zero-shot model, Ada-v2.Source: §4 · p. 4
• Ada-v2 is the strongest zero-shot dense model across the evaluated models, outperforming all others on every metric.Source: §4 · p. 4
• LegalSBERT-ft outperforms BM25 and every other method across all metrics, while SBERT-ft is second-best except on Recall@20.On Recall@20, SBERT-ft scores 79.42% versus 80.43% for BM25.Source: §4 · p. 4
• Fine-tuning enables a 33-million-parameter SBERT-ft model to outperform models four times larger, but it requires substantial computational resources and does not always beat BM25.The exception reported here is Recall@20.Source: §4 · p. 4

5.1 Qualitative analysis

• BM25 also outperforms SBERT-ft in examples with little or no substantial verbatim overlap, including queries sharing only five short expressions or none.By contrast, two of three shared-success examples contained verbatim quotes of the cited paragraph.Source: §5.1 · p. 5
• Both lexical and dense models perform well when queries closely reproduce cited paragraphs through verbatim quotation or close paraphrase.In the shared-success cases, queries were often entirely direct quotes or close paraphrases of the cited text.Source: §5.1 · pp. 4–5
• BM25 outperforms Ada-v2 when queries contain only partial overlap with cited paragraphs, whereas dense retrieval can be distracted by additional contextual information.In one example, Ada-v2 returned four of five paragraphs about a different topic discussed in the query.Source: §5.1 · pp. 5–6

5.2 Quantitative analysis

• 56.19 versus 41.90 shared 3-grams and 35.85 versus 23.14 shared 10-grams distinguish shared perfect recall from BM25-only success, respectively.For every N from 2 through 10, the perfect-recall scenario had significantly more shared N-grams; Figure 2 visualizes the comparison and standard deviations.Source: §5.2 · p. 6, Figure 2 · p. 6
• 98.52 versus 66.81 mean word edits and 67.59 versus 53.68 mean LCS likewise indicate less textual overlap when BM25 succeeds and Ada-v2 fails.The three complementary metrics consistently indicate fewer verbatim quotes and paraphrases in BM25-only successes.Source: §5.2 · p. 6
• BM25 performs better for longer queries: the mean length is 102.37 words when it outperforms dense retrieval versus 94.33 words when both models have perfect recall.The difference between the means is statistically significant at the 5% level, consistent with noise from additional information affecting dense retrieval.Source: §5.2 · p. 6

Comparison with BM25

• Figure 4 compares Recall@1, Recall@5, Recall@10, and Recall@20 for SBERT-ft models trained on different data percentages against BM25 on the 2019–2021 test set.The comparison covers the fine-tuning-data ablation and its BM25 baseline.Source: Figure 4 · p. 7, §6 · p. 7

Increasing training data generally improves performance

• 64.21% to 67.29%: Recall@5 rises as fine-tuning data increases from 10% to 80%, a 3.08% absolute increase.The increase from zero-shot to 10% fine-tuning is larger, at 8.8% absolute.Source: §6 · p. 7
• Performance increases tend to plateau as the amount of fine-tuning data grows, and higher percentages do not always outperform lower ones.Recall@1 is 42.13% with 50% of the data versus 41.8% with 80%, although randomness may affect small differences.Source: §6 · p. 7, Table 6 · p. 7

Comparison with BM25

• Fine-tuned SBERT-ft models outperform BM25 at most data percentages, especially with 70% and 80% fine-tuning.BM25 remains competitive at lower fine-tuning percentages and on some higher-recall metrics.Source: §6 · p. 7

• Verbatim overlap between queries and cited passages is one factor characterizing performance gaps between lexical and dense retrieval models.Source: §7 · p. 8
• BM25 outperforms off-the-shelf dense models on several metrics, but its advantage diminishes against dense models fine-tuned on domain-specific data.Source: §7 · p. 8
• The study analyzes how fine-tuning data volume affects dense-model performance and temporal robustness in CJEU retrieval.Source: §7 · p. 8
• Future work proposes hybrid retrieval, including BM25 coarse retrieval followed by dense-model refinement for precision ranking.Source: §7 · p. 8

• Citation-based relevance favors lexical matching because citing paragraphs often quote or paraphrase cited content, unlike typical practitioner searches.Source: §8 · p. 8
• The dataset omits surrounding context because it contains only paragraphs that cite or are cited, limiting broader contextual relevance assessment.Source: §8 · p. 8
• No hyperparameter grid search was performed for fine-tuned SBERT models, so their reported performance is a lower bound.Source: §8 · p. 8

9 Ethical Considerations

• Automated analysis may inadvertently reinforce objectionable biases and time-constrained societal norms embedded in historical judicial decisions.Source: §9 · p. 8

A Further dataset statistics

• Each decision with at least one citation averages 13.09 outbound citations, while each cited decision averages 12.31 inbound citations.Source: A Further dataset statistics · p. 9

B Model hyperparameters

• Dense-model parameter counts and embedding dimensions are tabulated, while BM25 and TF-IDF are omitted because those quantities do not apply.Source: B Model hyperparameters · p. 9, Table 8 · p. 9

C Qualitative examples of BM25 vs. dense retrieval models

• Qualitative examples compare BM25 with Ada-v2 and SBERT-ft by highlighting lexical overlap between citing and cited paragraphs across perfect-recall and BM25-superiority scenarios.Source: C Qualitative examples of BM25 vs.… · p. 9, Table 10 · p. 11, Table 11 · p. 12

D Qualitative example of top candidates retrieved for Ada-v2

• Ada-v2’s retrieved candidates include passages addressing European arrest warrants, detention conditions, fundamental rights, and related precedents.Source: D Qualitative example of top… · pp. 9, 14

• Higher N-gram overlap characterizes cases where BM25 and SBERT-ft both achieve perfect Recall@20, unlike cases where BM25 succeeds and SBERT-ft fails.The difference is statistically significant for every N from 2 through 10.Source: E Quantitative analysis for BM25 and… · p. 14, Figure 5 · p. 15
• Mean edit distance is 67.23 versus 109.09, while LCS is 65.86 versus 59.23, comparing joint perfect recall with BM25-only success.Both comparisons are statistically significant and indicate greater query–citation similarity in the joint-success scenario.Source: E Quantitative analysis for BM25 and… · p. 14, Figure 6 · p. 15
• When BM25 outperforms a dense model, the query and cited paragraph contain fewer verbatim quotes and paraphrases than when both methods achieve perfect recall.The three complementary metrics point to the same conclusion.Source: E Quantitative analysis for BM25 and… · pp. 14–15

F Results for di!erent "ne-tuning percentages for SBERT-ft

• Models fine-tuned on more data generally perform better on the test set, but performance increases tend to plateau as the fine-tuning amount grows.Source: F Results for di!erent "ne-tuning… · p. 15
• Fine-tuning on 50% of the CJEU data performs best among the tested models, with nDCG@10 of 0.6273 and MAP of 0.5737.These values exceed 0.6259 for 70% on nDCG@10 and 0.5724 for 80% on MAP, showing plateauing gains as data increases.Source: F Results for di!erent "ne-tuning… · p. 15, Table 13 · p. 15

G Discussion of cases in which BM25 achieves 0% recall and dense models 100% recall

• When BM25 fails and Ada-v2 succeeds, queries often contain synonyms or semantic role shifts, reflecting dense models’ ability to capture meaning beyond term overlap.One example contrasts an individual being entitled to bring an action with the State being held responsible.Source: G Discussion of cases in which BM25… · p. 15

• The 2019–2021 test period may create distribution shift, limiting how well the findings generalize to other time periods.The temporal split was chosen to approximate legal practice, where practitioners retrieve precedent from past decisions.Source: H Analysis of alternative temporal… · p. 15
• On the broader 2015–2021 test set, the fine-tuned model outperforms the other models across most metrics, while BM25 leads on R@10 and R@20.The comparison uses BM25, zero-shot Ada-v2, and SBERT-ft fine-tuned on 50% of the data.Source: H Analysis of alternative temporal… · pp. 15–16, Table 14 · p. 16
• The broader-split results largely align with the trends from the original 2019–2021 test split.Source: H Analysis of alternative temporal… · p. 16, Table 14 · p. 16