Source-linked AI summary
LegalPincite: Multi-level Legal Information Retrieval Dataset
Theresia Veronika Rampisela, Henrik Palmer Olsen, Giovanni Colavizza
TL;DR
Existing legal IR datasets can leak citation information and omit most non-citing paragraphs, limiting paragraph-level retrieval evaluation. LegalPincite introduces masked queries and complete CJEU paragraph corpora across three retrieval levels, showing leakage can inflate performance by up to 55.0%.
Problem
Existing legal IR datasets leak citation information in queries and exclude non-citing paragraphs, limiting realistic paragraph-level citation retrieval evaluation.
Method
LegalPincite merges and updates existing datasets, masks citation-related query information, and includes all case paragraphs across three retrieval levels.
Results
Citation leakage increases NDCG@10 by up to 0.111 (55.0%) for paragraph-to-case retrieval, while citation-masked queries perform best for paragraph-to-paragraph retrieval.
Takeaways & Limitations
LegalPincite provides a multi-level benchmark for realistic legal citation retrieval and supports evaluating methods with citation information removed.
Takeaways & Limitations
Expert annotations cover only the top-10 results of one dense retriever, which may limit their exhaustiveness.
Abstract
from arXiv · showhide
A common task in legal Information Retrieval (IR) is to find relevant legal sources from case-law collections. While legal practice often requires pinpoint citations (pincites) to specific case paragraphs, most existing public legal IR datasets lack paragraph-level citation annotations. Yet, publicly available datasets with such information contain data leakage in the query text and exclude paragraphs that are neither citing nor cited from the corpora, creating an unrealistic and oversimplified retrieval setting, potentially leading to inflated performance. To address these limitations, we contribute a large-scale legal IR dataset constructed from Court of Justice of the European Union (CJEU) judgments. The dataset contains: (i) masked case/paragraph queries, with removed citation information; (ii) a corpus that includes all paragraphs; and (iii) case- and paragraph-level ground-truth citations, with partial human expert validation. Our dataset supports both the development and rigorous evaluation of legal IR methods, at multiple query-document levels (case-to-case, paragraph-to-case, and paragraph-to-paragraph retrieval). Link to dataset: https://huggingface.co/datasets/theresiavr/legalpincite
1 Introduction
Legal IR retrieves legal information and citations that support legal analysis, judgments, and precedent-based decision making. LegalPincite addresses limitations in existing legal IR datasets by enabling citation retrieval across case-to-case, paragraph-to-case, and paragraph-to-paragraph levels while mitigating citation leakage.
- Legal citations can support legal analysis, substantiate judgments, and communicate precedents that lend authority to decisions.
- Most legal IR test collections focus on case-to-case or paragraph-to-case retrieval, making them unsuitable for finding pincites or retrieving paragraphs [16] [19] [9, 17, 22, 36].
- Only three paragraph-to-paragraph legal citation datasets were known, all derived from CJEU judgments, and their queries retained citation-related information that leaked the cited cases or paragraphs [25].
- LegalPincite supports case-to-case, paragraph-to-case, and paragraph-to-paragraph retrieval while using heuristics and named entity recognition to remove citation-related information from queries.The dataset is presented as an updated resource intended to advance legal IR research and potentially benefit adjacent communities such as legal natural language processing.
2 Dataset · 2.1 Dataset Construction · Data acquisition.
LegalPincite merges two existing CJEU legal IR datasets, addresses their quality and reusability issues, and incorporates data from the most recent four years. Its source data come from EUR-Lex and combine paragraph-level content, citation pairs, identifiers, and case metadata.
- 2.1 Dataset Construction: LegalPincite merges two existing datasets, resolves their quality and reusability issues, and updates them with data from the last four years.
- Data acquisition.: The unpublished Olsen et al. dataset contains all citing, cited, and non-citing paragraphs from CJEU judgments through 29 July 2024.It provides paragraph-level segmentations and supplies candidate paragraphs for LegalPincite.
- Data acquisition.: The published dataset contributes 110,601 citing–cited paragraph pairs from CJEU judgments through 6 October 2021.
- Data acquisition.: Both source datasets originate from EUR-Lex, which provides official and comprehensive access to EU legal documents.
- Data acquisition.: EUR-Lex documents may be reused commercially or non-commercially and are often available in multiple official EU languages.
- Data acquisition.: Paragraph records generally include CELEX case identifiers, paragraph numbers, and case-level metadata such as titles and dates.Human-annotated paragraphs are the stated exception to the paragraph identifiers.
Issues and resolutions.
The authors identify four dataset issues and resolve them in LegalPincite by standardizing language, restoring missing text, correcting segmentation, and making human annotations reusable.
- Language ambiguity: Language detection with langdetect removes about 5% of non-English paragraphs, enabling language-appropriate preprocessing.The source datasets contain English and French paragraphs without language labels.
- Missing text: The authors recover text for 268 unique citing or cited paragraphs missing from the citation dataset.They retrieve text automatically from the all-paragraph dataset and manually online, addressing missing query and candidate paragraphs.
- Segmentation errors: They correct parsing errors affecting more than 1,400 cases by extracting paragraph numbers and text from EUR-Lex HTML/XHTML through the Cellar API.Consecutive paragraphs that remain joined in online versions are left as is, and missing cases are added to ensure CELEX consistency across datasets.
- Non-reusable human annotations: They manually standardize and relink human validation annotations to original dataset identifiers so the data can be loaded and used.The released query-paragraph-label triples were not directly linkable and had inconsistent formatting, including extra fields and mixed labels and notes.
Data update. · 2.2 Data Preprocessing
The dataset update extracts cited and citing paragraphs from Cellar API sources, excludes unavailable cases, and integrates the corrected and updated data into a cleaned, leakage-mitigated IR collection. Preprocessing also derives case- and paragraph-level citation representations by aggregating paragraph text and citation links.
- Data update.: 155 cases unavailable online were excluded from the dataset during the data update.
- Data update.: Citing and cited paragraph text was extracted from HTML or XHTML sources through the Cellar API.
- Data update.: Extracted paragraph text was matched to citing and cited case-paragraph IDs using the new data and the all-paragraph dataset.
- 2.2 Data Preprocessing: The preprocessing workflow also addressed data leakage and converted the final dataset into a common IR test collection format.
- 2.2 Data Preprocessing: The corrected all-paragraph and citation datasets were merged with updated versions, duplicates were removed, and trailing whitespace was stripped.
- 2.2 Data Preprocessing: Case-level query and candidate text was formed by joining all paragraphs belonging to each case.
- 2.2 Data Preprocessing: Paragraph-to-case citations treated pincites as case citations, while case-to-case citations were aggregated from paragraph-to-paragraph links.
Data aggregation and cleaning.
Relevance annotations are aggregated across two experts using two rule-equivalence questions. Query–paragraph pairs judged non-relevant are removed.
- Annotation aggregation: Relevance labels are aggregated across two experts for each query–paragraph pair.A pair is considered relevant only when both experts answer “Yes” to at least one specified question.
- Relevance criteria: A candidate paragraph is relevant if it contains either a verbatim version or a different, more expanded version of the citing paragraph’s rule.Both experts must affirm at least one of these criteria.
- Cleaning: Pairs judged non-relevant are removed from the aggregated annotations.
Data split. · Query masking for data leakage mitigation. · Formatting.
The dataset uses year-based splits and leakage-mitigated queries to support temporally realistic legal retrieval. It is distributed as PyTerrier-compatible CSV files containing queries, documents, relevance judgments, and case metadata.
- Data split.: Queries are split by case year into pre-2018 training, 2018–2021 development, and 2022–2025 test sets.For development and test, relevant citations and candidate documents must predate every query in the split, preventing temporal leakage and simulating realistic retrieval.
- Data split.: Development and test candidates and ground-truth citations are restricted to documents published before all queries in their respective splits.This prevents later cases from serving as retrieval candidates or cited answers for earlier queries.
- Query masking for data leakage mitigation.: The masking procedure removes citation-revealing case titles, numbers, paragraph numbers, and parties from queries.A pre-trained legal Named Entity Recognition model [14] and regular expressions identify information for removal; its impact is analysed in §3.3.
- Formatting.: The dataset is formatted as CSV files compatible with the PyTerrier IR framework.The files follow a defined naming convention for queries, documents, and relevance judgments.
- Formatting.: Query files store identifiers and masked or unmasked text, while document files store candidate case or paragraph identifiers and text.These are represented in query_{split}_{level}.csv and doc_{split}_{level}.csv, respectively.
- Formatting.: Relevance files provide binary judgments linking queries to documents, with non-relevant pairs excluded.The qrel files contain qid, docno, label, and source fields; document files are provided only for development and test splits.
- Formatting.: Relevance sources are either EUR-Lex extraction or human expert annotation, and case metadata records CELEX, title, and date.The source field uses eur_lex or human, while metadata.csv contains the three case-level metadata columns.
2.3 Dataset Statistics
The released dataset reports query, ground-truth citation, and corpus statistics across query-document levels and splits, alongside distributions of query length, lexical overlap, and semantic similarity. Across these settings, lexical overlap and semantic similarity with relevant documents are low, making relevance difficult to determine through simple matching alone.
- Query and ground-truth statistics: Table 3 reports query and ground-truth citation statistics for each query-document level and split, including relevant-pair counts and mean and maximum relevant documents per query.The released dataset’s relevance statistics use #rel, Avg/q, and Max/q as defined in the table.
- Corpus statistics: Table 4 reports corpus statistics for candidate cases or paragraphs, including mean words per case or paragraph and mean paragraphs per case.The corpus documents tend to be long, consistent with other legal IR corpora.
- Distributional statistics: Figure 2 presents distributions of query length, lexical overlap, and semantic similarity between queries and relevant documents.Lexical overlap is computed over bags of words, and the dataset uses NLTK sentence- and word-level tokenizers for word counts.
- Similarity and difficulty: Low lexical overlap and semantic similarity (median < 0.5) across query-document levels and dev/test splits make the dataset a challenging legal IR benchmark.Lexical overlap uses Jaccard similarity, while semantic similarity uses BERTScore F1 with distilbert-base-uncased on the first 512 tokens; the finding indicates that simple lexical or semantic matching alone is insufficient [10].
2.4 Adherence to FAIR Principles
LegalPincite follows the FAIR principles through persistent identification, public and accessible distribution, interoperability with standard IR formats and tools, and licensing and documentation that support reuse.
- Findability and accessibility: LegalPincite is assigned a DOI and released publicly on Hugging Face Hub to improve findability, discoverability, and accessibility.The DOI provides a globally unique and persistent identifier.
- Interoperability: The dataset is interoperable through open CSV formatting, TREC qrel compatibility, and integration with common IR frameworks such as pyterrier.
- Reusability: LegalPincite supports reuse through a CC-BY license, documentation, and code examples for loading the data and running baseline retrieval.
3 Experiments
Experiments evaluate four retrieval baselines across all three LegalPincite query-document levels on dev and test splits. Results show retrieval performance varies by level, human-validated queries approximate full-dataset evaluation, and removing paragraph IDs alone does not eliminate leakage.
- Experimental setup: The evaluation uses Hit Rate and NDCG at k∈{3, 5, 10}, plus MAP and MRR over the top-1000 retrieved results, with hyperparameters tuned on dev when applicable.DPH is parameter-free; the best dev configuration by NDCG@10 is used for testing.
- Baseline effectiveness: LMIR performs best for case-to-case, TF-IDF for paragraph-to-case, and BM25—on par with TF-IDF—for paragraph-to-paragraph retrieval across dev and test splits.Effectiveness differs substantially across retrieval levels but generally changes little between dev and test splits.
- Baseline effectiveness: BM25 has the least stable performance across retrieval levels, scoring considerably worse for case-to-case and paragraph-to-case, possibly because it over-penalizes extremely long legal cases.DPH, by contrast, maintains stable performance across splits and levels.
- Human validation: NDCG@10 differences between EUR-Lex-only and EUR-Lex-plus-human query subsets are relatively minor, supporting the smaller human-validated subset as a reasonable evaluation estimate.The study therefore suggests evaluating only the EUR-Lex + human subset when computation is limited.
- Leakage analysis: Citation-related leakage increases NDCG@10 by up to 0.036 (35.6%) for case-to-case and 0.111 (55.0%) for paragraph-to-case, while paragraph-ID removal remains insufficient.The comparison uses original queries and corpus, queries and documents without paragraph IDs, and queries without citation information; BM25 is an exception for case-to-case, dropping by 0.031 (30.7%) in the original setting.
4 Discussion and Conclusion
LegalPincite is a large-scale, English, multi-level legal IR test collection supporting paragraph-to-paragraph, paragraph-to-case, and case-to-case retrieval. Its limitations concern citation-source feedback loops, incomplete expert annotations and masking, and residual personal information, while its pipelines enable broader future applications.
- Contributions: LegalPincite supports paragraph-to-paragraph, paragraph-to-case, and case-to-case retrieval in a large-scale English legal IR test collection.
- Future extensions: The dataset can support citation link prediction, cross-lingual retrieval, legal textual entailment, legal Retrieval-Augmented Generation, and evaluation of legal language-model reasoning.The authors include web-scraping and extraction pipelines to support future extensions.
- Limitations: Citation ground truth may reflect feedback loops from EUR-Lex retrieval, although query–document lexical and semantic overlap is generally low, with median below 0.5.Verbatim overlap may also arise from authors’ experience and domain knowledge rather than retrieval feedback.
- Limitations: Expert annotations cover top-10 results from one dense retriever, limiting exhaustiveness and favoring precision-oriented evaluation at short cut-offs over recall.Legal annotation requires domain expertise and is time-consuming, so it cannot be crowdsourced.
- Limitations: Automatic removal of non-English text and citation masking missed some instances, while a systematic audit remains future work.The authors do not expect these misses to significantly affect performance.
- Limitations: Some names of people and entities may remain in judgments despite attempted removal, although the judgment content is publicly available on EUR-Lex.
GenAI Usage Disclosure
The authors occasionally used GenAI tools for simple coding assistance and minor paper editing, and manually verified all generated output.
- GitHub Copilot and ChatGPT occasionally assisted with simple code writing, such as autocompletion.
- Gemini and Claude were used for minor paper editing, including refining short text segments.
- All GenAI output was manually verified.