The Massive Legal Embedding Benchmark (MLEB)

• In legal RAG, low-quality embeddings lead to poor search results and responses, and increased hallucinations.Source: §1 · p. 1
• Existing legal retrieval benchmarks are limited in quality, size, and diversity, so their performance can fail to predict real-world effectiveness.Source: §1 · p. 1
• MLEB aims to address these limitations by offering a larger, more diverse, and higher-quality benchmark.Source: §1 · p. 1

2.1 LegalBench-RAG

• LegalBench-RAG evaluates four pre-existing sets centered on contracts and largely sourced from the US.Source: §2.1 · pp. 1–2
• Legal professionals and advice seekers search across more than contracts, so LegalBench-RAG offers limited evidence about models that generalize across legal domains and jurisdictions.Source: §2.1 · p. 2

2.2 MTEB-Legal

• Manual inspection found many irrelevant query-passage pairs in MTEB-Legal’s AILA Casedocs2 and AILA Statutes datasets, which were constructed through an automated pairing method.The method paired case facts with cases and statutes cited by lawyers, rather than involving legal experts to identify relevant materials.Source: §2.2 · p. 2
• Among the remaining English-language MTEB-Legal sets, two concern consumer terms of service, one concerns legislation, and none concern case law; the sets are largely American.Source: §2.2 · p. 2
• The authors argue that cross-jurisdiction comparisons may be inappropriate when legal systems differ, while MTEB’s non-English sets cover only German and Chinese law across three selected tasks.Source: §2.2 · p. 2

3 The Massive Legal Embedding Benchmark (MLEB)

• Its design targets expert-quality provenance and labeling, real-world legal-tech utility, meaningful legal challenge, and breadth across jurisdictions, legal areas, and text types.The stated challenge includes tasks requiring legal reasoning as well as lexical analysis.Source: §3 · pp. 2–3
• MLEB spans ten sets, three task types, six jurisdictions, and five document types, with Table 1 overviewing jurisdictional and domain coverage.The tasks are retrieval, zero-shot classification, and question answering; the jurisdictions are the US, UK, EU, Australia, Ireland, and Singapore.Source: §3 · p. 3, Table 1 · p. 3
• Seven of MLEB’s ten datasets are new, built through subject-matter-expert labeling or adaptation of existing expert-labeled data.Source: §3 · p. 3

3.1 Bar Exam QA

• Bar Exam QA tests retrieval of cases and legal-literature passages relevant to US state bar questions, requiring lawyer-like analysis and extensive US-law knowledge.Source: §3.1 · pp. 3–4

3.2 SCALR

• SCALR pairs questions presented to the US Supreme Court with descriptions of final holdings, with half reserved for validation and no query overlap between splits.The included half forms the MLEB test set.Source: §3.2 · p. 4

3.3 Singaporean Judicial Keywords

• 500 Singaporean catchword–judgment pairs use real-world annotations created by Singaporean law reporters rather than labels constructed after the fact.Judgments have catchwords and preceding metadata-rich cover material removed to focus models on the texts’ core semantics.Source: §3.3 · p. 4
• Catchwords capture essential case concepts that may not be explicitly referenced in judgments, making the set useful for evaluating legal conceptual understanding and knowledge.The dataset was constructed to make models represent judgment text rather than metadata-rich cover sheets.Source: §3.3 · p. 4

3.4 GDPR Holdings Retrieval

• GDPR Holdings Retrieval pairs 500 fact patterns with holdings in European regulatory and court decisions to test retrieval of relevant decisions.The set was built from GDPRHub articles by separating their facts and holdings sections.Source: §3.4 · p. 4

3.5 Australian Tax Guidance Retrieval

• Australian Tax Guidance Retrieval pairs 112 real-life taxpayer questions with 105 relevant Australian Government guidance and policy documents.Questions came from the ATO community forum, where users often could not locate guidance that could answer them; accountants and ATO officials confirmed document relevance.Source: §3.5 · pp. 4–5
• Eight substantive-law questions were sampled from each of 14 non-service ATO forum subtopics, then paired with relevant text from the most pertinent linked source.Questions were selected from answers containing hyperlinks; the selected answer or an ATO employee’s answer was preferred under the stated selection rules.Source: §3.5 · p. 5

3.6 Irish Legislative Summaries

• Irish Legislative Summaries pairs 500 Irish laws with long titles summarizing their subject matter, scope, and purpose.The dataset removes long titles and preceding text from legislation to test retrieval based on the core semantics of acts rather than metadata-rich front matter.Source: §3.6 · p. 5

3.7 UK Legislative Long Titles

• UK Legislative Long Titles pairs 78 UK laws with their long titles and was constructed from publicly available laws in the same way as Irish Legislative Summaries.Source: §3.7 · p. 5

3.8 Contractual Clause Retrieval

• Contractual Clause Retrieval contains 90 pairings: two representative examples for each of 45 clause types, intended to test retrieval, zero-shot classification, and NLI from definitions alone.Clause types were designed to represent common commercial clauses while remaining distinct; examples were sourced from online clause-reference databases and stripped of potentially overlapping information.Source: §3.8 · pp. 5–6

3.9 License TL;DR Retrieval

• License TL;DR Retrieval pairs 65 human-created tl;drLegal summaries with full texts of publicly available licenses to test matching licenses to summaries of their terms.Source: §3.9 · p. 6

3.10 Consumer Contracts QA

• The MLEB Consumer Contracts QA set splits the pre-existing dataset in half for validation, with no queries overlapping between the splits.The underlying set tests retrieval of contractual clauses relevant to consumer-contract questions.Source: §3.10 · p. 6

4.1 Performance

• 86.03 NDCG@10 ranks Kanon 2 Embedder first among 21 models on MLEB, ahead of Voyage 3 Large at 85.71 and Voyage 3.5 at 84.07, as of 21 October 2025.The benchmark reports task and domain averages across evaluation sets and legal domain types.Source: §4.1 · p. 6, Table 2 · p. 7
• Figure 1 compares NDCG@10 averaged across Judicial, Contractual, and Regulatory datasets, with legal-adapted models showing particularly strong performance.The figure breaks out domain-level performance for the top ten embedding models.Source: Figure 1 · p. 8
• Gemini Embedding ranks first on MTEB but seventh on MLEB, while Voyage 3.5 ranks 23rd on MTEB and third on MLEB.The authors report that qualities associated with general multilingual retrieval performance do not necessarily match those associated with legal retrieval.Source: §4.1 · p. 8
• Strong MLEB performance correlates with legal domain adaptation; the leading models Kanon 2 Embedder, Voyage 3 Large, and Voyage 3.5 are legally optimized.Voyage Law 2 ranks eighth, ahead of OpenAI Text Embedding 3 Large, despite being described as older and likely smaller than most other high-scoring models.Source: §4.1 · p. 8

4.2 Speed

• Commercial-model evaluation time includes network latency, using batches of 16 documents and one query except Voyage AI long-sequence datasets, which use batch size one.Figure 2 plots NDCG@10 against total evaluation time and presents the speed–accuracy tradeoff among commercial offerings.Source: §4.2 · p. 8, Figure 2 · p. 9

4.3 Limitations

• Cohere models could not be evaluated because their terms forbid benchmarking, and Voyage AI, Jina, and Google models have potential data-leakage exposure through default training-data sharing.The authors note that customer evaluations, including benchmarks, would likely be included in shared data.Source: §4.3 · p. 8

5 Conclusion

• MLEB contains ten expert-annotated evaluation sets spanning six jurisdictions and five document types, making it a comprehensive benchmark for legal embeddings.The jurisdictions are the US, UK, EU, Australia, Ireland, and Singapore; the document types include cases, legislation, regulatory guidance, contracts, and literature.Source: §5 · p. 9
• Seven newly released datasets broaden representation of jurisdictions and legal domains beyond American law and contract law.Source: §5 · p. 9
• MLEB, its constituent datasets, and evaluation code are publicly released under open-source licenses, and the authors intend to expand the benchmark.The authors encourage reproductions of their results.Source: §5 · p. 9

6 Data Availability

• All MLEB datasets are available on Hugging Face under open-source licenses, and the evaluation code and benchmark implementation are on GitHub.Source: §6 · p. 9
• The listed datasets include Singaporean Judicial Keywords, GDPR Holdings Retrieval, Australian Tax Guidance Retrieval, and Irish Legislative Summaries.Source: §6 · p. 10
• The remaining listed datasets are UK Legislative Long Titles, Contractual Clause Retrieval, and License TL;DR Retrieval.Source: §6 · p. 10

7 Disclosures

• Isaacus created Kanon 2 Embedder and sponsored MLEB’s creation.Source: §7 · p. 10

Remedies and risk allocation

• Other clauses define remedies and risk allocation, including set-off, cumulative rights, break fees, penalties, and limits on force-majeure liability.The statements also include provisions for injunctive relief and waivers of jury trial or moral rights.Source: Appendix A · pp. 13–15

Other clause types

• Additional clause types address authority to sign, confidentiality, return or destruction of materials, workplace surveillance, and probationary termination.The list also includes disclaimers, powers of attorney, and further assurances.Source: Appendix A · pp. 13–14

Termination rights

• Termination clauses cover breach or default, insolvency, force majeure, convenience, and changes in the counterparty’s ownership or control.The statements describe when a party may terminate, including for any reason under a termination-for-convenience clause.Source: Appendix A · pp. 13–14

Performance and payment

• Operational and payment provisions cover conditions precedent, vesting, cure periods, suspension, overdue interest, payment currency, and bonuses.Further examples address compliance with applicable laws and steps needed to give effect to the contract.Source: Appendix A · pp. 13–15

Transfers and participation

• Transfer-related clauses address exclusivity, drag-along and tag-along rights, first refusal, assignment, and substitution of a contracting party.The statements distinguish majority holders’ ability to require a sale from minority holders’ ability to participate on the same terms.Source: Appendix A · pp. 13–15

Interpretation and dispute terms

• The statements also cover choice of venue, precedence of documents, and the order in which contract terms or related documents are interpreted.The appendix pairs each clause type with an NLI-style statement for the evaluation set.Source: Appendix A · pp. 13, 15