Source-linked AI summary
SaulLM-7B: A pioneering Large Language Model for Law
Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, Michael Desa
TL;DR
Legal language presents specialized linguistic challenges, while generic language models have not fully benefited the legal domain. The paper develops legal language models through dedicated legal-corpus pretraining and instructional fine-tuning, reporting strong legal-task performance and releases the models and evaluation code under the MIT License.
Problem
Legal text has specialized syntax and vocabulary, and the legal domain has not experienced the full benefit of large language models.
Method
The paper develops SaulLM-7B and SaulLM-7B-Instruct using extensive pretraining on dedicated legal corpora and instructional fine-tuning on legal data.
Results
SaulLM-7B-Instruct is reported to outperform existing models such as Mistral or Llama on a variety of legal tasks and to excel across associated tasks.
Takeaways & Limitations
The released legal models, evaluation protocol, and code support legal-language research and broader adoption under the MIT License.
Takeaways & Limitations
Original LegalBench prompts can substantially reduce measured accuracy because verbose outputs may be marked incorrect despite being human-classifiable as correct.
Abstract
from arXiv · showhide
In this paper, we introduce SaulLM-7B, a large language model (LLM) tailored for the legal domain. With 7 billion parameters, SaulLM-7B is the first LLM designed explicitly for legal text comprehension and generation. Leveraging the Mistral 7B architecture as its foundation, SaulLM-7B is trained on an English legal corpus of over 30 billion tokens. SaulLM-7B exhibits state-of-the-art proficiency in understanding and processing legal documents. Additionally, we present a novel instructional fine-tuning method that leverages legal datasets to further enhance SaulLM-7B's performance in legal tasks. SaulLM-7B is released under the MIT License.
1 Introduction
SaulLM-7B addresses the limited benefit LLMs have delivered to legal work by introducing a publicly available model specialized for legal text. The paper also contributes an instruction-tuned variant, an evaluation protocol, and open licensing.
- Motivation and model: SaulLM-7B is presented as a publicly available legal LLM designed to handle legal text’s specialized syntax and vocabulary.Its pretraining uses dedicated legal corpora from English-speaking jurisdictions.
- Model contributions: The SaulLM family includes a 7-billion-parameter model tailored to legal text and an instruction-tuned variant intended for legal tasks.The instruction-tuned model is described as outperforming Mistral and Llama on several legal tasks.
- Evaluation: The paper introduces LegalBench-Instruct and adds selected MMLU legal tasks to improve evaluation of legal language-model proficiency.The added MMLU areas include international law, professional law, and jurisprudence.
- Release: SaulLM-7B, SaulLM-7B-Instruct, and the evaluation code are released under the MIT License.The authors frame this licensing approach as supporting adoption and collaborative development.
2 SaulLM-7B: Extending the legal capabilities of Language Models
The model extends a general 7-billion-parameter backbone through continued legal pretraining and instruction tuning. Its construction combines legal, replay, general-instruction, and legal-instruction data, while alignment experiments did not yield meaningful gains.
- Backbone and training: The authors select Mistral 7B as the 7-billion-parameter open-source backbone for its benchmark and task performance.The methodology is described as a two-step process.
- Continued pretraining: Continued pretraining uses a filtered and deduplicated 30-billion-token legal corpus to strengthen legal capabilities.Replay data from general sources is included to reduce catastrophic forgetting.
- Instruction tuning: The construction procedure combines legal datasets with replay and instruction data, then further enriches instruction fine-tuning with legal instructions.The instruction-tuning process is intended to help the model follow requests while emphasizing legal expertise.
- Alignment: Early experiments found no meaningful performance improvement from an additional human-preference alignment step, so the authors did not pursue it.This choice concerns the present paper’s training approach.
3 Data
The data pipeline assembles legal text from multiple jurisdictions and public or existing sources, then cleans, deduplicates, and filters it. Instruction data combines general resources with synthetically generated legal conversations into a 600K-example dataset.
- Legal data composition: The legal corpus spans the U.S., Europe, and Australia to reflect differences among jurisdictions and legal traditions.The collection prioritizes English-language legal text.
- Legal data composition: Existing datasets and publicly scraped web sources are combined, despite substantial overlap among sources.The authors apply aggressive cleaning and deduplication to address that overlap.
- Pretraining mixture: The pretraining data includes roughly 2% general replay data and conversational instruction data alongside legal text.Replay sources include Wikipedia, StackExchange, and GitHub, while instruction sources include Super Natural Instruction and FLAN.
- Data cleaning: PDF-derived artifacts are filtered with normalization, rule-based cleanup, perplexity filtering, and duplicate removal, leaving roughly 30B tokens of high-quality text.The cleaning targets broken formatting, repeated characters, unusual Unicode, non-English text, and duplicates.
- Instruction data: The instruction dataset combines general sources with synthetically generated legal conversations and is curated into 600K instructions.Legal conversations are generated from legal texts augmented with metadata through staged conversational turns.
4 Evaluation of Legal Knowledge
The evaluation combines backbone perplexity on legal documents with LegalBench-Instruct and legal MMLU tasks, while revising LegalBench prompts to reduce instruction-following artifacts.
- Evaluation protocol: Three benchmarks assess legal abilities through backbone perplexity, LegalBench-Instruct, and legal sections of MMLU.The legal MMLU focus includes international law, professional law, and jurisprudence.
- Perplexity evaluation: Perplexity is measured across contracts, judicial decisions, opinion text, and legislation using up-to-date datasets.The datasets are sourced after the language-model training-data cutoff date.
- LegalBench-Instruct: LegalBench-Instruct addresses prompts that can undercount correct answers when verbose outputs fail first-word Yes/No scoring.The revised prompts remove distracting few-shot examples and request specific output tags.
- Metrics: Balanced accuracy is used for LegalBench and legal MMLU to better handle imbalanced classification tasks.Unless otherwise noted, reported scores in this section use balanced accuracy.
5 Experimental Setting
The experiments compare SaulLM models with open-source Mistral and Llama baselines using LegalBench-Instruct, supported by an open-source implementation and specified GPU resources.
- Baselines: The comparison includes instruction- and DPO-finetuned Mistral-7B variants, Zephyr-7B, and Llama2-7B-Chat and Llama2-13B-Chat.The baselines include Mistral-7B-Instruct-v0.1 and v0.2.
- Baselines: Figure 3 evaluates base-model performance on LegalBench-Instruct, including SaulLM-7B’s Mistral-7B initial checkpoint.The comparison concerns base models before instruction fine-tuning.
- Implementation: The codebase uses open-source frameworks with DeepSpeed, Flash Attention, PyTorch, and models released on Hugging Face.The implementation is built on open-source software components.
- Compute: Continuous pretraining uses 256 MI250 GPUs, instruction fine-tuning uses 16, and evaluation uses one MI250 GPU.The paper reports separate hardware allocations for training and evaluation.
6 Results
The results section summarizes LegalBench-Instruct experiments, emphasizing the effect of the SaulLM base model and continued instruction fine-tuning.
- LegalBench-Instruct: Figure 4 compares instruction fine-tuning from SaulLM-7B against starting from Mistral-7B.The figure also separates generic instruction fine-tuning from mixes that include legal instructions.
- LegalBench-Instruct: Generic instruction fine-tuning from SaulLM-7B outperforms its Mistral-Instruct counterpart, while adding legal instructions further boosts results.The comparison is presented as evidence of the base model’s influence and the additional contribution of legal instructions.
I. Legal continued pretraining brings significant improvements
Continued legal pretraining and instruction finetuning improve SaulLM-7B-Instruct across legal benchmarks and document types. Gains are strongest on tasks requiring legal expertise, while conclusion tasks and legislation remain relative weaknesses.
- Legal continued pretraining: 0.59 average score represents a 4-point improvement over Mistral-7B-Instruct-v0.1 after generic instruction finetuning.SaulLM-7B also performs near Llama2-7B-chat without dedicated instruction finetuning.
- Legal instruction finetuning: 0.61 average score on LegalBench-Instruct establishes a new state-of-the-art, an 11% relative improvement over the best open-source instruct model.The result follows finetuning on both general and legal instructions.
- Benchmark evaluation: 5 average absolute points separate SaulLM-7B-Instruct from Mistral-7B-Instruct-v0.1 across all three Legal-MMLU tasks.This provides additional evidence of consistent superiority on legal evaluation tasks.
- Per-task performance: SaulLM-7B-Instruct outperforms generic instruct models on four legal-expertise areas but falls short on conclusion tasks requiring more pure deductive reasoning.The reported areas are issue spotting, rule recall, interpretation, and rhetoric understanding.
- Perplexity analysis: 8.69 median perplexity is 5.5% lower than Mistral-7B and 10.8% lower than Llama2-7B across legal documents.Llama2-7B nevertheless performs better specifically on legislation documents.
7 Conclusion & Future Perspectives
The paper presents SaulLM-7B as an open-source legal decoder model delivering state-of-the-art performance among 7B models. Released under the MIT license, it is intended to support open-source development and community use.
- Conclusion: SaulLM-7B is an open-source decoder model delivering state-of-the-art performance compared to 7B models in the legal domain.
- Future perspectives: The model is released under the MIT license to contribute to the open-source ecosystem and community.