Source-linked AI summary
LexNLP: Natural language processing and information extraction for legal and regulatory texts
Michael J Bommarito II, Daniel Martin Katz, Eric M Detterman
TL;DR
Law has received less attention from NLP and machine learning, and the authors identify a lack of tools for converting real, unstructured legal documents into structured data. LexNLP brings together legal-text processing and machine-learning capabilities with associated models and data resources. The package includes contract-specific word2vec models and resources that the authors say can improve research or application quality.
Problem
Law has received substantially less attention than other domains, and the authors identify a lack of tools for turning real, unstructured legal documents into structured data.
Source: §1 · p. 1Method
LexNLP provides legal-text processing and machine-learning capabilities, building on NLTK and scikit-learn and offering associated models and data resources.
Source: §2.1 · p. 2, §2.2 · p. 2, §3.3 · p. 4Results
LexNLP distributes contract-specific CBOW word2vec models for credit, employment, services/consulting, and underwriting agreements, and its resources are described as capable of substantially improving research or application quality.
Source: §3.3 · p. 4, §3.4 · p. 5Takeaways & Limitations
The authors present LexNLP’s optional data resources as capable of substantially improving the quality of research or applications.
Source: §3.4 · p. 5Geoentity extraction requires configuration because loading all translations can create false positives; regulation extraction also requires care to avoid improper transcoding of § symbols.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
LexNLP is an open source Python package focused on natural language processing and machine learning for legal and regulatory text. The package includes functionality to (i) segment documents, (ii) identify key text such as titles and section headings, (iii) extract over eighteen types of structured information like distances and dates, (iv) extract named entities such as companies and geopolitical entities, (v) transform text into features for model training, and (vi) build unsupervised and supervised models such as word embedding or tagging models. LexNLP includes pre-trained models based on thousands of unit tests drawn from real documents available from the SEC EDGAR database as well as various judicial and regulatory proceedings. LexNLP is designed for use in both academic research and industrial applications, and is distributed at https://github.com/LexPredict/lexpredict-lexnlp.
1. Introduction
LexNLP addresses law’s relative lack of NLP and machine-learning tooling by providing tools and data for real legal and regulatory documents. Its goal is to simplify their analysis and use.
1. Introduction
- Open-source NLP and machine-learning libraries supported application development in many domains, while law received substantially less attention despite its ubiquity, societal importance, and specialized form.The authors identify a lack of tools for turning real, unstructured legal documents into structured data as a key impediment.Source: §1 · p. 1
- LexNLP is intended to support work with statutes, regulations, court opinions, briefs, contracts, and other legal work product.The stated goal includes simplifying analysis of these materials and migration of legacy contracts to smart-contract or distributed-ledger systems.Source: §1 · p. 1
1.1. History
- LexNLP and the LexPredict Legal Dataset have been developed since 2015; both, along with ContraxSuite, were open-sourced in 2017.The repositories were updated monthly, and public documentation had been available on ReadTheDocs since release 0.1.6.Source: §1.1 · p. 1
1.2. License and Support
- LexNLP used an Affero GPL license to support inclusion of open-source software and received support through GitHub issue tracking and email.The authors had considered whether and when to convert to MIT or Apache licensing.Source: §1.2 · p. 1
2. Design
LexNLP aims to provide a standardized Python interface by building on established NLP and machine-learning libraries rather than rewriting their core methods. Its design choices favor mature, documented, broadly supported tools, with language coverage and testing practices described as part of the package design.
2. Design
- LexNLP relies on established libraries such as NLTK, scikit-learn, and scipy rather than rewriting core linguistic and statistical methods.Package selection favored open licenses, maturity, documentation, broad platform and language support, and strong ecosystems.Source: §2 · p. 2
2.1. Natural Language Processing
- NLTK was selected for core NLP; an optional Stanford NLP interface, including POS and NER models, is disabled by default and must be enabled at runtime.The authors cited NLTK’s maturity, community, documentation, supplementary models and corpora, and Apache licensing; they noted some alternatives may offer better performance or wider language support for some tasks.Source: §2.1 · p. 2
2.2. Machine Learning
- Scikit-learn was selected for core machine learning, while its limited support for sophisticated deep learning reflects its lack of GPU support.The authors had begun evaluating optional Keras support for GPU computation and deep models ().Source: §2.2 · p. 2
2.3. Language Support
- English support was available, with full German support targeted for 2018 and French, Spanish, and Mandarin planned thereafter.Source: §2.3 · p. 2
2.4. Unit Testing and Code Coverage
- Continuous integration checks unit tests, code coverage, and code style, with test results verified on every commit.Thousands of test records were available on GitHub in CSV format.Source: §2.4 · p. 2
3. LexNLP Package
LexNLP combines legal-text processing, information extraction, text representations and classifiers, and associated data resources. Its components include models and resources developed from legal and regulatory materials, with optional lexicons and datasets described as capable of improving research or application quality.
3.1. Natural Language Processing
- Legal stopwords and collocations are distributed alongside segmentation for documents, pages, paragraphs, sections, and sentences.Document and section segmentation identify titles or headings; paragraph and sentence models were trained on SEC EDGAR contracts, and segmentation models are customizable.Source: §3.1 · pp. 2–3
- LexNLP also provides tokenization, stemming, lemmatization, part-of-speech tagging, and character-sequence, n-gram, and skipgram distributions.These distributions can support customization or development of segmentation and classification models.Source: §3.1 · p. 3
3.2. Information Extraction
- Information extraction covers addresses, non-monetary amounts, legal citations, conditional statements, and constraints, among other structured information types.The address model uses tags and includes common street and building abbreviation disambiguation.Source: §3.2 · p. 3
3.3. Word Embeddings and Text Classifiers
- Distributed SEC EDGAR word2vec models include vector sizes of 50, 100, 200, and 300 with windows of 5, 10, and 20; contract-specific models cover four agreement samples.The contract models are CBOW models with vector size 200 and window 10, trained on credit, employment, services/consulting, and underwriting agreements.Source: §3.3 · p. 4
- LexNLP was used to produce doc2vec models from SEC EDGAR material and Federal and State court opinions, with the models described as scheduled for distribution in release 0.1.10.Source: §3.3 · p. 4
- LexNLP and its doc2vec or word2vec models were used to train classifiers for contract versus noncontract documents, contract types, and clause types.The classifiers were described as scheduled for distribution in release 0.1.11.Source: §3.3 · pp. 4–5
3.4. Lexicons and Other Data
- The Legal Dataset repository includes accounting, financial, geopolitical, legal, and scientific resources, which the authors say can substantially improve research or application quality.Examples include accounting standards, regulator and agency names, common-law terms, chemical elements, compounds, and hazardous waste in multiple languages.Source: §3.4 · p. 5
4. Example Usage
A purchase-and-sale agreement illustrates LexNLP’s API for legal-text segmentation and extraction. The examples show sentence-boundary handling and extraction of contract information, amounts, regulatory references, and other entities.
4.1. Segmentation
- For Example 3, NLTK’s PunktSentenceTokenizer produces 5 sentences and LexNLP’s sentence segmenter produces 3; the passage says NLTK and other NLP packages misparse the “U.S.C” abbreviation.Source: §4.1 · p. 6
4.2. Extraction
- Geopolitical-entity extraction depends on selecting entity and translation data; loading all translations can produce false positives in English text.The German name for Iceland, “Island,” is one example; language filtering may also miss foreign names used without translation.Source: §4.2 · p. 6
- 1,057: LexNLP parses several spelled-out variants of that amount to the same numeric value.The example also shows monetary amounts written out in words.Source: §4.2 · p. 6
- For the text3 example, get_constraints returns 5 constraints, while get_regulations returns the pair (“United States Code”, “42 USC 6901”).Common regulations in the repository are automatically identified, although care is needed to ensure symbols such as § are not transcoded improperly.Source: §4.2 · p. 6
- Extracted citations can serve as document-to-document metadata for graph-based analysis.The paper notes that legal citations may convey semantic information absent from the surrounding text ().Source: §4.2 · pp. 6–7