Source-linked AI summary
Large-Scale Multi-Label Text Classification on EU Legislation
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Ion Androutsopoulos
TL;DR
The paper introduces EURLEX57K for legal large-scale multi-label text classification and evaluates neural classifiers, embeddings, document zones, and BERT. BIGRU-based methods, enhanced embeddings, and title-and-recital inputs yield strong results, while BERT performs best except for zero-shot labels.
Problem
EURLEX57K addresses the need for a larger EU legislative dataset than the widely used earlier resource and supports few- and zero-shot learning.
Method
The paper releases EURLEX57K and compares neural classifiers, including BIGRU and label-wise attention variants, while testing domain-specific embeddings, document zones, and fine-tuned BERT.
Results
BIGRU-based models outperform CNN-LWAN, specialized WORD2VEC and ELMO embeddings improve results, and fine-tuned BERT achieves the best results except for zero-shot labels.
Takeaways & Limitations
Using only document titles and recitals achieves almost the same performance as using full documents and enables fine-tuning BERT despite its maximum text length limit.
Takeaways & Limitations
The investigated methods are unsuitable for extreme multi-label classification with hundreds of thousands of labels, and recurrent methods are costly for long documents.
Abstract
from arXiv · showhide
We consider Large-Scale Multi-Label Text Classification (LMTC) in the legal domain. We release a new dataset of 57k legislative documents from EURLEX, annotated with ~4.3k EUROVOC labels, which is suitable for LMTC, few- and zero-shot learning. Experimenting with several neural classifiers, we show that BIGRUs with label-wise attention perform better than other current state of the art methods. Domain-specific WORD2VEC and context-sensitive ELMO embeddings further improve performance. We also find that considering only particular zones of the documents is sufficient. This allows us to bypass BERT's maximum text length limit and fine-tune BERT, obtaining the best results in all but zero-shot learning cases.
1 Introduction
The paper introduces EURLEX57K for legal large-scale multi-label classification and evaluates neural methods, document zones, and BERT fine-tuning. BIGRU-based models, improved embeddings, and selected document zones yield strong results, with BERT performing best except for zero-shot labels.
- EURLEX57K contains 57k English EU legislative documents tagged with approximately 4.3k EUROVOC concepts.
- Its label distribution makes EURLEX57K suitable for few- and zero-shot learning.EUROVOC labels that are rarely used are under-represented or absent in the dataset.
- A BIGRU with self-attention outperforms CNN-LWAN by a wide margin on EURLEX57K.Replacing CNN-LWAN’s CNN encoder with a BIGRU produces still better results.
- Domain-specific WORD2VEC and context-sensitive ELMO embeddings further improve performance.
- Considering only document titles and recitals achieves almost the same performance as using the full document.This selection bypasses BERT’s maximum text-length limit and enables BERT fine-tuning, which obtains the best results except for zero-shot labels.
2 Related Work
Prior LMTC work compared attention-based RNNs and label-wise attention methods across datasets, but label-wise approaches had not been compared with strong generic text-classification baselines. The paper addresses this comparison on legal text.
- Attention-based RNNs produced the best overall results on four of five LMTC datasets studied by You et al.
- CNN-LWAN uses one attention head per label and outperformed weak baselines for medical code prediction.
- ZERO-CNN-LWAN improved few-shot and zero-shot results on MIMIC datasets despite not outperforming CNN-LWAN overall.
- The paper compares label-wise attention methods with strong generic baselines, including attention-based RNNs and Hierarchical Attention Networks.
3 The New Dataset
EURLEX57K comprises annotated EU legislative documents organized into legal zones and split for large-scale, few-shot, and zero-shot evaluation. Its labels are highly imbalanced, with relatively few concepts used frequently.
- EURLEX57K contains 57k EUR-LEX legislative documents averaging 727 words.
- Each document has four zones: header, recitals, main body, and attachments.The main body is usually organized in articles.
- The preprocessing treats the header, recitals, each main-body article, and attachments as separate sections.
- Only 4,271 of approximately 7k EUROVOC concepts appear in EURLEX57K, and 2,049 are assigned to more than 10 documents.
- The dataset is split into 45k training, 6k development, and 6k test documents.
- Labels are divided into 746 frequent, 3,362 few-shot, and 163 zero-shot labels according to training-document counts.
4 Methods
The methods compare recurrent, hierarchical, label-wise-attention, and pretrained Transformer approaches for assigning probabilities to thousands of labels. BERT is fine-tuned with a task-specific output layer but must process truncated documents because of its 512-wordpiece limit.
- Baselines: The logistic-regression baseline represents documents with TF-IDF features for n-grams of orders 1 through 5.This provides a non-neural comparison method.
- BIGRU-ATT: BIGRU-ATT encodes word embeddings with stacked BIGRUs, self-attention, and a sigmoid output layer producing one probability per label.The document embedding is a self-attention-weighted sum of context-aware BIGRU representations.
- HAN: HAN applies word-level BIGRU self-attention separately within sections, then uses a second BIGRU with self-attention to form one document embedding.The resulting document embedding feeds an output layer analogous to BIGRU-ATT.
- CNN-LWAN and BIGRU-LWAN: BIGRU-LWAN replaces CNN-LWAN’s CNN encoder with a BIGRU that produces context-sensitive token representations.Both label-wise-attention models use one independent attention head and specialized document embedding per label.
- Zero-shot models: ZERO-CNN-LWAN compares encoder representations and label-specific document embeddings with label embeddings derived from descriptor word embeddings.ZERO-BIGRU-LWAN is the authors’ variant that replaces the CNN encoder with a BIGRU.
- BERT: BERT uses a pretrained Transformer with a task-specific dense sigmoid layer that produces one probability per label.Because BERT processes at most 512 wordpieces, it is applied to truncated EURLEX57K documents.
5 Experiments
Experiments compare LMTC evaluation measures and neural classifiers on EURLEX57K, including label-wise attention, embeddings, document zones, and truncated BERT. BIGRU-LWAN variants are strongest among full-document systems, while BERT using the first 512 tokens performs best overall except for zero-shot labels.
- Evaluation measures: R-Precision at K replaces K with the number of gold labels when that number is smaller, reducing the distortion of P@K and R@K.For K = 1, RP@K equals P@K; for large K, it approaches R@K.
- Evaluation measures: K = 5 is a reasonable evaluation point because EURLEX57K documents contain 5.07 labels on average.The paper reports similar conclusions at other K values.
- Full documents: BIGRU-LWAN achieves the best full-document results, indicating that replacing CNN-LWAN’s vanilla CNN encoder with a BIGRU addresses its main weakness.The comparison covers all, frequent, and few-shot labels.
- Full documents: Zero-shot CNN-LWAN and BIGRU-LWAN variants outperform other methods on zero-shot labels by exploiting label descriptors and frozen prior-knowledge embeddings.BIGRU-LWAN remains best for few-shot learning, and reported differences are statistically significant at p < 0.01.
- Embeddings: Legal WORD2VEC and ELMO embeddings further improve BIGRU-LWAN performance over GLOVE embeddings.The embeddings are compared within BIGRU-LWAN experiments.
6 Limitations and Future Work
The investigated methods target LMTC problems with labels in the thousands, not extreme settings with hundreds of thousands of labels. Future work addresses computational efficiency, BERT length limits, cross-domain validation, and generalized zero-shot learning.
- Limitations: The methods are unsuitable for Extreme Multi-Label Text Classification with hundreds of thousands of labels.The present work focuses on LMTC settings where labels are in the order of thousands.
- Future work: RNN- and GRU-based methods have high computational cost, especially for long documents.The authors propose investigating dilated CNNs and Transformers as more efficient alternatives.
- Future work: Future work includes hierarchical BERT models to surpass current length limitations.The paper also plans experiments on additional datasets and generalized zero-shot learning.
- Future work: Evaluating additional datasets is planned to test whether the conclusions hold across different domains.Named targets include RCV1, Amazon-13K, Wiki-30K, and MIMIC-III.
A EURLEX57K statistics
EUROVOC labels are unevenly distributed across EURLEX57K documents. Fewer than half of the labels occur in more than 10 documents, making few-shot and zero-shot learning practically important.
- Label distribution: Fewer than 50% of EUROVOC’s 7k labels appear in more than 10 EURLEX57K documents.The distribution is described as aggressively Zipfian.
- Label distribution: The label distribution demonstrates the practical importance of few-shot and zero-shot learning for legal code prediction.The paper notes a similar distribution in medical code prediction.
B Hyper-paramater tuning
Hyperparameters were selected with HYPEROPT, while BERT training used manually specified settings and early stopping. Figure 3 is unrelated to hyperparameter tuning and only depicts EUROVOC concept distribution.
- BERT settings: BERT used dropout 0.1, learning rate 5e-5, and batch size 8 because of GPU memory limitations.These settings followed the cited BERT recommendations except for batch size.
- BERT settings: BERT training used no-patience early stopping after non-convergence in the fourth epoch, averaging eight to nine epochs across five runs.The training schedule was an implementation response to the observed convergence behavior.
C Evaluation Measures
The paper questions P@K and R@K because they penalize predictions differently when documents contain fewer or more than K gold labels. It therefore favors RP@K and nDCG@K, which adjust for each document’s number of gold labels.
- Limitations of P@K and R@K: R@K penalizes documents with more than K gold labels, even when every returned label is correct.For one document with five gold labels, returning one correct label at K = 1 yields R@1 = 0.20.
- Limitations of P@K and R@K: P@K penalizes documents with fewer than K gold labels by counting unreturned positions against precision.For one document with a single gold label, evaluating at K = 5 yields P@1 = 0.20 as reported.
- Limitations of P@K and R@K: Both P@K and R@K can over- or under-estimate performance when the number of gold labels diverges substantially from K.The paper states that these distortions prevent the measures from reliably singling out the best methods.
- Alternative measures: RP@K and nDCG@K adjust evaluation to each document’s number of gold labels without over- or under-estimating performance for documents with few or many labels.The paper presents macro-averaged versions of both measures.
- Evaluation setting: The experiments report results at K = 5 because 57.7% of EURLEX57K documents contain at most five labels.The label-count distribution is shown in Figure 4.
D Experimental Results
Tables 7–9 report recall, R-Precision, and nDCG results on EURLEX57K across all, frequent, few-shot, and zero-shot labels. They compare full-document methods with starred methods using only the first 512 tokens, while defaulting to GLOVE embeddings unless otherwise stated.
- Recall results: Table 7 reports R@1, R@5, and R@10 for all, frequent, few-shot, and zero-shot labels.The table covers EURLEX57K and distinguishes label-frequency groups.
- Comparison conditions: Starred methods use the first 512 document tokens, whereas other methods use full documents; GLOVE is the default embedding unless stated otherwise.This convention applies to Tables 7–9.
- R-Precision results: Table 8 reports RP@1, RP@5, and RP@10 for all, frequent, few-shot, and zero-shot labels.The table uses the same label-frequency groupings as the recall results.
- nDCG results: Table 9 reports nDCG@1, nDCG@5, and nDCG@10 for all, frequent, few-shot, and zero-shot labels.The table evaluates the same EURLEX57K label-frequency categories.