PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

Ilias Chalkidis, Manos Fergadiotis, Ion Androutsopoulos

arXiv:2109.00904v2cs.CL

TL;DR

Legal NLP lacked a broad testbed for zero-shot cross-lingual transfer, especially under temporal concept drift. The paper introduces MULTI-EURLEX and evaluates multilingual models with adaptation strategies, finding that constrained adaptation improves transfer by retaining multilingual knowledge, while effects vary with model and label-set size.

  • Problem

    Cross-lingual transfer had not been studied in legal NLP, limiting evaluation of multilingual legal topic classification across languages.

  • Method

    The paper introduces MULTI-EURLEX, a 65k-document, 23-language multilingual legal dataset, and uses it to evaluate zero-shot transfer with monolingual and multilingual Transformer models and adaptation strategies.

  • Results

    Adaptation strategies substantially improve zero-shot cross-lingual transfer by retaining multilingual knowledge, while their impact depends on the pretrained model and label-set size.

  • Takeaways & Limitations

    Chronological splits are important because temporal concept drift makes random splits over-estimate legal topic-classification performance.

  • Takeaways & Limitations

    The trade-off between the number of trainable parameters and end-task performance remains for future investigation.

Abstract

from arXiv · show

We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another language (target). We find that fine-tuning a multilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set.

1 Introduction

The paper introduces MULTI-EURLEX to address underexplored cross-lingual transfer in legal NLP, using a multilingual legal classification benchmark to study zero-shot transfer and temporal concept drift. Experiments show that adaptation strategies preserve multilingual knowledge and improve transfer, while chronological splits expose performance overestimation from random splits.

  • Cross-lingual transfer has not yet been explored in legal NLP despite growth in legal NLP and multilingual learning.
  • MULTI-EURLEX contains 65k EU laws officially translated in 23 languages, with multiple EUROVOC labels organized hierarchically.
  • The dataset supports zero-shot transfer by training on labeled documents in one source language and classifying documents in another target language.
  • Fine-tuning multilingual models in one source language causes catastrophic forgetting and poor zero-shot transfer to target languages.
  • Partial fine-tuning, adapters, BITFIT, and LNFIT substantially improve zero-shot transfer by helping retain multilingual knowledge, with effects depending on the pretrained model and label-set size.
  • Chronological rather than random splits reveal temporal concept drift in legal topic classification and problems with random splits.

2 Related Work

Related work covers monolingual legal classification, broader multilingual transfer, and early multilingual legal studies. This paper differs by evaluating state-of-the-art multilingual Transformer models across 23 languages on a much larger legal dataset without requiring parallel training data or machine translation.

  • Earlier legal topic classification for EU legislation was primarily monolingual, while cross-lingual transfer had not been studied in the legal domain.
  • Cross-lingual NLP research increasingly uses multilingually pretrained models and adapter modules for transferring models to new languages.
  • Prior multilingual legal studies used 2.7k EU laws in four languages with monolingual SVMs and multilingual ensembles.
  • Another study transferred English sentence labels to German legal documents using similarity-based alignment with machine translations.
  • This work evaluates multilingual Transformer models across 23 languages and 65k EU laws, without requiring parallel training data or machine translation systems.

3 The MULTI-EURLEX Dataset 2

MULTI-EURLEX is a multilingual, multi-label legal topic-classification dataset covering EU laws across 23 languages and multiple EUROVOC granularities. Its chronological splits expose temporal concept drift, making evaluation more realistic than random splitting.

  • Dataset scope: MULTI-EURLEX contains 65k EU laws translated into 23 official EU languages.
  • Labeling: Each document receives one or more EUROVOC concepts, with four provided label sets spanning the first three hierarchy levels and the original annotation.
  • Supported tasks: The dataset supports legal topic classification and one-to-many, many-to-one, and many-to-many cross-lingual transfer scenarios.
  • Data splits: Training, development, and test sets are chronologically split into 55k, 5k, and 5k English documents, with the test documents replicated across 23 languages.
  • Temporal concept drift: Chronological splitting produces temporal concept drift, whereas random splitting yields near-zero or low label-distribution divergence and more stable apparent performance.

4 Methods

The methods adapt pretrained monolingual and multilingual Transformer encoders to multi-label legal classification. Experiments compare full fine-tuning with strategies that freeze most parameters or train small parameter subsets.

  • Pretrained models: Native BERT models and multilingual XLM-ROBERTA provide document representations for multi-label classification through a sigmoid output layer.
  • Pretrained models: MT5 is evaluated with an encoder-only setup because text generation is unnecessarily complex for hundreds or thousands of labels.
  • Transfer setting: Zero-shot transfer fine-tunes a multilingual model on labeled source-language documents and evaluates it without further training in other languages.
  • Adaptation strategies: Frozen-layer adaptation leaves the first N or all Transformer blocks and input embeddings unchanged during fine-tuning.
  • Adaptation strategies: Adapters insert bottleneck down- and up-projection modules inside Transformer blocks while leaving the remaining block parameters unchanged apart from layer normalization.
  • Adaptation strategies: BITFIT updates only bias terms, whereas LNFIT updates only layer-normalization parameters; the classification layer is always trained.

5 Experimental Setup

The experiments use reproducible Transformer training and chronological evaluation on level 3 EUROVOC labels. Performance is measured with mean R-Precision across three runs, with results reported for the principal multilingual benchmark.

  • Training: All methods use TensorFlow 2, pretrained Hugging Face models, BASE-sized 12-block encoders, and Adam optimization.
  • Training: Learning rates are grid-searched separately for each method using development-set classification performance.
  • Evaluation: Evaluation uses chronological splits, three-run averages, and mean R-Precision for large, skewed multi-label outputs.
  • Evaluation: The main benchmark uses EUROVOC level 3 with 567 labels, while Table 5 reports results for 16 widely spoken languages and the 23-language average.

6 Experiments and Discussion

Experiments evaluate multilingual models in one-to-one, one-to-many, and many-to-many settings. They show that adaptation generally improves XLM-ROBERTA cross-lingual transfer, while outcomes depend on model architecture, language family, and label-set size.

  • One-to-one transfer: XLM-ROBERTA is within 0.7 mRP of native BERTs on average in one-to-one evaluation, which serves as an upper bound requiring target-language training data.
  • One-to-many transfer: All XLM-ROBERTA adaptation strategies improve one-to-many transfer by up to 6.8 All mRP over end-to-end fine-tuning while remaining competitive in English.
  • One-to-many transfer: Adapters achieve the best overall XLM-ROBERTA results, reaching All mRP 56.1 with English source performance of 67.3.
  • MT5 adaptation: For MT5, freezing 3, 6, or 9 encoder blocks raises average mRP by up to 4.7, whereas adapters and LNFIT perform poorly.
  • Limitations: The authors leave the trade-off between trainable-parameter count and source-versus-all-language performance for future investigation.
  • One-to-many transfer: Adapters improve English-to-Danish transfer from 57 to 62 mRP but can slightly reduce monolingual performance, such as German from 68 to 67.
  • Language families: Adapter transfer changes more smoothly across language families: Danish-to-Spanish falls from 58 to 50 without adapters versus 62 to 59 with them.
  • Label-set size: As label granularity increases, performance deteriorates approximately 5–10% per level, while adaptation gains over end-to-end fine-tuning generally increase.

7 Conclusions and Future Work

MULTI-EURLEX provides a multilingual, multi-label legal classification dataset and a testbed for zero-shot cross-lingual transfer. Experiments show that adaptation strategies preserve multilingual knowledge and improve transfer, while temporal concept drift makes chronological splits important.

  • MULTI-EURLEX contains 65k EU legal documents in 23 languages, annotated with multiple EUROVOC concepts and alternative label granularities.
  • Fine-tuning multilingual models in one language causes catastrophic forgetting and poor zero-shot transfer to other languages.
  • Adaptation strategies retain multilingual pretraining knowledge and substantially improve zero-shot cross-lingual transfer, with gains increasing for larger label sets.
  • BITFIT and LNFIT remain competitive while fine-tuning less than 0.05% of parameters.
  • Future work will examine alternative adaptation and distributionally robust optimization techniques for temporal concept drift.

Ethics Statement

MULTI-EURLEX consists of publicly available EU laws without personal or sensitive information except trivial consent-based official names. It covers 23 languages from seven language families and is released under a Creative Commons Attribution 4.0 licence.

  • The dataset contains publicly available EU laws and excludes personal or sensitive information except trivial consent-based official names.
  • The collected data is licensed under the Creative Commons Attribution 4.0 International licence.
  • MULTI-EURLEX covers 23 languages from seven language families, reflecting EU official-language translations rather than all languages spoken in EU countries.
  • The authors provide a detailed Dataset Card for MULTI-EURLEX in Appendix D.

A Implementation Details

The implementation uses development-set model selection, early stopping, document truncation, label smoothing, and controlled compute settings. Adaptation methods tune very small parameter fractions but may not reduce runtime because they require more epochs.

  • Learning rates are selected by grid search using development performance, with early stopping after five epochs without mRP improvement.
  • Adapter bottleneck size K is selected from {64, 128, 256, 384, 512}, with 256 obtaining the best development results.
  • Documents are truncated to the first 512 tokens across all methods, based on their average length and prior results.
  • Label smoothing with α = 0.2 is used for EUROVOC levels 1–3 but severely harms performance with the full 7.4k-label assignment.
  • Experiments run on one NVIDIA V100 GPU at a time, with average training runtime reported separately.
  • BITFIT and LNFIT tune approximately 1–4× 1e-3% of parameters, yet can take equal or longer because they train for more epochs.

B Decoder Variants of MT5

The MT5 variants differ in how encoder and decoder representations produce classification outputs. Generative decoding performs poorly because it learns unnecessary label ordering, whereas encoder-only and decode-cls variants remain competitive with XLM-ROBERTA.

  • Generative variant: The generative MT5 variant alphabetically orders labels and predicts one label token per decoder timestep, ignoring predicted order during evaluation.
  • Decoder variants: The decode-cls variant uses one decoder timestep with a [cls] query and applies the encoder-only classification layer to its final representation.
  • Encoder-only variants: The first-pool and last-pool encoder-only variants represent documents using different top-level encoder token representations.
  • Results: Generative MT5 obtains mRP 2.5 because it cannot learn the unnecessary label ordering and stops after five epochs.
  • Results: Both encoder-only MT5 variants are comparable with XLM-ROBERTA, with English mRP approximately 73 and All scores comparable or better.
  • Model configuration: The examined models have comparable encoder capacity, while MT5 additionally includes a larger decoder component.

D.1 Dataset Description

MULTI-EURLEX contains 65k EU laws in 23 official languages, annotated with EUROVOC concepts for multilingual legal topic classification. Its chronological splits and multilingual task configurations support realistic temporal generalization and cross-lingual transfer studies.

  • Documents and languages: MULTI-EURLEX comprises 65k EU laws published from 1958–2016 in 23 official EU languages, each annotated with EUROVOC concepts.EUROVOC label descriptors are available in all 23 languages.
  • Documents and languages: The dataset covers seven language families, with Bulgarian using Cyrillic and Greek using Greek script while the other languages use Latin script.Irish is excluded because EU laws are not translated into it for resource-related reasons.
  • Annotation: EU Publications Office annotations use EUROVOC’s hierarchical concepts, with four provided label sets formed from original concepts and ancestors at levels 1–3.Levels 4–8 cannot be used independently because many documents contain level-3 gold concepts.
  • Splits and concept drift: The dataset is chronologically split into 55k training, 5k development, and 5k test documents, with the same test documents available across 23 languages.For some languages, only a subset of the 55k training documents is available.
  • Splits and concept drift: Chronological splitting introduces temporal concept drift in label distributions and phrasing, representing a realistic temporal generalization problem unlike random splits.The chronological setup is described as avoiding performance overestimation associated with random splits.
  • Supported tasks: MULTI-EURLEX supports multilingual legal topic classification at three EUROVOC granularities and one-to-many, many-to-one, and many-to-many transfer scenarios.The dataset is intended as a testbed for cross-lingual multi-label classification.

E More Detailed Results

Detailed results compare multilingual models, adaptation strategies, language settings, and EUROVOC label levels. The reported findings show that adapter-based fine-tuning is strongest overall for cross-lingual and many-to-many transfer, while native BERT models lead in many individual languages.

  • XLM-ROBERTA results: Native BERT models obtain the best results in 12 out of 15 languages in the detailed one-to-many comparison.The comparison is reported for XLM-ROBERTA and adaptation strategies with English as the source language.
  • XLM-ROBERTA results: XLM-ROBERTA trained in a monolingual one-to-one setting produces competitive results in the detailed comparison.This finding is reported alongside native BERT and adapter results.
  • XLM-ROBERTA results: Adapter-module fine-tuning achieves the best overall results in cross-lingual transfer and the many-to-many setting.The comparison covers end-to-end XLM-ROBERTA fine-tuning and alternative adaptation strategies.
  • Cross-lingual settings: Tables 15–16 report one-to-many results for each of the 23 languages as a source language, comparing end-to-end and adapter-based XLM-ROBERTA fine-tuning.The tables evaluate Level 3 with 567 labels.
  • MT5 results: Table 17 reports MT5 first-pool results across all 23 languages for the alternative adaptation strategies.The evaluation concerns cross-lingual classification at Level 3 with 567 labels.
  • Label levels and temporal drift: Table 18 compares XLM-ROBERTA end-to-end and adapter fine-tuning across all EUROVOC levels, while Table 13 compares random and chronological splits at Level 3.Table 13 evaluates English one-to-one classification with XLM-ROBERTA and 567 labels.
Loading 2109.00904v2…