PaperlayerView exact revision on arXiv

Source-linked AI summary

Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases

Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, Prodromos Malakasiotis

arXiv:2103.13084v1cs.CL

TL;DR

Legal NLP had studied judgment prediction without investigating how models justify their decisions, while common rationale work focused on word-level rather than paragraph-level selections. The paper introduces paragraph-level rationale extraction for alleged ECHR violations and releases an annotated ECtHR dataset to study rationale constraints. It finds that constraint effects differ: continuity is not beneficial, comprehensiveness needs re-formulation for the multi-label task, and singularity improves rationales even compared with noisy supervision.

  • Problem

    Legal judgment prediction had been studied, but prior work had not investigated justification of model decisions, and the proposed task requires paragraph-level rationales for multi-label predictions.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper builds a paragraph-level, rationale-by-construction approach to predict alleged ECHR-article violations from ECtHR case facts and studies rationale constraints as regularizers.

    Source: §1 · pp. 1–2

  • Results

    Continuity is not beneficial, comprehensiveness needs re-formulation for the multi-label task, and singularity improves rationales even compared with noisy silver supervision.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Paragraph-level rationale extraction remains challenging, motivating further study of constraints better suited to the task and techniques to de-bias models.

    Source: §7 · p. 9

    The authors did not conduct subjective human evaluation of rationale plausibility because of limited resources.

    Source: §6.6 · p. 9

Abstract

from arXiv · show

Interpretability or explainability is an emerging research field in NLP. From a user-centric point of view, the goal is to build models that provide proper justification for their decisions, similar to those of humans, by requiring the models to satisfy additional constraints. To this end, we introduce a new application on legal text where, contrary to mainstream literature targeting word-level rationales, we conceive rationales as selected paragraphs in multi-paragraph structured court cases. We also release a new dataset comprising European Court of Human Rights cases, including annotations for paragraph-level rationales. We use this dataset to study the effect of already proposed rationale constraints, i.e., sparsity, continuity, and comprehensiveness, formulated as regularizers. Our findings indicate that some of these constraints are not beneficial in paragraph-level rationale extraction, while others need re-formulation to better handle the multi-label nature of the task we consider. We also introduce a new constraint, singularity, which further improves the quality of rationales, even compared with noisy rationale supervision. Experimental results indicate that the newly introduced task is very challenging and there is a large scope for further research.

1 Introduction

The introduction frames user-centric rationale extraction as a way for models to justify decisions and applies it to alleged ECHR-article violations in ECtHR cases. It introduces paragraph-level rationales, a new annotated dataset, and a study of rationale constraints.

1 Introduction

  • The approach has models select input subsets that justify their decisions, with constraints encouraging concise rationales rather than post-hoc explanations.Source: §1 · p. 1
  • The paper introduces alleged-violation prediction in ECtHR cases as a more tractable multi-label task requiring paragraph-level rather than word-level rationales.The task predicts applicants’ alleged violations from case facts, unlike legal judgment prediction, which predicts the court’s decision.Source: §1 · pp. 1–2
  • Continuity is not beneficial for paragraph-level extraction, comprehensiveness needs reformulation for the multi-label task, and singularity improves rationales even against noisy silver supervision.The constraints are studied as regularizers.Source: §1 · p. 2
  • The released English dataset contains 11k ECtHR cases, with silver rationales from court-decision references and gold rationales from ECHR-experienced lawyers.Source: §1 · p. 2
  • The authors identify their work as the first rationale-extraction study to fine-tune pretrained Transformer models end to end.Source: §1 · p. 2

2 Related Work

Earlier work studied legal judgment prediction but generally did not justify model decisions; rationale methods mainly selected words for binary classification. This paper instead targets paragraph-level rationales for a preliminary multi-label ECtHR task.

2 Related Work

  • Prior ECtHR prediction work simplified judgment prediction by ignoring legal argument and case law or treating article violations as a binary per-case decision.Earlier work also assumed the Court considers all Convention articles, whereas it rules on alleged violations of particular articles.Source: §2 · p. 3
  • Rationale methods introduced self-supervised word selection and constraints for short, coherent rationales, while later work added comprehensiveness and faithfulness approaches.Lei et al. (2016) introduced rationale extraction by construction; Yu et al. (2019) proposed a comprehensiveness constraint, and Jain et al. (2020) decoupled prediction from rationale extraction.Source: §2 · p. 3
  • Existing rationale work largely used word-level selections for binary classification, despite sentence-level rationales being more suitable for some multi-passage tasks.The paper positions alleged article prediction as a preliminary task for ECtHR applications and a first test-bed for multi-label paragraph-level rationales.Source: §2 · p. 3

3 The New ECtHR Dataset

The paper introduces an ECtHR dataset for predicting alleged Convention-article violations from case facts and identifying supporting paragraphs. It combines case information and rationale annotations, including expert gold annotations for a test subset.

3 The New ECtHR Dataset

  • The dataset comprises 11k ECtHR cases, enriching the ECtHR dataset of Chalkidis et al. (2019) with ground truth for alleged article violations and rationales, and includes court decisions on violations.A legal expert annotated 50 test cases to identify relevant case facts supporting allegations.Source: §3 · p. 4
  • Cases comprise chronologically ordered fact paragraphs, and judges’ references to facts provide silver allegation rationales while a legal expert annotated 50 test cases.The expert annotations identify paragraphs supporting allegations; a case decision’s references to facts are used as silver rationales because judges refer to them when ruling.Source: §3 · p. 4
  • Alleged violation prediction uses case facts to predict which of 40 violable ECHR articles applicants allegedly violated, while also identifying facts supporting predictions.The court’s decisions on whether articles were actually violated are included in the dataset but not used in this study’s experiments.Source: §3 · p. 4

4 Methods

The methods build on a hierarchical BERT model that selects paragraphs as rationales and classify with those selected facts. They adapt rationale constraints for paragraph-level, multi-label prediction and introduce singularity to compare selected rationales against random masks.

4 Methods

  • The methods use the Lei et al. (2016) rationale-by-construction framework as a baseline and add regularizers for paragraph-level and multi-label settings.The authors also evaluate noisy silver-rationale supervision as an alternative method variant.Source: §4 · p. 4, §4.2 · p. 6

4.1 Baseline Model

  • HIERBERT-HA encodes paragraphs with shared BERT and contextual Transformer layers, then produces separate paragraph representations for classification and rationale extraction.Rationale scores are thresholded at 0.5 to create hard masks; selected paragraph representations are pooled and used to estimate probabilities for each Convention article.Source: §4.1 · p. 5

4.2 Rationale Constraints as Regularizers

  • Sparsity regularization controls the number of selected facts, with threshold T specifying the desired percentage of facts selected per case.The rationale is that a small number of facts should sufficiently justify allegations while other facts may be trivial or secondary.Source: §4.2 · p. 5
  • Because ECtHR fact paragraphs are self-contained, the authors hypothesize that forcing contiguous paragraph selections is not beneficial and test continuity empirically.Continuity had been proposed for word-level rationales to encourage coherent selections.Source: §4.2 · pp. 5–6
  • The comprehensiveness constraint compares predictions from selected facts with predictions from their complement, requiring the complement’s classification loss to exceed the selected facts’ loss by a margin.The paper also proposes probability- and representation-based variants of this constraint for multi-label classification.Source: §4.2 · p. 6
  • Singularity requires the selected mask to outperform a randomly generated mask that selects as many facts as the sparsity threshold allows.It addresses comprehensiveness’s limitation: outperforming the complement does not ensure the selected mask is best.Source: §4.2 · p. 6

5 Experimental Setup

The experimental setup evaluates classification, faithfulness, and rationale quality, using LEGAL-BERT-SMALL and averages across five runs. Rationale quality is assessed objectively against annotations, with mean R-Precision accounting for differences in rationale sparsity.

5 Experimental Setup

  • A shared paragraph encoder had comparable performance and better rationale quality than separate paragraph encoders in preliminary experiments.Experiments used LEGAL-BERT-SMALL and reported averages and standard deviations across five runs.Source: §5 · p. 7
  • Evaluation covers classification performance, faithfulness, and rationale quality, using micro-F1, sufficiency and comprehensiveness, and rationale metrics.For classification, micro-F1 is averaged across Convention articles; faithfulness is assessed through sufficiency and comprehensiveness.Source: §5 · p. 7
  • Mean R-Precision compares the model’s highest-confidence selected paragraphs with the number of paragraphs in gold rationales, addressing varying rationale sparsity.The study also reports F1, which the authors consider less fair because methods select different numbers of paragraphs.Source: §5 · p. 7

6 Experimental Results

Across the experiments, singularity produced the strongest rationale-quality results, but all models remained far from human rationales.

6.1 Initial Classification Performance

  • 72.5% or higher F1 was reached for most articles with at least 1,000 training instances; scores were higher for fact-dependent Articles 2, 3, 5, and 6.Source: §6.1 · p. 7

6.4 Task Performance and Faithfulness

  • Constraint-regularized hard-attention models matched HIERBERT-ALL classification performance while improving faithfulness, including 20% deterioration in label probabilities on rationale complements.Source: Table 3 · p. 8, §6.4 · p. 8

6.2 Tuning the λ Hyper-parameters

  • Greedy tuning fixed one regularizer at a time; despite a best development setting, continuity was omitted from later experiments.Source: §6.2 · p. 8

6.3 Comprehensiveness/Singularity Variants

  • Document-representation targeting worked best for comprehensiveness, but label-probability targeting worked best for singularity; representation dissimilarity may over-regularize the latter.Source: Table 4 · p. 8, §6.3 · p. 8, Table 5 · p. 8

6.5 Rationale Quality

  • Singularity achieved the best rationale quality in both silver- and gold-rationale settings, outperforming even silver-rationale supervision.Source: Table 6 · p. 9, §6.5 · p. 9

6.6 Qualitative Analysis

  • Silver rationales often omit expert-annotated facts and include extra facts, yet system rankings measured against silver and gold rationales match on shared cases.Source: Table 6 · p. 9, §6.6 · p. 9
  • Models sometimes treated health or medical references as Article 3 support and higher-court appeals as Article 5 support without concrete evidence; manual inspection found no demographic bias.Source: §6.6 · p. 9
  • Human plausibility comparisons were not conducted for lack of resources, although experts considered partial rationale subsets potentially convincing.Source: §6.6 · p. 9

7 Conclusions and Future work

The paper introduces rationale extraction for alleged violations in European Court of Human Rights cases, releases a dataset, and studies paragraph-level rationale constraints. It identifies future work in constraint design and model debiasing.

7 Conclusions and Future work

  • The paper introduces an alleged-violation classification task and releases an ECtHR dataset with paragraph-level rationale annotations.It compares rationale constraints and introduces singularity, which improves rationale quality, including against noisy rationale supervision.Source: §7 · p. 9
  • Singularity improves rationale quality, including compared with noisy rationale supervision.The paper also reports comparing rationale constraints as regularizers in a paragraph-level setup against silver and gold rationales.Source: §7 · p. 9
  • Future work will investigate better-fitting paragraph-level constraints and techniques to reduce model bias and improve rationale quality.The authors also identify extractive summarization for denoising long documents and case-law retrieval as future directions.Source: §7 · p. 9

A Why is the new dataset any different?

The dataset is presented as the first legal rationale-extraction dataset, with rationales selected at the factual-paragraph level. Its design reflects the paragraph granularity of legal case descriptions.

A Why is the new dataset any different?

  • The dataset is described as the first rationale-extraction dataset for the legal domain, where justifying decisions can require complicated reasoning.The authors present it as a challenging test bed for rationale-extraction research.Source: A Why is the new dataset any different? · p. 12
  • Rationales select entire factual paragraphs, rather than particular words or phrases, matching the granularity of multi-paragraph case descriptions.The case descriptions enumerate facts chronologically, with each paragraph providing information at a granularity considered appropriate for legal reasoning.Source: A Why is the new dataset any different? · p. 12
  • Predicting and justifying alleged article violations is described as practically helpful and as assistance for legal judgment prediction.Source: A Why is the new dataset any different? · p. 12

B Baseline Model

The baseline model uses a shared paragraph encoder for classification and rationale generation. The gold-rationale annotation process selects alleged articles and the factual paragraphs that indicate them.

B Baseline Model

  • The 41.9M-parameter shared HIERBERT paragraph encoder has comparable classification performance and better rationale quality than the two independent-encoder and projection-omission alternatives.Source: Table 7 · p. 12, B Baseline Model · p. 12

C Annotation of Gold Rationales

D Additional Experimental Results

Additional experiments report development results across sparsity and continuity hyperparameters and evaluate rationale quality on the full silver-rationale test set. The reported ranking of methods holds, although full-set quality is slightly lower in most cases.

D Additional Experimental Results

E Using ECtHR dataset via

The ECtHR dataset is available online and can be loaded in Python using the `datasets` library.

E Using ECtHR dataset via

F Examples of extracted Rationales from ECtHR cases with comments

Figures 3–7 compare gold rationales with those extracted by the best model and comment on false positives and false negatives. The examples show varied mismatches between selected paragraphs and the allegations they are meant to support.

F Examples of extracted Rationales from ECtHR cases with comments

  • Figure captions explain false positives and false negatives, while green highlighting marks gold rationales and green dots mark model-extracted rationales.The examples use HIERBERT-HA + Ls + Lg (Eqs. 4 and 6) as the best model.Source: F Examples of extracted Rationales… · p. 13
  • In Knezecic v. Croatia, the model extracted 5 of 15 relevant facts (33%), although the legal expert considered 10 of 15 (67%) relevant.The model was trained at a predefined sparsity level of 30%.Source: Figure 3 · p. 14
  • In K.I. v. Russia, the model missed paragraphs indicating a possible Article 5 violation and selected paragraphs about ill-treatment linked to Article 3.The missed paragraphs concern continuous extensions of the applicant’s detention.Source: Figure 4 · p. 14
  • In Kaimova and Others v. Russia, the model selected a paragraph about medication but missed paragraphs describing ignored health risks.The example suggests sensitivity to general health language rather than well-defined allegations of ill-treatment on the merits.Source: Figure 5 · p. 15
  • In Brajovic and Others v. Montenegro, the model’s selections suggest it may rely on higher-court references rather than connecting trial paragraphs to infer a mistrial.The authors characterize this as possibly poor, greedy reasoning.Source: Figure 6 · p. 15
  • In Rajak v. Montenegro, the model selected paragraph 11, which the authors associate with the property allegation rather than the mistrial argument.The mistrial argument concerns the timing of the verdict, described in paragraphs 5 and 18–19.Source: Figure 7 · p. 16
Loading 2103.13084v1…