PaperlayerView exact revision on arXiv

Source-linked AI summary

SCM-based Fairness and Faithful Explainability for Legal Document Classification

Yasmina El Kacemi, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag

arXiv:2610.00045v1cs.CL

TL;DR

Fairness and explanation faithfulness are often evaluated separately, leaving their relationship under debiasing unclear. This study tests SCM-based LegalBERT regularization on ECtHR judgments across performance, fairness, and SHAP faithfulness. It finds that fairness does not improve while explanation sufficiency consistently worsens, demonstrating that explanation changes are not a proxy for fairness in this setting.

  • Problem

    Research has rarely tested whether debiasing changes how faithfully post-hoc explanations reflect model reasoning, despite fairness and explanation quality both mattering in legal decision support.

    Source: §1 · p. 1

  • Method

    The study compares baseline LegalBERT with SCM-based fairness regularization on ECtHR judgments, evaluating demographic fairness, SHAP faithfulness, and classification performance.

    Source: §3.4 · p. 5, §3.6 · p. 6

  • Results

    SCM regularization does not improve fairness across tested strengths, consistently increases SHAP sufficiency, and leaves comprehensiveness and classification performance essentially unchanged.

    Source: §6 · p. 9

  • Takeaways & Limitations

    In this setting, explanation faithfulness is not a proxy for fairness, so fairness must be measured directly in legal-NLP auditing.

    Source: §6 · p. 9

    The conclusions are scoped to one corpus and language, a small encoder set, imperfect keyword-based groups, and token-masking faithfulness metrics; the λ fairness analysis uses one seed.

    Source: §6 · p. 9

Abstract

from arXiv · show

Transformer models such as LegalBERT are increasingly used in legal decision support, raising concerns about both fairness and the transparency of model explanations. These properties are usually evaluated separately, leaving open whether a debiasing intervention that changes fairness also changes how faithfully explanations reflect model reasoning. This study investigates that relationship on the ECtHR alleged-violations corpus from LexGLUE. It compares a LegalBERT baseline with a fairness-regularized variant that penalizes stereotypical warmth and competence representations during fine-tuning. The evaluation covers predictive performance, demographic fairness, and SHAP explanation faithfulness across five random seeds. At the performance-optimal regularization strength, the intervention does not reduce demographic disparity. This null result holds across two fairness definitions and a conventional word-pair control on the gender axis. Classification performance is largely unchanged. However, the intervention consistently degrades explanation sufficiency across all five seeds and three thresholds. A shuffled-pair control reproduces this degradation while leaving performance and fairness unchanged, indicating that the effect arises from contrastive representational regularization rather than specifically from the warmth and competence structure. The results demonstrate a dissociation between fairness and explanation faithfulness: changes in explanation behavior do not necessarily indicate changes in fairness, and fairness must therefore be evaluated directly.

1 Introduction

Legal document classifiers raise concerns about fairness and whether explanations faithfully reflect model reasoning. This study tests whether SCM-based regularization changes demographic disparity, SHAP faithfulness, or classification performance on ECtHR judgments.

1 Introduction

  • In legal decision support, unfaithful explanations can conceal biased or arbitrary reasoning while projecting principled transparency.The paper frames fairness and faithful explainability as conditions for legal accountability because ECtHR judgments bear directly on individual rights.Source: §1 · p. 1
  • The study addresses the open question of whether SCM-based debiasing changes the faithfulness of post-hoc explanations alongside fairness on the ECtHR corpus.Prior work on that corpus found fairness and explainability largely orthogonal using rationale plausibility and AOPC sufficiency, but had not examined SCM-based debiasing and post-hoc explanation faithfulness.Source: §1 · p. 1
  • The study evaluates LegalBERT on the ECtHR LexGLUE corpus for demographic disparity, SHAP explanation faithfulness, and classification performance.It treats transparency as faithfulness: whether a post-hoc explanation reflects the model’s actual reasoning rather than merely appearing plausible.Source: §1 · pp. 1–2
  • Across five seeds, SCM-based fairness regularization does not reduce ECtHR demographic disparity, consistently degrades SHAP sufficiency, and has no reliable performance cost.A shuffled-pair control reproduces the pattern, so the effects are not specific to warmth–competence structure.Source: §1 · p. 2

2 Related Work

Prior research often evaluates fairness and explainability separately, while SCM offers a group-agnostic basis for debiasing and faithfulness provides a stricter explanation criterion. The paper’s related work positions its study at the intersection of these lines.

2.1 Fairness and Explainability as Joint Concerns

  • Prior joint studies report that fairness and explainability do not automatically improve together, but none examined SCM-based debiasing with post-hoc explanation faithfulness.The closest ECtHR study assessed rationale plausibility and AOPC sufficiency rather than SHAP faithfulness.Source: §2.1 · p. 2

2.2 The Stereotype Content Model as a Basis for Debiasing

  • SCM organizes stereotypes around warmth and competence, and prior work has used those dimensions for representation- and task-level debiasing.This paper applies an SCM-inspired contrastive margin regularizer during LegalBERT fine-tuning and evaluates downstream fairness and post-hoc explanation faithfulness.Source: §2.2 · pp. 2–3

2.3 Explainability and Faithfulness in NLP

  • Faithfulness asks whether an explanation reflects model reasoning; perturbation-based sufficiency and comprehensiveness assess this by masking features deemed important.Prior work studied debiasing effects on classification and explanations without asking whether debiasing affects explanation faithfulness.Source: §2.3 · p. 3

2.4 Fairness Definitions and Metrics

  • The paper uses DPD, DI, and EOD to capture unequal positive-prediction rates, their ratio, and label-conditioned error disparities, respectively.It assesses fairness by article type and pre-registers a reduction of ΔDPD ≥0.05 as practically meaningful.Source: §2.4 · p. 3

3 Methodology

The methodology uses matched LegalBERT conditions on the ECtHR alleged-violations corpus to test how regularization affects performance, fairness, and explanation faithfulness. It combines contrastive controls, task-level fairness measures, perturbation-based explanation metrics, and a pre-specified evaluation procedure.

3.1 Research Design

  • Four matched conditions—baseline, SCM pairs, shuffled pairs, and demographic word pairs—use identical data and five seeds to isolate regularization effects.The design also pre-registers hypotheses about fairness, faithfulness, and regularization-strength trade-offs.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 Dataset

  • The ECtHR ecthr_a corpus contains 11,000 judgments with binary labels for ten ECHR articles and severe class imbalance.The official split has 9,000 training, 1,000 validation, and 1,000 test judgments; long documents are head-and-tail truncated to 512 tokens.Source: §3.2 · p. 4

3.3 Baseline Model

  • The Legal-BERT baseline uses a linear multi-label classification head and weighted binary cross-entropy with log-smoothed inverse-frequency class weights.Per-article thresholds are tuned on validation data to maximize F1, avoiding near-zero recall on minority articles caused by a fixed 0.5 threshold.Source: §3.3 · p. 4

3.4 Fairness-Regularised Model

  • The intervention adds a contrastive margin loss over warmth–competence antonym pairs to the weighted classification loss during fine-tuning.It uses 38 pairs; each contributes equally, with a 0.5 margin that pushes embeddings apart until cosine similarity is at most −0.5. The constraint does not remove demographic-term associations with SCM traits.Source: §3.4 · p. 5

3.5 Control Conditions

  • The shuffled-pair control randomizes partners while retaining the SCM vocabulary and loss setup; the demographic control substitutes protected-attribute pairs.The gender control uses binary opposites, while the ethnicity control uses deliberately artificial minority–majority contrasts because ethnicity has no natural antonyms.Source: §3.5 · p. 5

3.6 Evaluation Framework

  • Performance is measured by held-out macro F1, fairness by DPD and EOD with DI diagnostic, and explanation faithfulness by SHAP sufficiency and comprehensiveness.Fairness groups are defined by demographic keywords; only seven articles meeting the reliability threshold enter primary fairness conclusions. Faithfulness uses top-token masking at 1%, 5%, and 10%; lower sufficiency and higher comprehensiveness indicate greater faithfulness.Source: §3.6 · pp. 5–6

3.7 Experimental Procedure and Pre-specified Decisions

  • The procedure selects regularization strength using validation macro F1, compares selected models with baseline across outcomes, then conducts sensitivity analysis over λ.Because attribution-based faithfulness evaluation is costly, its λ sweep uses one seed, while fairness sensitivity analysis aggregates five seeds.Source: §3.7 · p. 6

4 Results

Across five seeds, SCM regularisation does not consistently improve fairness or macro F1, but it degrades explanation sufficiency. A shuffled-pair control reproduces these patterns, while the fairness null also persists across regularisation strengths.

Classification performance

  • Macro F1 is 0.670 ± 0.013 for the baseline and 0.656 ± 0.012 for SCM, a mean difference of −0.014 across five seeds.The distributions overlap substantially, and the paper reports no reliable performance improvement or harm.Source: Table 2 · p. 6, §4 · p. 6

Fairness

  • SCM produces no consistent reduction in demographic disparity: neither DPD nor EOD improves consistently, and the demographic word-pair control also yields a gender-axis null.The fairness analysis covers seven reliable articles across five seeds.Source: Table 3 · p. 7, §4 · p. 7
  • For most article–seed combinations, disparity changes are indistinguishable from zero: paired-bootstrap intervals cross zero for a mean of 5.6 of 7 gender articles and 5.8 of 7 ethnicity articles.An independent named-entity ethnicity grouping shows a mean of 5.0 of 7 articles crossing zero.Source: §4 · p. 7

Explanation faithfulness

  • SHAP sufficiency rises by approximately +0.031 at k=1% and +0.053 at k=10%, indicating lower faithfulness; this direction holds across all five seeds.The direction also holds at all three thresholds, with non-overlapping SHAP bands; Integrated Gradients agrees in direction.Source: Table 4 · p. 7, §4 · p. 7

Shuffled-pair control

  • Shuffled-pair regularisation reproduces flat performance, flat fairness, and degraded SHAP sufficiency, indicating the effect is not specific to warmth–competence pairing.At k=5%, SHAP sufficiency rises from 0.1228 ± 0.0064 to 0.1618 ± 0.0070 under the shuffled-pair control.Source: §4 · p. 7

Regularisation-strength analysis

  • Across the five-seed λ sweep, mean |DPD| over seven reliable articles never improves on baseline, and no λ reaches the pre-registered ΔDPD ≥0.05 threshold.The illustrative single-seed faithfulness curve is non-monotonic.Source: §4 · p. 7

5 Discussion

Across five seeds, SCM regularisation leaves demographic disparity and classification performance largely unchanged while consistently degrading SHAP sufficiency. A shuffled-pair control reproduces the explanation effect, indicating that the dissociation is not specific to the warmth–competence structure and that fairness must be measured directly.

5 Discussion

  • Across five seeds, SCM regularisation leaves demographic disparity and classification performance largely unchanged while consistently degrading explanation faithfulness.The performance-neutral result makes the explanation change difficult to attribute simply to a weaker classifier.Source: §5 · p. 7
  • The fairness null holds across two definitions and the better-balanced ethnicity axis, while keyword-based groups and a gender word-pair control also show no improvement.The ethnicity grouping contains 136 versus 864 documents; the comparison with Brandl et al. [5] reaches a null on the same corpus despite a different protected-group definition.Source: §5 · p. 8
  • SHAP sufficiency increases at every tested k across all five seeds, indicating less faithful explanations; comprehensiveness remains effectively unchanged.Integrated Gradients shows a consistent direction, clearest at smaller k, while its gap at k = 10% falls within the method’s noise band.Source: §5 · p. 8
  • A shuffled-pair control reproduces the effects, locating them in contrastive representational regularisation rather than SCM structure; explanation changes therefore cannot proxy fairness.In this setting, auditors must measure fairness directly; explanation behavior moved while demographic disparity did not.Source: §5 · p. 8
  • The fairness null is specific to text-inferred groups, while masking-based faithfulness metrics may partly reflect out-of-distribution inputs.The gender keyword split is severely imbalanced (931 versus 69); SHAP–IG agreement mitigates but does not eliminate the masking concern.Source: §5 · pp. 8–9
  • External validity is narrow, and validation-F1 selection may miss fairness effects; a single-seed λ analysis does not resolve that concern.The study uses one corpus, language, and a small encoder set, and its margin-based loss is not full projection-based debiasing. Future work calls for structured case metadata, non-masking faithfulness measures, and testing an intervention that actually changes fairness.Source: §5 · p. 9

6 Conclusion

The study finds that fairness and explanation faithfulness do not move together under SCM regularisation: disparity does not meaningfully fall, while SHAP sufficiency worsens without a reliable performance cost. The fairness null also holds under shuffled-pair regularisation, and the authors caution that it does not establish that the model is unbiased.

6 Conclusion

  • SCM regularisation does not meaningfully reduce demographic disparity, but consistently increases SHAP sufficiency while leaving comprehensiveness and classification performance essentially unchanged.The sufficiency increase appears at every threshold across all five seeds, with non-overlapping standard-deviation bands; Integrated Gradients shows the same direction.Source: §6 · p. 9
  • No tested regularisation strength improves fairness, and shuffled-pair regularisation reproduces the overall pattern rather than making it specific to warmth and competence.The observed trade-off concerns predictive performance and local explanation sufficiency, not fairness.Source: §6 · p. 9
  • In this setting, explanation behavior cannot serve as a proxy for fairness, so legal-NLP audits must measure fairness directly.The authors caution that the fairness null is not evidence that the model is unbiased.Source: §6 · p. 9
  • The conclusions are limited to one corpus, one language, a small set of transformer encoders, imperfect keyword-based group proxies, and token-masking faithfulness metrics.The authors also note that the fairness null should be retested with a more reliable group definition and an intervention that changes fairness.Source: §6 · p. 9

A Faithfulness and fairness across regularisation strength

Across the tested regularisation strengths, fairness does not improve below baseline. The faithfulness sweep illustrates a single-seed sufficiency curve, while the main-text faithfulness results aggregate five seeds using each seed’s validation-selected regularisation strength.

A Faithfulness and fairness across regularisation strength

  • The fairness sweep aggregates five seeds, whereas the faithfulness curve is single-seed; main-text faithfulness results aggregate five seeds at each seed’s validation-selected λ.Source: A Faithfulness and fairness across… · pp. 10–11
  • In seed 42, sufficiency is lowest at baseline, then rises to its maximum at λ=0.5; λ=0.1 optimizes classification performance, not faithfulness.Lower sufficiency indicates a more faithful explanation; this curve illustrates one seed rather than an across-seed aggregate.Source: Figure 2 · p. 11
  • No tested regularisation strength reduces disparity below baseline on either fairness axis across five seeds.None approaches the pre-registered ΔDPD ≥0.05 improvement threshold.Source: Table 5 · p. 11

B Fairness Metrics with Confidence Intervals

The appendix distinguishes seed-42 per-article estimates from five-seed summaries that underpin the main-text fairness conclusions. Across-seed ΔDPD intervals generally cross zero under both keyword and NER ethnicity groupings.

B Fairness Metrics with Confidence Intervals

  • The per-article tables are seed-42 estimates, while the across-seed summary and Disparate Impact table aggregate five seeds and support the main-text conclusions.The per-article estimates use 95% bootstrap confidence intervals based on resampling 1,000 test documents 10,000 times.Source: B Fairness Metrics with Confidence… · p. 11
  • Table 6 summarizes per-article ΔDPD zero-crossing counts across five seeds for keyword and NER ethnicity groupings.A count near five indicates no detectable change; the NER grouping uses 59 protected documents versus 136 for keyword matching.Source: Table 6 · p. 11
  • Across keyword and NER ethnicity operationalisations, no article excludes zero consistently across seeds.NER resolves ethnicity membership through named-entity recognition and yields a smaller protected group of 59 documents.Source: B Fairness Metrics with Confidence… · p. 11

B.1 Change in DPD under regularisation (paired bootstrap, seed 42)

Across these appendix analyses, the fairness results do not show a reproducible change under SCM regularisation. The paired-bootstrap, Disparate Impact, and Equalized Odds analyses clarify the estimates and their limits, alongside details of the regularisation pairs and group operationalisations.

B.1 Change in DPD under regularisation (paired bootstrap, seed 42)

  • For seed 42, paired ΔDPD intervals mostly include zero; Art.6’s ethnicity interval excludes zero in only two of five seeds, with inconsistent magnitude.The paired bootstrap resamples the same document indices for both models, accounting for correlated errors; an interval containing zero indicates no distinguishable change.Source: §B.1 · pp. 11–12, Table 7 · p. 12

B.2 DPD with confidence intervals (seed 42)

B.3 Disparate Impact

  • Gender Disparate Impact is unstable with 69 unprotected documents, while ethnicity DI changes from 1.54 ± 0.10 to 1.48 ± 0.08, consistent with DPD and EOD nulls.DI is a secondary diagnostic, not a basis for the primary fairness conclusions.Source: §B.3 · p. 12, Table 10 · p. 12

B.4 Equalized Odds with confidence intervals (seed 42)

  • Baseline and SCM EOD intervals overlap substantially for every article on both axes, extending the fairness null to a label-conditioned criterion.EOD is reported as the larger per-group true-positive-rate or false-positive-rate gap.Source: §B.4 · pp. 12–13, Table 11 · p. 13, Table 12 · p. 13

C SCM Antonym Pairs

D Protected-Group Keyword Lists

  • Keyword matching assigns protected status when a document contains a listed whole-word term; NER yields 59 protected ethnicity documents versus 136 under keyword matching.The ethnicity list is purpose-built for ECtHR rather than adopted wholesale from a validated source.Source: D Protected-Group Keyword Lists · pp. 13–14, Table 14 · p. 14

E Instance-Weighted Regularisation

Weighting SCM loss toward stereotype-laden documents did not yield consistent fairness improvement across the seven reliable articles, while performance remained unaffected. The aggregate null persisted despite mixed per-article findings.

E Instance-Weighted Regularisation

  • Mean absolute DPD was essentially unchanged across seven reliable articles with instance weighting: gender +0.006 and ethnicity −0.007.Per-article signs were mixed, and macro F1 remained 0.6758.Source: E Instance-Weighted Regularisation · p. 14
  • Instance weighting scaled each document’s SCM loss by its share of detected antonym pairs, with weights averaging 0.23 (standard deviation 0.13).The most stereotype-laden documents therefore received the strongest regularisation.Source: E Instance-Weighted Regularisation · p. 14
  • Two ethnicity articles had single-seed point-estimate reductions beyond the pre-registered threshold, but lacked bootstrap confidence intervals and did not change the flat aggregate.Art. 5 had ΔDPD = −0.051 and Art. 6 had ΔDPD = −0.057; both were uncorrected point estimates.Source: E Instance-Weighted Regularisation · p. 14

F Cross-Encoder Replication

Across additional encoder families, SCM-related sufficiency degradation appeared in four of six runs, clearest on RoBERTa and weak and mixed on BERT. These replications support, but do not establish, a cross-encoder pattern.

F Cross-Encoder Replication

  • The comparison replicated baseline versus SCM on bert-base-uncased and roberta-base using the identical pipeline and three seeds per encoder.The runs tested the direction of the sufficiency effect across encoder families, not uniform replication.Source: F Cross-Encoder Replication · p. 14
  • At k=5%, RoBERTa mean sufficiency rose from 0.116 ± 0.028 to 0.159 ± 0.060, while BERT’s mean increase was +0.0003.Four of six encoder–seed runs showed degradation; one BERT seed improved, and only three seeds were run per encoder.Source: F Cross-Encoder Replication · p. 14, Figure 3 · p. 14
  • Across-seed variance was large with only three seeds per additional encoder, so these findings are supporting evidence rather than a second primary result.The reported degradation was neither uniform across encoders nor consistent on BERT.Source: F Cross-Encoder Replication · p. 14

G Demographic Word-Pair Debiasing Control

The demographic word-pair control did not produce a practically meaningful DPD reduction on either axis, and classification performance was unaffected. Its gender-axis result indicates the fairness null is not specific to SCM or contrastive regularisation.

G Demographic Word-Pair Debiasing Control

  • The control used the identical contrastive loss, margin (m=0.5), and λ grid as SCM, substituting demographic pairs and training each axis separately against its own baseline.Gender used nineteen genuine binary opposites; ethnicity used sixteen minority terms paired with unmarked majority-referent terms, a deliberately artificial construction.Source: G Demographic Word-Pair Debiasing… · p. 14
  • No seed or axis reached the pre-registered ΔDPD ≥0.05 threshold; mean changes were −0.001 for gender and −0.000 for ethnicity.Mean ΔF1 was −0.002; Table 15 reports mean absolute DPD over seven reliable articles per seed and axis.Source: G Demographic Word-Pair Debiasing… · p. 14, Table 15 · p. 15, Figure 4 · p. 15
  • The gender-axis null, where word-pair debiasing is designed to apply, indicates the fairness null is not an artefact of SCM design or contrastive regularisation in particular.The per-seed changes had inconsistent signs on both axes.Source: G Demographic Word-Pair Debiasing… · pp. 14–15
Loading 2610.00045v1…