Source-linked AI summary
LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics
Gaurab Baral
TL;DR
Legal-meaning metrics can pass identical and unrelated endpoint checks by tracking lexical overlap rather than legal force. The paper introduces LEXFLIP, a dissociation benchmark of minimal Quebec statutory French perturbations, and finds that overlap-based metrics barely react to legal reversals while bidirectional NLI does, although a length feature outperforms semantic metrics on FRJUDGE.
Problem
Existing identical and unrelated-pair checks cannot establish legal meaning preservation because lexical overlap and legal force move together, allowing monotone overlap functions to pass.
Method
LEXFLIP releases 373 minimal perturbations of held-out Quebec statutory French that preserve surface form while changing legal force, evaluated with a common harness.
Results
The seven embedding and BERTScore variants spend 0.022 to 0.039 of their range on legal reversals, versus 0.670 for bidirectional NLI; on FRJUDGE, length features beat every semantic metric.
Takeaways & Limitations
Legal-meaning evaluation should include dissociation tests such as LEXFLIP alongside identical and unrelated-pair checks.
Takeaways & Limitations
The edits are single, template-generated, and definitional, so systems may be tuned to their forms without acquiring general legal sensitivity.
Abstract
from arXiv · showhide
Does a simplified legal clause still say what the original said? The checks in current use cannot establish that it does: requiring an identical pair to score highest and an unrelated pair lowest moves lexical overlap and legal force together, so any monotone function of token overlap satisfies both. Our remedy is a dissociation, an item holding surface form fixed while legal force moves. We release LexFlip, 373 minimal perturbations of Quebec statutory French that reverse legal force while preserving 0.93 of the tokens, with a harness scoring metrics, regressors and prompted judges alike. The seven embedding and BERTScore metrics we test spend only 0.022 to 0.039 of their identical-to-unrelated range on such an edit, against 0.670 for bidirectional NLI, the one family the identical-pair check would disqualify. On FrJudge, against a measured human ceiling of r=0.597, a bare length feature outscores every semantic metric and has the lowest margin we measure.
1 Introduction
Automatic checks for legal-text simplification are inadequate because legal errors can have materially higher stakes than ordinary summary mistakes, while current evaluation relies on endpoint pairings.
- Legal simplification errors can leave readers uninsured when an insurance clause loses a qualifier.
- Current checks require identical pairs to score highest and unrelated pairs lowest, but the supplied test description is incomplete.
QUEBEC HIGHWAY SAFETY CODE
The paper identifies a failure in endpoint checks for legal-meaning metrics: lexical overlap and legal force change together, so overlap-based scores can pass both checks. LEXFLIP separates them by minimally changing legal force while preserving surface form.
- Endpoint checks do not test what a metric reads because lexical overlap and legal meaning move together.Any monotone function of token overlap can satisfy both endpoint conditions by construction.
- Replacing doit with peut changes an obligation into a permission while leaving the sentence fluent and nearly all tokens unchanged.The edit preserves statutory register and approximately ninety-odd percent of the tokens.
- 373 LEXFLIP perturbations reverse legal force while keeping surface form nearly fixed.The release includes a harness for untrained metrics, regressors, and prompted judges, plus a four-part validation protocol.
2 What Would Validate a Legal Meaning Metric?
The paper proposes four validation requirements: measure a human-referenced ceiling, compare systems under identical calibration, control for trivial surface features, and dissociate legal force from overlap.
- R1: a measured ceiling: A measured ceiling compares metric correlations with held-out human ratings and with the reliability of the aggregate label.This distinguishes performance against an individual expert from performance against the easier mean rating.
- R2: identical supervision: Identical supervision fits a monotone map from raw scores to the human scale using training data, preserving rankings while removing scale differences.Isotonic regression cannot improve a ranking, and rank statistics move by at most 0.014.
- R3: a trivial-feature control: A trivial-feature control requires a supervised baseline using surface features alone because shorter simplifications and omission labels can be correlated.The baseline is fitted under the same calibration requirement.
- R4: dissociation: Dissociation measures the fraction of the identical-to-unrelated score range spent on a legally decisive edit with near-identical token overlap.The margin compares the original–edited change with the metric’s own original–unrelated range.
3 LEXFLIP
LEXFLIP constructs minimal legal-force edits from held-out Quebec statutes and evaluates them alongside identical and unrelated sentence pairs. Its inventory is organised by legal effect rather than linguistic category.
- Source sentences come from the Automobile Insurance Act and Highway Safety Code, held out from every corpus used.Each sentence receives one minimal edit that changes legal force while leaving the sentence intact.
- LEXFLIP organises perturbations by legal effect, with tier 1 covering unambiguous changes in who is bound, what is covered, or how much.The passage begins with six tier-1 families, including modality, party, and polarity changes.
- Each perturbation is evaluated as an original pair, an edited pair, and an unrelated same-register pair.This three-way design supports the identical, LEXFLIP, and unrelated comparisons used by the diagnostic.
- Table 1 reports per-metric min–max-normalised scores, discrimination, and the margin spent on the legal edit.Discrimination is the share of items ranked identical above edited; margin is the identical-to-unrelated range used by the edit.
- The harness includes cosine-based metrics, bidirectional NLI, token Jaccard, and surface features.Bidirectional NLI uses entailment failures to represent hallucination and omission, while the other additions provide overlap and trivial-feature controls.
4 Results
The diagnostic separates calibrated agreement with human labels from sensitivity to legally decisive edits. Similarity metrics perform poorly on the latter, while the FRJUDGE analysis shows that simple length features can outperform semantic metrics.
- 4 Results: 0.022–0.039 of working range was spent by seven embedding and BERTScore metrics on edits reversing the law.BERTScore over CamemBERTv2 scored a flipped sentence 0.982 versus 1.000 for the original and 0.193 for an unrelated statute.
- 4 Results: 0.670 of working range was spent by bidirectional NLI on legal-force reversals, with correct ranking on 93.3% of items.It placed flipped sentences nearer unrelated pairs than identical pairs.
- 4 Results: The existing identical-pair check would exclude bidirectional NLI because its identical score was 0.784.mDe-BERTa’s mean self-entailment was 0.780 and cleared the 99% threshold on only 1.3% of items.
- 4 Results: FRJUDGE was reconstructed from a raw export of 297 clauses rated by five law students, enabling the reported human-ceiling analysis.The public release omitted annotator identities, so the authors recovered five pseudonymised annotators from the Prodigy export.
- 4 Results: r = 0.62 for a ridge using word counts and token overlap reached the human ceiling and exceeded every tested semantic metric, whose best reached r = 0.482.A bare length difference reached r = 0.641, while the measured human ceiling was r = 0.597.
5 Discussion and Conclusion
The discussion argues that legal-meaning metrics need a dissociation test because strong agreement with human ratings does not ensure sensitivity to legal changes. It presents LEXFLIP as necessary but bounded evidence, with limitations involving template scope, training familiarity, corpus mismatch, and evaluation uncertainty.
- Results: ρ = 0.24 (95% interval [−0.39, 0.75]) across 13 scorers shows no resolved relationship between calibrated human correlation and LEXFLIP margin.The durable pattern is at the extremes: the length feature performs strongly on correlation but weakly on legal sensitivity, while bidirectional NLI reacts to the edit.
- Conclusion: Three checks—identical, unrelated, and LEXFLIP pairs—are proposed because lexical overlap can satisfy the first two without testing legal sensitivity.The third check asks whether a metric moves when, and only when, the law moves.
- Limitations: LEXFLIP labels are definitional rather than lawyer-annotated, and its single template-generated edits make it a necessary condition rather than a benchmark to optimize.The authors state that a small expert study would provide stronger evidence.
- Limitations: Bidirectional NLI’s 0.670 margin may reflect familiarity with negation, modality, quantifier, and antonym forms rather than legal sensitivity.The next diagnostic version should include a meaning-preserving paraphrase control.
- Limitations: The FRJUDGE comparison and LEXFLIP diagnostic use different texts and answer different questions, while judge results rely on one model and a reconstructed rubric.The reported ceiling uses all 297 items, whereas model correlations use 89-item splits, so comparisons carry sampling uncertainty.
- Deployment: A metric that cannot distinguish doit from peut should support triage for human review rather than certification of safe publication.The perturbed sentences are deliberate misstatements of Quebec law and are not legal advice.