PaperlayerView exact revision on arXiv

Source-linked AI summary

When Can a Machine Trust a Statute? A Survival Certificate for Machine-Extracted Legal Logic

Surya Saka

arXiv:2609.01741v1cs.AIcs.CL

TL;DR

Machine-extracted statutory attributes disagree, raising the question of which formal implications survive extraction noise. The paper tests canonical implications against measured disagreement; per-chapter calibration improves held-out informativeness, while global calibration and cross-jurisdiction transfer remain fragile.

  • Problem

    Machine-extracted statutory attributes are treated as exact by downstream symbolic logic despite extractor disagreement on features legal logic depends on.

    Source: §1 · p. 1

  • Method

    The paper tests whether canonical implications survive measured disagreement, using per-chapter calibration and non-vacuous survival under perturbed contexts.

    Source: §3.5 · p. 4, §6.3 · p. 10

  • Results

    Per-chapter calibration lifts held-out mean certified fraction from 0.038 to 0.128, while 93.2% of held-out chapters fall below the informativeness floor under global calibration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Calibrate per chapter or family rather than globally, and do not transfer across jurisdictions without recalibration.

    Source: §6.3 · p. 10

    The reference extractor is not validated ground truth, and the certificate does not establish whether extraction or an implication is legally correct.

    Source: §6.2 · p. 10

Abstract

from arXiv · show

Statutes are increasingly parsed by machines before people read them, and the parsers disagree: on Missouri's statutes, two independently written extractors diverge on numeric-threshold presence at a false-negative rate of 0.43. We ask what formal logic survives such noise. We build a passive survival certificate for the Duquenne-Guigues implication basis of machine-extracted statutory contexts: per-attribute inter-extractor disagreement is measured, replayed against the basis in 1,000 Monte Carlo trials, and an implication is certified only when a one-sided Wilson 95% lower bound on survival reaches 0.95; every certified implication carries premise spans and a minimal counterexample. On 29,365 Missouri sections and 502 Indian central-Act sections, the preregistered held-out gate passes (10 statute families across 7 Titles exact; 16 across 11 with 5% tolerance), yet under one globally deployed error model 93.2% of held-out chapters fall below the informativeness floor, and a 2x2 factorial assigns that to calibration-rate transfer, not selection. The certificate is usable but fragile: deploy it per-chapter-calibrated or error-tolerant. Code, data products, and the audit trail, including one retracted claim, are released.

1 Introduction

Machine-extracted statutory attributes can disagree on features legal logic depends on, so support alone cannot establish which implications are robust. The paper builds a survival certificate and reports that its informativeness depends on calibration scope.

1 Introduction

  • Two independently written Missouri extractors disagree about numeric-threshold presence at a false-negative rate of 0.43, on attributes that legal logic depends on.The paper argues that robustness depends on an implication’s attribute footprint interacting with the extractor’s measured error structure.Source: §1 · p. 1
  • The survival certificate replays measured per-attribute disagreement against the Duquenne–Guigues basis and certifies implications when a one-sided Wilson 95% lower bound reaches 0.95.Certified implications carry premise and consequent spans, a minimal counterexample, and a calibration-linked likely failure mode.Source: §1 · pp. 1–2
  • Per-chapter calibration lifts the held-out mean certified fraction from 0.038 to 0.128, while 93.2% of held-out chapters fall below the informativeness floor under global calibration.A preregistered factorial attributes the global-calibration failure to calibration-rate transfer, not held-out selection.Source: §1 · p. 2
  • The preregistered held-out Missouri gate passes for 10 eligible chapters across 7 Titles under exact semantics and 16 across 11 Titles under 5% error-tolerant semantics.The held-out IndiaCode replication has 0 of 6 eligible Acts in the informative band.Source: §1 · p. 2
  • The certificate bounds survival under measured parser disagreement, not whether an extraction or implication is legally correct.The paper describes the instrument as conditional and fragile.Source: §1 · p. 2

2 Related Work

Prior work studies canonical implication bases, approximate rules, stability, learning guarantees, and interactive correction, but the paper focuses on exact statutory implications under measured extraction disagreement.

2 Related Work

  • Formal Concept Analysis yields a canonical, minimal, complete Duquenne–Guigues basis, while prior work assumes a correct context [9, 12, 10].This paper studies the basis when its context is machine-extracted and measurably noisy.Source: §2 · p. 2
  • Approximate-rule approaches relax the object, whereas this paper retains the exact canonical basis and measures which members survive perturbation.Its ε-tolerant semantics is treated as a declared sensitivity parameter.Source: §2 · p. 2
  • Concept stability addresses resistance to object deletion, while PAC implication bases provide distributional learning guarantees [6]; this paper instead gives a per-implication frequentist Wilson lower bound.Set-level multiplicity is handled with Benjamini–Hochberg.Source: §2 · p. 2
  • Interactive correction work relies on a human adjudicator, whereas this paper uses a measured error model and a counterexample carrier that explains how an implication would break [13].The carrier points opposite to the approach in, explaining a certified implication’s failure rather than flagging a corrupt row.Source: §2 · pp. 2–3
  • The paper identifies a gap in robustness analysis for exact symbolic implications induced from machine-extracted legal attributes under measured disagreement.The closest legal FCA application cited concerns Brazil’s LGPD and does not model extraction noise.Source: §2 · p. 3

3 Method

The method constructs a binary statutory context, estimates attribute-specific disagreement between two extractors, and perturbs the canonical implication basis to assess survival and support auditable certificates.

3.1 Binary statutory context

  • A formal context represents sections as objects, machine-verifiable attributes as binary features, and their presence as incidence; the reference context is treated as documentary truth within the experiment.The analysis concerns robustness to extraction disagreement, not extraction correctness.Source: §3.1 · p. 3

3.2 Duquenne–Guigues basis

  • The Duquenne–Guigues basis is computed with NextClosure and cross-checked against an independent pseudo-intent oracle and the concepts library.The evaluation drops zero-support implications and reports the schema tautology xref_res →xref separately.Source: §3.2 · p. 3

3.3 Calibration: measured inter-extractor disagreement

  • A separately authored second parser is run on the same sections, and conditional false-positive and false-negative disagreement rates are measured for each attribute.The pooled false-negative rate is 0.429 for numthresh and 0 for prohib, xref, except, and deadline.Source: §3.3 · pp. 3–4
  • The measured quantity is disagreement between two hand-built parsers, not externally validated extraction error; shared upstream choices may cause it to understate real error.The certificate is conditional on the measured disagreement model.Source: §3.3 · p. 4

3.4 Perturbation model

  • The primary perturbation model independently flips each attribute cell using its measured conditional addition and removal rates.A joint-error diagnostic resamples observed per-section error vectors to test whether independence understates damage.Source: §3.4 · p. 4

3.5 Survival and certification

  • In perturbed contexts, an implication survives non-vacuously when some row still supports its premise and the implication holds on all supported rows, or on at least 1−ε of them.Non-survival is separated into premise collapse and an instantiated violation, whose killing row can be checked.Source: §3.5 · p. 4
  • With N=1000 trials, certification requires a one-sided Wilson 95% survival lower bound of at least 0.95.The threshold depends on the Monte Carlo budget; reported comparisons fix N=1000, and set-level certified fractions use a Benjamini–Hochberg companion at q=0.05.Source: §3.5 · p. 4

3.6 Counterexample and provenance carrier

  • Each evaluated implication carries premise and consequent spans, a minimal counterexample, and a calibration-linked likely counterexample to support manual auditing.The payload also includes observed counterexamples from the independent extractor’s context and colocated exception markers as proximity signals.Source: §3.6 · p. 4

4 Experimental Setup

The experiments use Missouri statutes as the primary corpus and IndiaCode as a held-out jurisdictional replication, with preregistered calibration, evaluation, and informativeness criteria.

4 Experimental Setup

  • The primary Missouri corpus contains 29,365 section entries across 458 chapters, while the IndiaCode replication covers 15 central Acts and 502 parsed sections.Missouri entries include 29,350 unique section ids and 15 disclosed duplicate files in chapters 1–3.Source: §4 · pp. 4–5
  • The controlling evaluation excludes the eight development-selected design families, evaluates 148 held-out Missouri chapters (146 eligible), and transfers an error model measured on a disjoint calibration half.IndiaCode is the held-out jurisdiction.Source: §4 · p. 5
  • Scale analyses cover all 304 chapters with at least 30 sections and all 458 chapters in the full-corpus run.The full-corpus run includes a stability curve over a noise-scaling grid.Source: §4 · p. 5
  • Confirmatory runs use N=1000, pinned seeds, and a Wilson lower-bound threshold τ=0.95; the informative certified-fraction band is [0.10, 0.90].The gate requires at least two held-out families in two Titles, with neval ≥10 and a basis below 10,000 implications.Source: §4 · p. 5
  • Per-context metrics include certified fraction, mean survival, premise-collapse share, and mean whole-basis edit distance; per-implication reporting includes survival and provenance.Source: §4 · p. 5

5 Results

The results show that support does not predict survival, the preregistered held-out Missouri gate passes despite weak global deployment, and local calibration improves informativeness. The cross-jurisdiction replication fails, while negative controls indicate that attribute-specific error matters.

5.1 Support does not predict survival

  • {deadline} →{numthresh} has support 792 but survival 0.000, while a support-12 implication to {prohib} survives at 0.965.Support or confidence alone cannot distinguish the rules; their outcomes are tied to the attributes and sections responsible.Source: §5.1 · pp. 5–6

5.2 Does the certificate survive on held-out Missouri?

  • 10 eligible chapters across 7 Titles pass the exact held-out gate, and 16 across 11 Titles pass under 5% tolerance.The controlling gate requires at least two families across two Titles; the held-out table excludes design families and transfers rates measured on a disjoint calibration half.Source: §5.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6

5.3 Why does global deployment fail?

  • +0.090 held-out and +0.070 design mean certified-fraction gains accompany local rather than transferred calibration.At fixed calibration, moving from design to held-out families changes the mean by −0.031 (transferred) and −0.051 (local); the factorial identifies calibration-rate transfer, not selection, as the driver of weak global deployment. Figure 2 depicts the comparison.Source: §5.3 · p. 6, Table 3 · p. 7, Figure 2 · p. 7

5.4 Does local calibration help?

  • Per-chapter calibration raises the held-out mean certified fraction from 0.038 to 0.128 and the in-band share from 6.9% to 52.7%.The full-corpus exact mean is 0.212 across 458 chapters under in-corpus calibration; that is a different estimand from the transferred held-out estimate.Source: §5.4 · p. 7, Figure 3 · p. 8

5.5 Cross-jurisdiction replication

  • 0 of 6 eligible IndiaCode Acts land in the exact informative band, so the Missouri-passing certificate does not transfer under the reported setup.The study does not isolate the cause of the India result.Source: §5.5 · pp. 7–8

5.6 Controls: calibration, not noise volume

  • Attribute-specific controls distinguish measured-error calibration from generic noise: uniform marginal-matched flips certify 0.0 where calibration yields 0.017–0.067.Marginal shuffling produces larger random-context bases than the real bases, and the independent DG cross-check agrees with NextClosure on every real context.Source: §5.6 · p. 8, Table 4 · p. 8

6 Discussion and Limitations

The certificate is usable because it identifies and documents an auditable core, but its reliability depends on calibration and scope. The discussion recommends local calibration or declared tolerance, while warning against treating implication survival as basis stability.

6.1 Interpretation

  • The provenance layer makes certified implications auditable and fragile ones checkable; non-survival is overwhelmingly instantiated violation rather than premise collapse.Table 5 distinguishes non-vacuous survival from vacuous retention, with their gap representing premise collapse.Source: §6.1 · pp. 8–9, Table 5 · p. 9
  • A single global model leaves 93.2% of chapters below the floor, IndiaCode has 0/6 eligible Acts in band, and implication survival does not guarantee basis stability.At measured noise, mean whole-basis edit distance is 21.0 across 419 chapters and peaks at 59.7 on ch407, against a 57-implication reference basis.Source: §6.1 · p. 9, Figure 4 · p. 9
  • Tolerance helps only with premise support: certified fractions rise from 0.047 to 0.330 at support 20–49 and from 0.147 to 0.618 at support ≥50.Below support 20 there is no lift; at ε=0.20 the pooled certificate reaches 0.967, above the informativeness ceiling, so ε should remain a declared sensitivity parameter.Source: §6.1 · p. 9

6.2 Limitations

  • The certificate bounds survival under measured parser disagreement, not legal correctness; the reference extractor is documentary truth only within this experiment.The authors note that shared upstream text splitting, schema, and vocabulary may understate real error; validation would require construction-gold data or a differently built third extractor.Source: §6.2 · p. 10
  • Method A assumes per-attribute independence, and NextClosure replaces the originally specified LinCbO/In-Close class, with correctness cross-checked against two references.The D⋆ diagnostic preserves polarity-conditioned joint co-errors but is not a second independent extractor.Source: §6.2 · p. 10
  • The favorable core is modest: the gate pass rests on 10 held-out families under global deployment, and preregistration lacks a third-party timestamped registry.The in-sample band analysis is retrospective, so the held-out design carries the verdict; the authors do not generalize beyond Missouri.Source: §6.2 · p. 10

6.3 Deployment protocol

  • Per-chapter or per-family calibration is preferred: the factorial gives held-out means of 0.038 versus 0.128 for global deployment.Run both extractors on the target chapter and estimate false-positive and false-negative rates locally.Source: §6.3 · p. 10
  • If local calibration is impossible, use tolerance with the full ε grid, package each certificate with counterexamples and edit distance, and withhold certificates below the floor.Do not tune ε post hoc or transfer across jurisdictions without recalibration.Source: §6.3 · p. 10

7 Conclusion

The paper concludes that a small, auditable core of extracted statutory logic survives measured disagreement, but the certificate passes in Missouri and fails in IndiaCode. Its deployment guidance is local calibration or a declared tolerance grid, with each certificate carrying audit information.

7 Conclusion

  • A small, auditable core survives measured disagreement, while support alone does not predict survival and attribute footprint crossed with error structure does.The certificate passes its preregistered Missouri gate and fails cleanly on IndiaCode.Source: §7 · pp. 10–11
  • Deploy per chapter rather than globally, or run tolerance with the full grid attached; ship each certified implication with survival probability, edit distance, and a minimal counterexample.The authors identify validated extraction error and a cross-jurisdiction account of India's result as open problems.Source: §7 · p. 11

A Attribute schema

B In-sample headline

The in-sample headline analysis is explicitly descriptive because its contexts were selected during development. Its supplied findings examine tolerance, disagreement, premise support, survival decomposition, and stability under noise scaling.

B In-sample headline

  • The nine development contexts are descriptive only; the certified fraction is the Wilson one-sided 95% lower-bound criterion at survival ≥0.95 with N=1000.Figure 5 displays certified fraction by estimator and semantics against the preregistered informative region.Source: B In-sample headline · p. 12, Table 7 · p. 12, Figure 5 · p. 13

C Full ε grid

  • At ε=0.20, POOLED exceeds the 0.90 informativeness ceiling, showing that sufficiently large tolerance can trivialize the instrument.The full ε grid reports Method A certified fractions across headline contexts.Source: Table 8 · p. 13, Figure 6 · p. 14

D Per-attribute confusion matrix

  • The per-attribute table reports inter-extractor disagreement against the reference, not validated extraction error.Source: Table 9 · p. 14

E Premise-support strata

  • Tolerance and exact certificates coincide below premise support 20, while the lift is large above that support.Table 10 reports mean survival and certified fraction by support stratum for exact and ε=0.05 semantics.Source: Table 10 · p. 14, Figure 7 · p. 15

F Survival decomposition

  • The survival decomposition frames implication deaths as checkable violations rather than support artifacts.The table separates non-vacuous survival from vacuous retention; their difference measures premise collapse.Source: F Survival decomposition · p. 15

G Stability under noise scaling

  • High certified fraction does not imply a stable basis; the stability figures track noise scaling and compare basis-level movement with per-implication survival.Table 11 reports mean certified exact fraction versus noise scaling λ, with λ=1 denoting measured disagreement.Source: Table 11 · p. 15, Figure 9 · p. 16, Figure 8 · p. 16

H IndiaCode per-Act results

IndiaCode's per-Act results flag small-Act variability, while the held-out eligible set has no Acts in the exact informative band. The held-out Missouri table lists the ten exact in-band chapters spanning seven Titles.

H IndiaCode per-Act results

  • IndiaCode per-Act results are small-sample sensitive, and the held-out design applies the neval ≥10 filter.The replication table covers acts identified by year–serial identifier and uses Act-scoped cross-reference resolution.Source: Table 12 · p. 17

I Held-out in-band families

J Ablations and sensitivity

Certified fractions are highly sensitive to load-bearing attributes and the certification threshold, while seed-to-seed variation is small. The support floor leaves POOLED unchanged.

J Ablations and sensitivity

  • Dropping xref or xref_res reduces the POOLED certified fraction to 0.0; dropping numthresh reduces it to 0.037, while dropping oblig leaves 0.077 as the basis shrinks from 30 to 13.The ablations link fragility to measured per-attribute error and the schema’s load-bearing cross-references.Source: J Ablations and sensitivity · p. 17
  • The POOLED exact fraction is 0.133, 0.067, and 0.033 at τ=0.90, 0.95, and 0.99, respectively; varying N from 200 to 1,000 yields 0.033, 0.033, and 0.067.Changing Smin from 1 to 10 leaves POOLED unchanged; at N=500 the controlling gate passes with 4 exact and 9 tolerant families in band.Source: J Ablations and sensitivity · p. 17
  • Across seeds 0, 1, and 2, certified-fraction standard deviation ranges from 0.000 to 0.0157, and same-seed reruns reproduce survival counts exactly.POOLED fractions across the three seeds are 0.067, 0.033, and 0.033.Source: J Ablations and sensitivity · p. 18

K Audit trail and disowned estimators

The appendix documents red-team findings that corrected the gate and error analysis, including retraction of a claim based on an invalid joint-error estimator. The review also records reproducibility checks and distinguishes sampling uncertainty from the certification gate.

K Audit trail and disowned estimators

  • The restored preregistered ≥2-family gate passes at N=1,000 with 10 exact families across 7 Titles, and at the reviewer’s N=500 with 4 exact and 9 tolerant families.The earlier revision substituted a stricter majority-in-band gate and reported a “confirmatory FAIL” inconsistent with its own 4 eligible exact families; majority-in-band is now a post-hoc diagnostic that fails under both semantics.Source: K Audit trail and disowned estimators · p. 18
  • The preregistered 2×2 factorial attributes below-floor held-out prevalence mainly to rate transfer, while selection has a small negative effect; the earlier post-selection explanation was replaced.The earlier causal claim was removed.Source: K Audit trail and disowned estimators · p. 18
  • The unconditional donor-vector XOR estimator injected false negatives on zero-pfn attributes, so its “joint errors are more destructive” claim was retracted.With polarity-conditional donor residuals, D⋆ is at least A in 8 of 9 headline contexts, equal at ch375/ch407 and lower only at POOLED; joint co-error structure is therefore not more destructive than independent flips.Source: K Audit trail and disowned estimators · p. 18
  • Method C resamples sections from the fixed observed matrix to quantify sampling uncertainty, but never gates; its headline range is 0.263–0.600.It addresses uncertainty in the observed context rather than certification decisions.Source: K Audit trail and disowned estimators · p. 18
  • A fresh-copy red-team reproduction had 0 SHA-256 mismatches, byte-identical headline outputs, and zero non-timing differences across scientific leaves.The appendix records defects found by an independent red-team review and the effects of the fixes.Source: K Audit trail and disowned estimators · p. 18
Loading 2609.01741v1…