PaperlayerView exact revision on arXiv

Source-linked AI summary

Four Ways to Forge a Bundle My Own Verifier Calls Clean: Refusal-Site Mutation Testing of an Evidence-Bundle Verifier

Erik Hill

arXiv:2608.26183v1cs.SEcs.CR

TL;DR

The paper asks whether an offline verifier’s acceptance logic actually checks the obligations it claims to check. It studies this with refusal-site mutation and liveness testing on one evidence-bundle verifier, finding extensive silent acceptance paths and improved detection when refusals are tested directly. The results are explicitly bounded as self-measured evidence from one codebase and repository registry.

  • Problem

    An evidence-bundle verifier can accept false claims when its intended comparison or binding is never executed, despite honest hashes and a passing verdict.

  • Method

    The paper measures refusal-site liveness by deleting refusal statements and checking whether the full suite, liveness control, or tamper corpus detects the mutation.

  • Results

    Systematic refusal-site testing raised the score to 112/119, while regression testing the four found defects changed coverage from 37/112 to 39/119.

  • Takeaways & Limitations

    Testing whether verifier refusals can fire exposes acceptance logic that ordinary regression tests and tamper fixtures can leave unexamined.

  • Takeaways & Limitations

    The study is n = 1, self-measured, and uses a registry of the author’s own repositories.

Abstract

from arXiv · show

I built a protocol whose premise is that a stranger can re-run my claims offline and get the same answer. An outside engineer audited it and broke it: a bundle whose headline numbers were false verified clean, the cheapest forgery four bytes. I merged his fix, then pointed my own instruments at the fixed verifier and found the same defect four more times, in places his audit did not reach. The cheapest is one capital letter. The unifying defect is not cryptographic or exotic: a check that reports success along a path where it never examined anything. Vacuous pass is a working label, not a discovery; Section 4 names the literatures already occupying it. So I stopped collecting anecdotes and measured. At f59fb62, under the extraction rule of Section 6, the verifier exposes 112 refusal sites; 75 could be deleted with the whole suite and every tamper fixture still green, a score of 0.330. Three of the four hand-found forgeries fall in surviving classes; the fourth is an obligation with no refusal site. Scored alone, the sixteen-fixture corpus built to prove the verifier can refuse catches 10. Testing the refusals themselves took it to 0.941, then to 1.000 at 92e4548 over a grown population of 146 sites; those denominators differ and the series between them is non-monotone, so Section 7.5 carries all eleven, not just the five rows of Table 2. Fixing the four found defects instead moved 37/112 to 39/119, leaving the pre-existing sites at 37. Seven times during this study my own measuring tools reported success while measuring nothing; four were built to detect this class, and one returned a perfect 1.000. Every number here is self-measured on a system I wrote, over a registry that is a closed loop of my own repositories; the one external data point is the audit of Section 2.2. That is stated here rather than buried: it is the paper's credibility, not a caveat.

1 Introduction

This paper examines whether an evidence-bundle verifier’s refusal logic can actually reject invalid inputs. It introduces refusal-site liveness testing and measures how regression tests, tamper fixtures, and systematic refusal testing expose otherwise silent acceptance paths.

  • The paper presents five fail-open and vacuous-pass classes, each with a working forgery, including the external audit’s null-artifact case.Four classes were found against a verifier already hardened once; the fifth was the auditor’s pre-hardening forgery.
  • Regression testing the discovered defects moved coverage from 37/112 to 39/119, while systematic refusal-site liveness testing moved it to 112/119.The two interventions were not effort-matched, and the second was measured on top of the first.
  • The survivor classification anticipated three of four hand-found forgeries, while the paper’s broader hypothesis about report artifacts was refuted at n = 4.The classification result and the artifact-binding hypothesis are reported as separate contributions.
  • The committed sixteen-fixture tamper corpus caught 10 of 112 refusal sites on its own, against a structural ceiling of 16/112.The ceiling follows from the corpus containing sixteen fixtures designed to demonstrate refusal.
  • Seven measuring instruments reported success while measuring nothing, including one that returned a perfect score.Four of these instruments were built to detect the studied class.

2 The system under study

VAC is an offline evidence-bundle verifier that hashes artifacts, binds headline numbers to recomputable quantities, and publishes accepted results. An external audit exposed additional acceptance defects and shaped the hardened baseline used for the study.

  • VAC bundles pin artifacts by sha256, declare headline numbers, and bind those numbers to recomputable artifact quantities.The verifier reports schema, presence, hashing, closure, limitations, stamp agreement, and recomputation checks.
  • Accepted bundles are copied into a public registry and rendered on a published page, making verifier-surviving lies publicly visible.The verifier therefore mediates the trust boundary between the issuer and the reader.
  • The external audit produced four pull requests; the paper analyzes the first two and records two later findings after measurement.The later findings were not folded into the measured baseline.
  • A bundle declaring 9999 instead of an artifact value of 3 passed after evidence/bundle.json was replaced with the four bytes null and re-pinned honestly.The hash binding remained intact, but recomputation ran over nothing and found no mismatch.
  • The audit’s first pull request reported eight bundles passing structural verification, including false declared numbers and distinct content-boundary defects.The defects included path handling, closure scanning, and archive extraction issues.
  • Four of the auditor’s findings were refuted by the project specification, while a byte-identity check caught mojibake in the auditor’s output.These episodes are presented as evidence that corrections and review artifacts require their own supporting evidence.
  • A later open finding allowed a symlinked artifact outside the bundle to pass, changing the measured refusal-site population and remaining unlanded.At 92e4548, the present-day figures used a population of 146 raw refusal sites.

3 The class: a check that cannot fail

The paper defines its central class as acceptance without execution of an intended check, and connects it to existing weakness taxonomies and related examples. Its common mechanism is an absence assertion without proof that the detector can fire.

  • A vacuous pass occurs when a verifier accepts while an intended binding, comparison, or coverage obligation is absent, bypassed, or semantically unexamined.The paper distinguishes this verifier-specific operational meaning from formal vacuity in model checking.
  • The shared ancestor is an absence assertion with no liveness proof: “nothing bad was found” matters only after establishing that detection can occur.This framing motivates testing whether refusal paths are live.
  • The weakness taxonomy maps the class across CWE-703, CWE-754, CWE-636, and CWE-390, depending on the unchecked or fail-open mechanism.CWE-390 is identified as the tightest fit for the instrument failures in Section 9.
  • Related examples include a determinism test missing a 69-diff divergence, a dead health path returning 200, and a shell gate matching tally text regardless of test outcome.The paper presents these as the author’s motivating observations rather than as study data.
  • An external cosign incident shared the absent-subject shape, although its type-filter mechanism differed from the paper’s operand-existence guard.The paper explicitly distinguishes the shared shape from the different mechanism.

4 Related work

Prior work supplies the conceptual framing, mutation operators, oracle-quality criteria, and adjacent applications, while narrowing this paper’s novelty to a specific combination: mutating a verifier’s own refusal logic and measuring its liveness.

  • Oracles and vacuity: Oracle research frames correctness against an unknowable conceptual oracle and warns that implicit checks can masquerade as specified oracles.Checked coverage likewise showed that high statement coverage can coexist with parsed results that are not actually checked.
  • Mutation testing and deletion: Mutation testing is established as an adequacy criterion rather than a correctness argument, with statement deletion and error-handling deletion already published operators.The paper positions its deletion operator within prior mutation-testing research rather than presenting the operator itself as new.
  • Adjacent checker mutation: Prior checker-focused work mutates the artifacts being judged, whereas this paper mutates the judge’s own acceptance logic.MASC and related work therefore occupy the broader idea of grading a checker while leaving the verifier source outside the mutation target.
  • Attestation context: Supply-chain attestation work establishes machine-readable evidence and verifier soundness as a broader setting, including obligations concerning closure and completeness.The paper connects its verifier problem to TUF, Sigstore, reproducible builds, in-toto, and SLSA without claiming those frameworks generally share the defect.

5 Four forgeries against the hardened verifier

Four forgeries against the hardened verifier exploit checks that pass without performing the intended comparison. They arise from permissive type handling, deleted checks, unknown labels, and absent artifact-side operands, despite corruption being rejected in nearby paths.

  • Summary values: Retyping every summary number as a JSON string makes structural verification pass without comparing any artifact or re-pinning any hash.The verifier’s specification explicitly allowed non-numeric descriptive values to pass through, making this a specification hole.
  • Deleted checks: Deleting a recomputation check moves its number into a loose branch that accepts any value found in the recomputed pool.The manifest validator requires a non-empty check list but does not require every listed evidence artifact to be covered by a check.
  • Severity labels: Re-casing failed severity labels makes their weights zero, producing vulnerability_score = 0.0 through ordinary arithmetic without triggering a refusal path.The specification mapped unknown severities to zero, and the tampered example reports a passing structural-verification result after honest hash repinning.
  • Deleted stamps: Deleting artifact-side stamp keys lets an issuer honestly re-pin the bundle and pass all four comparisons without any comparison executing.The same fields are rejected when corrupted rather than removed, while another profile fails closed on a deleted key.
  • General pattern: A comparison guarded on both operands existing is not a check, because the party supplying one operand controls whether the comparison happens.The stamp forgery demonstrates this general fail-open pattern directly.

6 Method: refusal-site deletion

This method measures how much of the verifier’s refusal logic is exercised by deleting refusal statements and checking deterministic failure detectors. It defines the measured population, kill predicate, exclusions, and validity conditions while exposing a denominator vulnerability.

  • Population and operator: Each mutant deletes one refusal site by replacing it with pass, restricting statement deletion to the verifier’s refusal statements.The operator generates one mutant per selected site.
  • Population and operator: 112 refusal sites were enumerated at f59fb62 by matching statements that append named failure reasons to verify_bundle’s failure list.The sweep uses a regular expression covering both f.append( and failures.append( forms, although the live idioms differ by file.
  • Kill predicate and score: A mutant is caught when the unit suite fails, the valid-bundle liveness control breaks, or a committed tamper fixture is no longer refused.Otherwise, the mutant survives; the score is caught mutants divided by refusal sites minus exclusions.
  • Kill predicate and score: The three detectors are evaluated in order, so fixture attribution is marginal rather than absolute and the tamper corpus is also scored independently.The fixture disjunct is reached only for mutants already missed by the unit suite, making the standalone fixture result different.
  • Interpretation: The resulting score measures detection of a chosen class of artificial changes, not whether the verifier itself is correct.This frames refusal-site mutation as an adequacy measurement rather than a correctness proof.
  • Exclusions and validity: Three unreachable refusal lines are excluded using source fragments with expected match counts, preventing silent denominator changes from renames or refactors.At 92e4548, the exclusions cover two OSError wrappers and three source lines, whose reachability was empirically probed.
  • Exclusions and validity: The sweep requires a green baseline and checks exclusion arity, but it does not pin the refusal-site population, allowing refactoring to shrink the denominator silently.Moving four refusals into one helper changed the scored denominator from 143 to 140 while the suite and exclusion counts remained unchanged; a later site-count pin aborted such a run.

7 Results

At the baseline, most refusal sites could be deleted without detection, while systematic refusal testing exposed substantially more liveness than bug-specific regression testing. The operator anticipated three of four independent forgeries but could not detect an obligation for which no refusal site existed.

  • 7.1 RQ1: baseline refusal-site liveness: 75 of 112 refusal sites survived undetected, yielding a mutation score of 37/112 = 0.330.The unit suite caught 37 mutants, while the tamper sweep caught none of the 75 that unit tests missed.
  • 7.1 RQ1: baseline refusal-site liveness: 10 of 112 mutants were caught by the sixteen-fixture corpus alone, for a score of 0.089.Ten fixtures each flipped exactly one mutant, while six flipped none.
  • 7.1 RQ1: baseline refusal-site liveness: The corpus had a structural ceiling of 16/112 because each single-boolean fixture could flip at most one mutant.Fixtures whose failures trace to multiple refusal sites cannot be flipped by deleting any single site.
  • 7.5 RQ3: what did move it, testing the refusals rather than the bugs: The sweep’s endpoint reached 143/143 after excluding three unreachable sites, but that population is not identical to the baseline 112-site set.The endpoint bounds the reachability concern without proving survivor-by-survivor reachability at baseline.
  • 7.2 RQ4: does the operator anticipate the manual findings?: Three of four hand-found forgeries made surviving refusals unreachable, whereas the fourth involved an absent obligation with no refusal site.The operator therefore anticipated three of four forgeries, but was blind to the fourth by construction.
  • 7.3 RQ2: what bug-specific hardening bought: Bug-specific hardening changed coverage from 37/112 = 0.330 to 39/119 = 0.328, while the pre-existing population remained 37/112.The four demonstrated vulnerabilities were closed, but seven new refusal statements were mostly untested.
  • 7.5 RQ3: what did move it, testing the refusals rather than the bugs: Systematic refusal-site testing ultimately reached 112/119, while the recorded eleven-score series was non-monotone as new gates arrived untested.One post-hardening run reported 119/119 = 1.000, but the paper distinguishes this from a vacuous perfect score and documents later denominator-driven declines.

8 Why the existing suite missed all of this

The existing suite exercised the verifier extensively but did not establish that refusal branches could fire. Hand-written fixtures encoded the author’s threat model, leaving its blind spots untested.

  • 9 Why the existing suite missed all of this: Every fixture encoded a forgery the author had already imagined, so the hand-built corpus could not contain its own blind spot by construction.The proposed correction is to derive fixtures from a published attack taxonomy or generate them adversarially.
  • 9 Why the existing suite missed all of this: The 114 tests executed the verifier extensively but did not assert that refusals could fire.The resulting gap was 75 surviving refusal sites at the baseline.
  • 9 Why the existing suite missed all of this: The verifier was a grader for evalmut bundles, but evalmut had never been applied to the verifier itself.The paper frames this as failing to grade the grader before the external audit exposed the issue.

9 Instrument failures observed during the study

The study recorded seven instrument failures that produced plausible success signals, including failures in tools designed to detect vacuous passes. These incidents motivate loud failure paths, adjacent liveness controls, and rechecking exclusions after changes.

  • 9 Instrument failures observed during the study: The tamper sweep reported all 16 fixtures refused even though every invocation exited 127, converting startup failure into apparent success.A later run exited 2 because of a wrong module path; the live control caught that failure.
  • 9 Instrument failures observed during the study: The probe script reported exit=0 after an edit failed on a wrong filename, so the unmodified bundle appeared to pass.The tool built to find vacuous passes reproduced the same failure mode.
  • 9 Instrument failures observed during the study: A stale local origin made a proof-of-concept finding void until git fetch showed that its premise had evaporated.This was another instance of measuring a repository state different from the intended one.
  • 9 Instrument failures observed during the study: The first post-hardening mutation run scored 1.000 against an already-red baseline because every pytest invocation exited nonzero.The tool treated all 119 mutants as caught, presenting a broken measurement as the best possible result.
  • 9 Instrument failures observed during the study: A no-op str.replace edit regenerated the PDF while leaving stale abstract numbers, because the operation could not fail on no-match.The remediation switched to an editor that errors when no replacement occurs.
  • 9 Instrument failures observed during the study: Seven instrument failures produced plausible output, and four occurred inside tools specifically designed to detect this defect class.The tamper sweep, probe script, and mutation sweep all contained such failures; one returned a perfect score.
  • 9 Instrument failures observed during the study: The study’s surviving rules require proving instruments before findings, adjacent liveness controls, rerunning exclusions after fixes, and preferring loud failures over silent no-ops.The paper identifies silent conversions of “did not happen” into “fine” as the shared hazard.

10 Discussion

The paper proposes auditing eval suites, CI gates, and verifiers for checks that can pass without executing and for detectors proven able to fire. Mutation testing operationalizes this argument: a gate never observed failing has not been observed working.

  • Two questions audit any eval suite, CI gate, or verifier: can the check pass without executing, and has the detector fired in this run on this host?Examples include empty inputs, missing files, swallowed exceptions, and unknown enum values absorbed by defaults.
  • A gate never observed failing has not been observed working, motivating mutation-testing eval suites instead of trusting green results.The argument parallels sanity-checking practices previously applied to model checkers.
  • A grep that matches nothing is evidence only after separately establishing that the pattern can match.At f59fb62, grep -rn severity tests/ returned zero; at 92e4548, it returned 13 after tests were added.

11 Scope of claims

The paper claims a reproducible case study, a refusal-site liveness instrument, empirical comparisons on one verifier, detector results, a hypothesis, and seven instrument failures. It explicitly limits these claims to a self-measured, single-author setting and disclaims broad generalization or novelty claims.

  • 11 Scope of claims: The case study documents five fail-open and vacuous-pass classes, each with a working forgery, including one found by an external audit.Four were found against a previously hardened verifier; the null-artifact forgery came from the audit.
  • 11 Scope of claims: The refusal-site instrument combines deletion, an enumerated denominator, a deterministic two-detector kill predicate, exclusions, and a validity gate.The paper presents site selection and the verifier as its subject, while identifying the operator family and mutation-testing practice as prior art.
  • 11 Scope of claims: The before-and-after study compares discovered-bug regression testing with systematic refusal-site liveness testing on one real verifier.The two arms are not effort-matched, and the second is measured on top of the first.
  • 11 Scope of claims: 10 of 112 refusal sites were caught by the committed tamper corpus alone, against a structural ceiling of 16/112.This result is scored separately from the broader liveness intervention.
  • 11 Scope of claims: The paper records a finding-shaped, untested hypothesis that report-emitted evidence artifacts are systematically underbound relative to check inputs, plus seven instrument failures.One failure produced a perfect score.
  • 11 Scope of claims: The claims are n = 1, self-measured, and based on a closed loop of the author’s own repositories, with one external audit as the sole external data point.The paper does not claim general results about verification systems, transfer, post-audit practice, or methodological novelty.

12 Threats to validity

The score is complete only for a deliberately narrow refusal-append population, not for verifier correctness or all failure modes. Validity is further constrained by omitted rejection forms, artifact-read rather than field-binding coverage, author-selected measurement, provenance issues, reachability, single-system scope, and unmatched interventions.

  • 12 Threats to validity: 1.000 discharges the refusal-append obligation population; it does not establish verifier correctness or bound broader unknowns.Common mutation operators also miss 17% of real faults coupled to no generated mutant.
  • 12 Threats to validity: A verifier scoring 1.000 on this operator can accept a symlinked artifact outside the bundle while reporting structural verification PASS.The bytes hash identically, but their location differs from the bundle.
  • 12 Threats to validity: The source-level denominator misses raises, early returns, propagated exit codes, helper-built reasons, and fail-open behavior before any refusal is reached.At 92e4548, six rejection points were outside the mutated population; parser defaults, decoding, path resolution, duplicate keys, and swallowed exceptions are also omitted.
  • 12 Threats to validity: All 146 mutants parsed, none swallowed neighboring refusal sites, and sampled mutants died on assertion failures, supporting operator soundness.115 of 146 statements were multi-line, handled by the parenthesis-balance span.
  • 12 Threats to validity: Artifact-read closure does not prove field binding: at 92e4548, four of 11 actually corrupted referenced artifacts still verified successfully.Two of 13 references carried no corruptible value and were skipped.
  • 12 Threats to validity: The author selected the operator and measurement, and the proposed preregistration control for selection-after-results bias was not yet done.The paper also reports unreliable concurrent-worker provenance, seven instrument failures, and incomplete per-survivor reachability checks.
  • 12 Threats to validity: The study has one verifier, one author, one annotator, and 11 registry entries across five repositories owned by that author, with no control condition.The two intervention arms differ in effort and targeting, and the score is host-dependent.

13 Fixes

The fixes tighten refusal handling, fixture scoring, CI enforcement, population accounting, provenance, artifact binding, and cross-platform robustness. Several actions remain aimed at external validation, broader coverage, preregistered operators, and attack-taxonomy-based fixture design.

  • 13 Fixes: Unknown severity, uncovered artifacts, non-numeric summaries, and absent stamp keys now receive explicit refusal outcomes rather than being silently accepted or skipped.The specification records the corresponding rules, including evidence-unchecked and summary-outruns-checks.
  • 13 Fixes: 20 tamper fixtures now accompany fixes, all refused against a passing live control, and the mutation sweep runs as a CI floor at 0.99.The two unreachable refusals were resolved by deletion or documented exclusion.
  • 13 Fixes: The auditor’s finding that real-series accuracy was unbound from its fails vectors remains a concrete specification defect requiring evidence binding.In fixtures/valid, beta:b1 declares 0.66 and 0.9 despite materially lower implied values.
  • 13 Fixes: The symlink escape remains an open fix because a covered artifact outside the bundle can still produce structural verification PASS.This is a fifth vacuous pass outside the refusal-site population.
  • 13 Fixes: Planned hardening includes preventing skipped issuer-checkout tests, re-emitting drifted bundles, and obtaining an external issuer.Five same-author repositories constitute one fixture rather than five independent issuers.
  • 13 Fixes: The proposed coverage ladder advances from artifact-read to field-binding to claim coverage, where externally visible claims identify their evidence sources and verifier obligations.The paper also proposes preregistering a second operator family and running cross-platform CI.
  • 13 Fixes: Tamper fixtures should be derived from a published attack taxonomy rather than imagination, because a threat model cannot contain its own blind spot.A robustness pull request also targets interpreter-dependent decoder behavior.
  • 13 Fixes: The refusal-site population should be pinned so refactors that merge sites abort the sweep instead of silently shrinking its denominator.The implementation adds expected raw and scored population checks plus an explicit override for deliberate measurements at another revision.

14 Conclusion

The study finds that systematic refusal-site liveness testing exposes substantially more untested acceptance logic than regression testing of discovered defects. Its strongest limitation is that the evidence is self-measured, with no external issuer yet testing the verifier independently.

  • 112/119 refusal sites were covered after writing one liveness test per surviving refusal, compared with 39/119 after regression testing the four discovered defects.The liveness arm reached 112/119; covering the remainder reached 1.000.
  • The sixteen-fixture corpus scored 10/112, or 0.089, as a standalone detector of refusal-site failures.Downstream of a unit suite asserting each fixture’s verdict, its marginal contribution was 0.
  • The study’s most consequential missing validation is an external issuer building its own bundle and triggering an unanticipated refusal.The author identifies one external issuer as the change most likely to alter the picture.

Reproducibility and artifacts

The paper provides public, version-pinned artifacts and commands for reproducing its measurements, while documenting checkout, host, and line-ending conditions that can silently alter what is measured or why tests refuse.

  • Artifacts: Public code, specifications, mutation sweeps, fixtures, registry, and archived outputs are available under the MIT licence.The repository is identified as the source for the reproduction materials.
  • Artifacts: DOI 10.5281/zenodo.22000912 pins release v0.1.0 to commit 11291be, while every quantitative claim names the commit producing it.The concept DOI identifies the project; the version DOI pins a specific result.
  • Reproduction commands: At 92e4548, the documented commands collect 258 tests, verify valid fixtures successfully, reject tamper fixtures, and run the mutation sweep over 146 refusal sites.The commands require Python 3.11 or 3.12 and installation with pip install -e ".[test]".
  • Forgery reproduction: The four hand-found forgeries are pinned by committed fixtures, with Table 4 mapping their refusal reasons and identifying which reproduce the pre-fix behavior.At 92e4548 all four exit 1; in an f59fb62 checkout, the first, second, and fourth exit 0, while the third needs a second fixture for the post-fix refusal.
  • Reproduction boundaries: Reproduction is host-dependent: sibling issuer checkouts require variables that the workflow does not fully set, and binding only three of five adds six tests rather than fourteen without reporting the discrepancy.With none bound, the suite runs 238 passed, 16 skipped, and 4 xfailed; with all five, it runs fourteen more.
  • Reproduction boundaries: Windows clones with core.autocrlf enabled rewrite fixture bytes, causing thirteen hash mismatches and making fourteen of twenty tamper fixtures refuse for the wrong reason.The repository lacked .gitattributes at 92e4548; adding one line at 81f50cf makes both differential-clone arms exit 0.
  • Archived measurements: The archived mutation measurements include 37/112, 39/119, 112/119, 126/133, and 143/143, with seven additional sweeps documenting a non-monotone record.The additional files include intermediate results such as 39/119, 112/117, 120/125, and 143/143.
  • Measured reproduction: The standalone fixture-corpus score is 10/112 = 0.089, while its liveness detector catches 0/112 = 0.000.Both measurements come from fixture_corpus_score.py, with the detector flag enabled for the second result.
Loading 2608.26183v1…