PaperlayerView exact revision on arXiv

Source-linked AI summary

Boundary-Mutation Testing for Pattern-Based Secret Detection: A Rule-Level Method and Cross-Scanner Evaluation

Shweta Mishra

arXiv:2609.02983v1cs.CRcs.SE

TL;DR

Example-based validation fixes the credential’s surrounding context, though anchored regular expressions can depend on that boundary. Boundary-mutation testing varies realistic contexts using rule-derived credentials and rule-level outcome classification. Across three scanners it exposes distinct behaviors, while the comparison supports no ranking across ten credential types.

  • Problem

    Example-based tests can share a canonical embedding and miss boundary-dependent failures, even though all five defective primary-subject rules and six Gitleaks rules pass their projects’ own suites.

    Source: A. Example-based validation has a… · p. 14

  • Method

    Boundary-mutation testing generates credentials from each rule, embeds them in realistic contexts, and judges outcomes at rule level to distinguish misses from mislabelled detections.

    Source: §XVI · p. 16

  • Results

    Across the scanners, one widely deployed credential format is structurally immune while another fails once in 64; the primary subject’s five defective rules were repaired, with no new real-code false positives.

    Source: §XVI · p. 16

  • Takeaways & Limitations

    Rule-label and severity metrics matter because identical worst-case boundary-robustness scores can represent either silent leaks or downgraded alerts.

    Source: B. Rule-level oracles are not optional · p. 14

    Per-type detection is deterministic, leaving ten credential types—not hundreds of samples—as the effective comparison sample, and no pairwise difference reaches significance.

    Source: D. Conclusion validity · p. 16

Abstract

from arXiv · show

Pattern-based secret scanners are commonly validated with example-based fixtures that fix one variable: the text surrounding a credential. We introduce boundary-mutation testing to vary that context, generating credentials from each rule's own regular expression, embedding them in realistic source contexts, and classifying outcomes at the rule level rather than the tool level, yielding three detection metrics. Applied to three scanners - a 43-rule open-source scanner, Gitleaks 8.21.2, and TruffleHog 3.82.13 - detection in the primary subject holds at >=0.9976 across ten contexts but collapses to 0.5233 when a credential ends in a hyphen. Five rules are affected: two, with fixed-count quantifiers, fail totally and deterministically; three, with variable-count quantifiers, backtrack and match a truncated credential; an entropy fallback rescues some failures but downgrades their severity. We validate a repair restoring full robustness with no new false positives, and report a caution: a plausible first attempt silently regressed two rules, caught only by re-running the same battery. Gitleaks has an unrelated, source-confirmed defect - a hard-coded terminator allowlist causing total misses for most credential types - while TruffleHog shows no boundary fragility but narrowest coverage. We report marginal, not conditional, failure probabilities: one common token format is structurally immune, another fails once in 64; on 292,527 lines of real code, the false-positive ordering inverts relative to the synthetic corpus. Because per-type detection is deterministic, the comparison unit is ten credential types, not hundreds of samples; no recall difference reaches significance, so we report the null result, not a ranking.

I. INTRODUCTION

Example-based fixtures hold credential context constant, leaving context-dependent rule failures untested. The paper introduces boundary-mutation testing and applies it across three scanners to characterize detector behavior rather than rank tools.

I. INTRODUCTION

  • Example-based fixtures commonly place credentials in one canonical context, so they can miss failures triggered by different surroundings or final characters.The paper identifies bare function arguments and hyphen-ending credentials as examples of untested cases.Source: §I · p. 1
  • Boundary-mutation testing generates credentials from each rule, embeds them in realistic contexts, and classifies outcomes at rule level, yielding three metrics.The metrics distinguish boundary robustness, rule-label preservation, and severity preservation.Source: §I · pp. 1–2
  • The method transfers across three scanners and exposes distinct behaviors, but the evidence does not support a statistically powered ranking.The authors report scanner-specific defects and a boundary-robust detector whose limits lie elsewhere.Source: §I · pp. 1–2

II. RESEARCH QUESTIONS

  • The research questions cover context sensitivity and its mechanisms, transfer to other scanners, tool comparisons under varied conditions, and false positives on real code.The related-work discussion says prior studies had not systematically varied credential context while attributing outcomes at rule level.Source: §II · pp. 2–3, §III · p. 3

III. RELATED WORK

  • The method adapts metamorphic testing by generating inputs from the detector’s own rules and using a rule-level rather than output-level oracle.The paper distinguishes syntactic rule scope from provider-semantic validity.Source: §III · p. 3
  • For deterministic outcomes within a credential type, the type—not each credential sample—is the independent observation in tool comparisons.The authors use paired testing at the type level to avoid inflating evidence with repeated samples from the same type.Source: §III · p. 3

IV. SUBJECT SYSTEMS

The study evaluates secret-scanning rules as the primary subject, distinguishing that scope from surrounding system components. It also introduces boundary-mutation testing as the method whose elements are defined in the next section.

IV. SUBJECT SYSTEMS

  • The primary subject is a 43-rule regex scanner with per-rule severities and an entropy fallback, selected for exhaustive study rather than deployment-scale claims.All 43 rules were examined; the paper describes the subject as structurally representative of regex-plus-entropy scanners.Source: §IV · p. 3
  • The scanner is the object of the research claims; ingress, queue, and worker components are supplementary, and the LLM routing layer is unmeasured.The authors say the operational characteristics of those other subsystems are outside the research questions.Source: §IV · p. 3
  • The comparison systems are Gitleaks 8.21.2 and TruffleHog 3.82.13; TruffleHog’s different architecture is treated as a distinct evaluation condition.Gitleaks is rule-table-driven, while TruffleHog uses detector-driven co-occurrence and live-verification logic.Source: §IV · p. 4

V. METHOD: BOUNDARY-MUTATION TESTING

  • Section V defines the elements of boundary-mutation testing summarized in Fig. 1.The figure caption describes rule-derived credentials embedded in realistic contexts and outcomes classified at rule level.Source: §V · p. 4

A. Definition

Boundary-mutation testing checks whether credentials generated from a rule remain detectable as their source context changes. It distinguishes syntactic scope from provider-issued validity and separately measures detection, rule labeling, and severity.

A. Definition

  • For each rule, a generator produces strings from its regex language, embeddings place credentials in source lines, and the tested tool returns fired rule names.An embedding is applicable only when it preserves membership in the rule’s language.Source: A. Definition · p. 4, The metamorphic relation · p. 5

Syntactic versus semantic validity:

  • Generated strings are syntactically in-scope, not necessarily provider-minted; a miss on an impossible credential may have no field consequence.The method detects inconsistency with the rule’s declared scope, not provider-semantic invalidity.Source: Syntactic versus semantic validity · pp. 4–5

The metamorphic relation:

  • The metamorphic relation requires detectability to persist across semantics-preserving context changes, localizing inconsistency without an independent detector oracle.The compared embeddings differ textually but preserve whether the string is a credential.Source: The metamorphic relation · p. 5

B. Metrics

  • Boundary robustness captures worst-case detection across applicable contexts; rule-label preservation tracks correct labels, while severity preservation separately tracks reported severity.A wrong label can retain the correct severity, so label and severity preservation are distinct.Source: B. Metrics · p. 5
  • SP < 1 indicates incorrect reported severity and RLP < 1 incorrect rule attribution; severity preservation is reported only where a severity oracle exists.The external scanners’ severity models are not mapped to the primary subject’s.Source: B. Metrics · p. 5

C. Instantiation

The instantiation generates credentials from the primary subject’s rules and tests them across a representative, mechanism-based boundary battery.

C. Instantiation

  • Of 43 rules, seven structural and twelve keyword-anchored rules are excluded, leaving 24 analyzed rules with 120 samples per cell.Keyword-anchored rules would be unmatchable under a neutral embedding by construction.Source: C. Instantiation · p. 6
  • The boundary battery contains twelve embeddings from six families, each representing a distinct syntactic mechanism that places a character beside a credential.Members within a family vary the adjacent character while holding the mechanism fixed.Source: Constructing the boundary battery · p. 6

Constructing the boundary battery:

  • The battery is representative rather than exhaustive; its adequacy is argued from mechanism coverage and the distinct scanner behaviors it exposed.The authors explicitly note that a boundary family is missing.Source: Constructing the boundary battery · p. 6

Freezing the expected rule for external scanners:

External scanners’ expected rules are frozen from canonical-context behavior, while controls address oracle masking, measurement scope, and researcher bias.

Freezing the expected rule for external scanners:

  • For each external scanner, the expected rule is empirically identified in canonical embeddings and frozen; absent or inconsistent detections are reported as coverage gaps, not fragility.A type is included in aggregate results only when the same identifier fires on all canonical samples.Source: Freezing the expected rule for… · p. 6
  • A plausible but wrong, consistent canonical attribution remains a residual risk, though inspection found every frozen identifier semantically correct.The procedure cannot detect an attribution that is both consistent and semantically plausible.Source: Freezing the expected rule for… · p. 6

D. A control that proved necessary

  • Using SECRET as an embedding identifier triggered generic keyword rules and masked specific-rule failures; reported results therefore use the neutral identifier cfg_value.The initial harness appeared to detect every sample, illustrating why rule-level attribution mattered.Source: D. A control that proved necessary · p. 6

E. What we did not measure

  • The study measures neither the subject’s LLM behavior nor its operational behavior; those systems are outside the rule-table research questions.No funded provider credential or outbound access was available for the LLM evaluation; operational measurements appear in an online resource.Source: E. What we did not measure · p. 6

F. Researcher bias and evaluation controls

  • Controls froze the subject before benchmarking, pinned external binaries, used identical corpora, retained contradictory outputs, and made no post-result fixes; unfavorable findings include the subject’s own defect.The authors report no statistically significant recall differences favoring the subject and withdraw a ranking claim.Source: F. Researcher bias and evaluation… · pp. 6–7
  • Credentials generated from each tool’s declared rules constrain corpus tuning that might otherwise favor one scanner’s credential shapes.Measurements ran in one containerized Linux environment; the comparison binaries were pinned Gitleaks 8.21.2 and TruffleHog 3.82.13 releases.Source: F. Researcher bias and evaluation… · p. 7, A. Environment · p. 7

B. Corpora and sampling

The context battery reveals a sharp hyphen-boundary failure despite near-perfect detection in the other contexts. Results are interpreted at the credential-type level because outcomes are deterministic within each type.

B. Corpora and sampling

  • The mutation corpus covers 24 rules with 120 samples per rule-context cell; comparison uses 400 positives across ten shared credential types, while real-code evaluation scans 292,527 lines.Comparison negatives are fixed at 700; the real-code corpus contains unmodified source from four open-source projects.Source: B. Corpora and sampling · p. 7

C. Determinism and seeding

A. Reproducibility and repeated-run robustness

  • Two fixed-count rules have BR = 0.0000 deterministically; three variable-count rules produce truncated matches at rates from 92.5% to 98.3% across repetitions.Variable-length slack enables backtracking; whether truncation matches depends on generated length and the preceding character.Source: A. Reproducibility and repeated-run… · pp. 7–8

VII. RQ1: CONTEXT SENSITIVITY

  • ≥0.9976 detection holds across ten of twelve contexts, but hyphen termination drops aggregate detection to 0.5233 (23.2% entropy-rescued; 24.5% unreported).Underscore termination remains harmless at 1.0000.Source: §VII · p. 7, Table IV · p. 8

B. The three metrics, per rule

Boundary robustness and label-preservation metrics reveal distinct failures that aggregate detection alone obscures. Hyphen-ending credentials expose an anchor failure whose severity depends on regex quantifier type and whether the entropy fallback can rescue the candidate.

B. The three metrics, per rule

  • SendGrid and GCP both have BR = 0.0000, but SendGrid has RLP = 1.0000 while GCP has RLP = 0.9116 because their failures are respectively total misses and rescued, downgraded detections.The table shows the remaining nineteen rules score 1.0000 on all three metrics.Source: B. The three metrics, per rule · p. 8, Table V · p. 8
  • Five rules are boundary-fragile: OpenAI API Key (new), GCP API Key, Google OAuth Token, SendGrid API Key, and JWT Token.A single scalar cannot capture both detection and labelling failures; the metric that moves indicates which kind of problem occurred.Source: §VIII · p. 8

VIII. RQ2: MECHANISM AND MARGINAL PROBABILITY

  • A trailing \b fails when a hyphen-ending credential is followed by non-word text, because the join is not a word boundary.The affected rules permit hyphens in their character classes, and suppression predicates were false in the failing samples.Source: §VIII · p. 8
  • Fixed-count quantifiers fail totally, while variable-count quantifiers backtrack and match a credential one character short.SendGrid and GCP have 0.0% rule firing; the variable-count rules fire at 84.2%–89.2% in the listed cases.Source: §VIII · pp. 8–9
  • The entropy fallback rescues dotless GCP and OpenAI candidates but downgrades them to MEDIUM; dot-containing SendGrid, Google OAuth, and JWT candidates are not rescued.The fallback’s character class excludes dots, so rescue and severity preservation diverge across the affected credential types.Source: §VIII · p. 9

A. Marginal, not conditional, probability

The marginal risk depends on how a credential’s tail is generated: uniform 64-symbol tails end in a hyphen once in 64, while JWT signature encoding can make that ending impossible. The section also specifies two local replacement candidates for the faulty trailing boundary assertion.

A. Marginal, not conditional, probability

  • Uniform 64-symbol tails have P(ends in “-”) = 1/64, while HS256 JWTs are structurally immune and HS384 JWTs are not.The paper reports the JWT case as impossible or roughly one in 548 depending on signature algorithm.Source: A. Marginal, not conditional,… · p. 9
  • The proposed local remedy replaces the trailing \b with (?![A-Za-z0-9_]) in the five affected rules.The paper describes the replacement as removing the hyphen special case while preserving the original word-boundary condition.Source: A. Marginal, not conditional,… · p. 9

IX. AUTOMATED REPAIR AND FIX VALIDATION

  • The two candidate repairs were applied in memory and assessed by rerunning the full boundary-mutation battery without modifying the subject repository.Source: §IX · p. 10

A. Two candidates

  • Candidate A forbids a following word character or hyphen, whereas Candidate B forbids only a following word character.Both candidates replace the trailing \b on the same five rules, leaving all other rules and the entropy fallback unchanged.Source: A. Two candidates · pp. 10–11

B. Both candidates repair the defect

Both repair candidates restore boundary robustness on the five affected rules, but only Candidate B preserves behavior on a regression probe that the original rules handled. Applying the same mutation battery across scanners exposes distinct, tool-specific degradation patterns.

C. Candidate A regresses; Candidate B does not

  • Candidate A fails at 0% on the two fixed-count rules when a word-character-ending credential is followed by a hyphen; Candidate B fires at 100% on all five rules.Candidate A’s lookahead unconditionally forbids the following hyphen, while Candidate B preserves the original rule’s accepted case.Source: C. Candidate A regresses; Candidate B… · p. 11

X. RQ3: DOES THE METHOD TRANSFER?

  • The battery transfers across scanners: two degrade in different contexts, while the third shows no boundary degradation.External-tool expected rules were identified from each tool’s own behavior in the canonical context, and uncovered types were reported separately from fragility.Source: §X · p. 11, Figure 7 · p. 12
  • Gitleaks rule-level detection falls to 0.333 on bare call arguments, with six credential types dropping to 0.000 because their rules use a hard-coded terminator allowlist.The source-confirmed allowlist omits rejected terminators such as commas and closing parentheses, while three unaffected types lack that trailing group.Source: §X · p. 11
  • TruffleHog shows no boundary fragility on covered types but covers the fewest, detecting 6 of 10 credential types canonically.The paper relates this boundary-robust profile to its co-occurrence- and verification-oriented detectors.Source: §X · p. 11
  • The three scanners exhibit distinct boundary strategies: implicit anchors have probabilistic failures, terminator allowlists have deterministic failures, and boundary independence accompanies narrower coverage.Source: §X · pp. 11–12

B. Both candidates repair the defect

XI. RQ4: COMPARISON, CORPUS, AND DEPLOYMENT CONDITION

The comparison does not support ranking the scanners: per-type detection is deterministic, leaving only ten credential types as independent comparison units, and no difference reaches significance. Corpus construction and deployment conditions nevertheless reveal meaningful trade-offs in recall and false positives.

XI. RQ4: COMPARISON, CORPUS, AND DEPLOYMENT CONDITION

  • On the synthetic corpus, Gitleaks recall rises from 0.800 to 0.900 with corpus construction, while the subject remains at 1.000 and has 17 false positives.The subject’s shallower prefix, character-class, and length rules are identified as the reason for its perfect recall and false positives.Source: §XI · p. 12

A. The comparison supports no ranking

XII. RQ5: FALSE POSITIVES ON THE EVALUATED REAL-CODE CORPUS

On the four-project real-code corpus, the subject had the lowest findings-per-kLOC rate, reversing the synthetic-corpus ordering; the authors caution that the result is limited to these projects. The section also shows how fixture design, rule-level outcome measurement, scanner boundary strategies, and benchmark construction affect conclusions.

XII. RQ5: FALSE POSITIVES ON THE EVALUATED REAL-CODE CORPUS

  • All ten subject findings were false positives: six documentation URL templates, three doctest passwords, and one entropy-fallback match on an RFC 3986 unreserved-character constant.The scans covered 292,527 lines from four pinned, unmodified, Python-centric projects; the authors make no general false-positive-rate claim.Source: §XII · pp. 13–14

A. Example-based validation has a blind spot with a shape

  • All five defective subject rules and all six defective Gitleaks rules pass their own suites because fixtures use canonical credential positions, missing boundary-dependent behavior.The paper says a small battery of twelve rule-generated embeddings, run in under twenty seconds over 24 rules, detects these defects.Source: A. Example-based validation has a… · p. 14

B. Rule-level oracles are not optional

  • Rule-level oracles separate missed credentials from detected credentials assigned the wrong rule or severity, distinctions a tool-level reported/not-reported score conceals.The paper notes that GCP and OpenAI keys can appear detected while their severity is downgraded; BR, RLP, and SP distinguish these outcomes.Source: B. Rule-level oracles are not optional · p. 14

C. Boundary strategy is a design axis with real consequences

  • Implicit anchors fail according to a credential’s final character, terminator allowlists according to the following character, while boundary-independent detection avoids both but covers fewer shapes.The authors characterize these as different scanner strategies, not a basis for declaring a winner.Source: C. Boundary strategy is a design axis… · p. 14

D. Benchmark construction, not only tool quality, decides what a comparison reports

  • Benchmark choices changed reported outcomes: corpus construction moved Gitleaks by +0.10 recall, deployment condition moved TruffleHog by +0.33 detection, and negative-corpus choice inverted precision ordering.Fig. 9 indicates TruffleHog’s recall depends strongly on corroborating context; the other two tools do not.Source: D. Benchmark construction, not only… · p. 14, Figure 9 · p. 14

XII. RQ5: FALSE POSITIVES ON THE EVALUATED REAL-CODE CORPUS

  • 0.034 findings per kLOC for the subject, versus Gitleaks’ 0.208 and TruffleHog’s 0.233, reverses the synthetic-corpus ordering.The authors attribute the inversion to synthetic negatives over-representing bare high-entropy base64 assignments and under-representing documentation and configuration idioms.Source: §XII · pp. 13–14, Figure 10 · p. 15

XIV. THREATS TO VALIDITY

The authors qualify the evaluation by noting limits in generated credentials, probability assumptions, harness validity, corpus scope, and the effective sample size for cross-tool comparisons. In particular, deterministic per-type detection leaves ten credential types—not hundreds of generated credentials—as the comparison unit.

A. Construct validity

  • Generated positives guarantee syntactic, not semantic, validity, so field consequences may differ for strings a provider would never issue.Marginal failure probabilities assume a uniform final character within each credential type’s alphabet; JWT rows additionally depend on enumerated signature lengths, and variable-quantifier truncated-match rates vary across generated lengths.Source: A. Construct validity · p. 15

B. Internal validity

  • The corrected seeded harness is reproducible, but expected-rule freezing cannot mechanically exclude a consistent, semantically plausible attribution error.The authors also report that TruffleHog was run with --no-verification, and its co-occurrence mechanism was established by probing rather than source inspection.Source: B. Internal validity · p. 15

C. External validity

  • The real-code evaluation covers four Python-centric projects, not a representative sample of GitHub, and cross-tool results cover ten shared credential types rather than full rule tables.The paper therefore makes no general claim about field false-positive rates.Source: C. External validity · pp. 15–16

D. Conclusion validity

  • Ten credential types, not 400 credentials, are the effective sample for tool comparison; no pairwise difference reaches significance, so the authors draw no ranking.Real-code ground truth assumes released public code contains no live credentials; an undetected genuine leak would shift the counts, and TruffleHog’s figures are a lower bound under adversarial conditions.Source: D. Conclusion validity · p. 16

XV. FUTURE WORK

The paper proposes extending the method to encoding transformations, full external-scanner rule tables, broader real-code corpora, external severity oracles, and a repair-and-validation loop for Gitleaks. Its conclusion emphasizes testing rules and proposed repairs with the same battery, while reporting that the comparison cannot rank the tools.

XV. FUTURE WORK

  • Future work proposes testing encoding transformations, scanning the full external rule tables, broadening real-code sampling across languages, adding severity oracles, and validating a Gitleaks repair.These extensions would address currently untested transformations, limited cross-tool credential coverage, a four-project corpus, missing external severity comparisons, and an unpatched terminator-allowlist defect.Source: §XV · p. 16

XVI. CONCLUSION

  • Boundary-mutation testing transferred to three scanners and exposed distinct boundary behaviors; the primary subject’s five-rule repair passed the full battery with no new real-code false positives.A plausible inspection-only repair regressed two rules, caught only by re-running the same probe; the paper also concludes that the comparison cannot rank the tools.Source: §XVI · p. 16

STATEMENTS AND DECLARATIONS

  • The author is the creator and maintainer of the evaluated subject; the paper reports controls including a frozen subject version, pinned external binaries, identical corpora and conditions, and retention of raw outputs.Raw results and harnesses are included in a replication package archived on Zenodo.Source: STATEMENTS AND DECLARATIONS · pp. 16–17
Loading 2609.02983v1…