PaperlayerView exact revision on arXiv

Source-linked AI summary

The Exclusion Ratchet: False-Positive Suppression Accumulates and Persists in Detection Rule Repositories

Sudaroli Dhananjeyan, Kumaran U

arXiv:2608.31062v1cs.CRcs.SE

TL;DR

False-positive exclusions may narrow detection rules in ways that longitudinal structural analyses miss, while their later persistence remains poorly measured. This paper applies a deterministic semantic test to nine years of SigmaHQ revisions and finds that exclusions accumulate and usually remain, motivating consequence-aware review rather than verdicts about benignness.

  • Problem

    The paper addresses the missing evidence on narrowing that is never undone, beyond prior restoration-time analysis limited to reverted revisions.

  • Method

    The study semantically tests 8,234 SigmaHQ revisions for growth in predicates held under negation without compensating coverage growth, using deterministic definitions released as code.

  • Results

    Exclusions were added 5.4 times per withdrawal overall and 13 times per withdrawal at the individual-rule level, while 31 per cent of narrowing escaped structural comparison.

  • Takeaways & Limitations

    The findings show that narrowing accumulates, is rarely revisited, and can be prioritized using breadth and whether an adversary can satisfy an exclusion without privilege.

  • Takeaways & Limitations

    The study cannot determine whether an exclusion is wrong or benign because it observes no deployment environment.

Abstract

from arXiv · show

When a rule produces too many false alarms an analyst adds an exclusion, and the rule thereafter declines to alert in that circumstance. Each such decision is locally reasonable; what becomes of them collectively is not known. Recent longitudinal work established that curation does not converge, but measured restoration time only for revisions that were later reverted -- a measure silent about narrowing that is never undone. We measure that. Across nine years and 8,234 revisions of the SigmaHQ corpus we detect suppression semantically -- growth in the set of predicates held under negation without compensating growth in coverage -- and validate it against blinded hand labelling (precision 0.828, recall 0.911). The test is deterministic: nothing is learned from the data, and the definitions are released as code. Exclusions were added 1,642 times and withdrawn 304, a ratio of 5.4 to 1 that rises to 13 to 1 at the level of the individual rule. Thirty-one per cent of the narrowing is invisible to structural comparison, which existing structural accounts therefore undercount. Estimated by Kaplan-Meier, 86.7 per cent of exclusions remain in force three years on, and persistence is independent of whether the rule is the only coverage for its ATT&CK technique (p = 0.49). Of path-valued exclusions, 64.1 per cent can be satisfied by an unprivileged process that chooses a filename. Narrowing accumulates, is rarely revisited, and is not triaged by consequence. We give a criterion for deciding which exclusions to examine first.

1. Introduction

The paper examines whether locally reasonable false-positive exclusions accumulate into persistent, semantically meaningful narrowing. It introduces a deterministic semantic detector and finds that narrowing is common, often invisible structurally, and rarely reversed.

  • The tuning problem: 64.1 per cent of path-valued exclusions can be satisfied by an unprivileged process choosing a filename, compared with 33.0 per cent requiring privilege.The cost of an exclusion therefore depends on whether an adversary can arrange the excluded value.
  • The research gap: The prior restoration-time measure covers only revisions that were later reverted, leaving persistent narrowing outside its population.The paper addresses this gap by detecting exclusion additions independently of later restoration.
  • Contributions: The deterministic semantic test is defined over predicates held under negation, uses no learned component, and is validated against blinded hand labelling.Its validation achieved precision 0.828 and recall 0.911.
  • Why structural analysis cannot locate suppression: 31 per cent of narrowing is invisible to structural comparison because appending a value to an existing list changes behavior without changing rule structure.The paper therefore treats exclusion growth at both predicate and value level.
  • Contributions: Exclusions were added 5.4 times for every withdrawal overall and 13 times for every withdrawal within individual rules.The contribution describes this pattern as an exclusion ratchet.
  • Contributions: 86.7 per cent of exclusions remain in force three years after addition, while persistence is independent of whether the rule is the only coverage for its ATT&CK technique.The paper also characterizes exclusion breadth and adversary satisfiability to prioritize review.

2. Background

Sigma is a repository format for detection rules that match log records and raise alerts, while exclusions alter those rules to suppress selected matches. The paper’s background explains Sigma’s rule structure, its ATT&CK tags, and the different ways exclusions can be represented.

  • Detection rules and the Sigma format: A detection rule is a standing query over telemetry fields, and Sigma expresses such patterns independently of a specific SIEM before compilation.Matching records raise alerts for analyst triage.
  • Detection rules and the Sigma format: A predicate is a single field–value test, and every quantity reported in the study derives from changes to the detection block.Rule lineages are joined by stable UUIDs rather than file paths.
  • Detection rules and the Sigma format: A Sigma detection block conjoins keys, disjoins values under one key, and treats a list of maps as disjoined alternatives.These conventions determine the boolean expression evaluated against each log record.
  • How suppression is expressed: A list-valued field remains one predicate even when literals are appended, so behavior can narrow while predicate count and condition shape remain unchanged.The LSASS example adds one value and stops matching taskhostw.exe-originated access, illustrating why structural comparison misses this case.
  • How suppression is expressed: Suppression means changing rule content so the rule matches a narrower set of records, rather than throttling, deduplicating, or prioritizing alerts after firing.Sigma expresses one exception by adding a named block whose complement is conjoined with the existing selections.
  • How suppression is expressed: SigmaHQ later introduced standalone filter rules that apply exclusions centrally across multiple detection rules, but those exclusions may fall outside the analyzed rule lineages.The mechanism existed for roughly the final twenty months of the observation window.
  • ATT&CK tagging and the corpus: ATT&CK tags connect local rule edits to technique coverage, but they are self-reported, can be inaccurate, and can change as the catalogue evolves.The analysis therefore keys coverage on technique identifiers rather than tactic names and treats coverage as fixed at the snapshot.

3. Related work

Prior work explains why analysts narrow noisy rules and how alerts and rulesets evolve, but largely does not measure what individual rule text records over time. This paper bridges that gap by measuring semantic narrowing in maintained detection artefacts and its coverage consequences.

  • Practitioner studies report that exclusions are added routinely and revisited rarely, but rely on interviews rather than rule artefacts.
  • Alert-stream research makes downstream triage more tractable, whereas narrowing changes the rule itself and has received little study.
  • Longitudinal ruleset studies track rule presence or syntactic transformations, not the semantic effect of one rule’s revisions on matched events.
  • The paper measures accumulation where detection artefacts are maintained and narrowing costs coverage, complementing work showing that detections can be defeated.
  • Structural comparison misses narrowing expressed through negation or added values, although these forms account for 5,927 of 8,234 predicate-changing revisions, or 72 per cent.

4. Method

The method infers suppression from adjacent Sigma rule versions using a deterministic semantic test, then characterises exclusions by adversary controllability and validates blind spots against hand labelling. Its scope is the SigmaHQ repository and its released lineage pipeline, not any particular enterprise deployment.

  • Unit and test: The unit of analysis is an adjacent pair of versions within one rule lineage, evaluated across 8,234 predicate-changing revisions.
  • Unit and test: The classifier returns five deterministic verdicts: suppression, relaxation, guarded expansion, rewrite with no net change, or no exclusion-set change.
  • Unit and test: Suppression is identified semantically as exclusion growth without compensating coverage growth, while guarded expansions are protected from being counted as suppression.
  • Unit and test: The detector ignores wildcard and empty-string literals as non-constraining predicates, preventing formatting artefacts from inflating exclusion counts.
  • Blind spots: Three mechanisms remain invisible: broad generalisation, compound revisions, and case-variant deduplication; together they bias reported narrowing downward.
  • Scope and validation: Validation uses agreement with blinded hand labelling, but one rater means no inter-rater agreement is available.
  • Exclusion consequence: Path-valued exclusions are classified by whether an adversary needs privileged access or can satisfy the value through an unprivileged action.
  • Scope and validation: Coverage is assessed at the snapshot while exclusions retain historical dates, answering whether standing exclusions now have fallback coverage rather than what maintainers knew when editing.

5. Results

Exclusions accumulate across SigmaHQ revisions, and this narrowing is concentrated within rules rather than being offset by later relaxations. Structural comparisons miss roughly a third of suppression because value-level additions can leave rule structure unchanged.

  • 1,642 suppressions versus 304 relaxations produce a 5.4-to-1 withdrawal ratio across 8,234 revision steps.
  • 86.0 per cent of modified lineages end the observation period net-narrower, while narrowing rules tend to narrow repeatedly and widening rules usually widen once.
  • 4.9 to 1 remains the suppression-to-relaxation ratio after removing the twenty busiest days, supporting accumulation beyond bulk editing.
  • 31 per cent of suppressions add values to existing exclusions without changing predicate, block, or condition-line structure.
  • Structural accounts therefore undercount narrowing by roughly a third before additional unseen mechanisms are considered.

5.3. Exclusions are effectively permanent

Exclusions are rarely removed after introduction, and their persistence does not depend on whether other rules cover the same ATT&CK technique. Removals occur mainly early, while sole-coverage exclusions remain especially consequential because they lack fallback detection.

  • 86.7 per cent of exclusions survive three years, according to the Kaplan–Meier estimate.
  • 978 of 1,095 days is the restricted mean survival time over three years, meaning exclusions remain in force for about 89 per cent of that window.
  • 52 per cent of removals occur within six months and 76 per cent within one year; the hazard approaches zero after roughly two years.
  • Persistence is indistinguishable between sole-coverage and redundantly covered rules, with a log-rank p = 0.49.
  • Sole-coverage exclusions remain a severity concern: 63 of 71 such exclusions are still in force.

5.5. What the exclusions are written on

Most exclusions target attributes an adversary can influence, but path exclusions differ in the privilege and breadth required to satisfy them. The resulting two-axis classification identifies broad, freely enterable exclusions as the first review priority.

  • 86.0 per cent of the 3,606 exclusion predicates target process, file, or registry content, while 10.0 per cent target network identity.
  • 64.1 per cent of path-valued exclusions can be entered by an unprivileged process choosing a filename, while 33.0 per cent require privileged access.
  • 50.5 per cent of path literals are freely enterable and specific, 12.3 per cent freely enterable and broad, 18.4 per cent privileged and narrow, and 13.3 per cent privileged and broad.
  • 40.3 per cent of protected-path exclusions are directory trees or multi-wildcard patterns rather than named files.
  • Review should start with the 12.3 per cent that are broad and freely enterable; the 18.4 per cent that are narrow and privileged should be left alone.
  • Command-line exclusions have a median length of 19 characters, compared with 47 for registry and data exclusions.

5.6. Temporal structure

Suppression becomes more common as SigmaHQ evolves and is not confined to initial rule tuning. Exclusions are often added well after rule creation, then persist through the remainder of the observed lifecycle.

  • 0.57 net exclusions per live rule by 2026 rises from under 0.1 in the early years, despite corpus growth.
  • Removing the twenty busiest days leaves the suppression-to-relaxation ratio at 4.9 to 1, so the trend is not explained by bulk editing.
  • 492 days is the median rule age when an exclusion is added, and 56 per cent of suppressions occur after the rule’s first year.
  • Suppression rises from 3.3 per cent to 7.9 per cent of revision activity between the first and second halves of the period.
  • The overall suppression effect doubles, while the predicate mechanism rises 3.4-fold; serial correlation makes rank-correlation significance optimistic.

5.7. Worked example

A worked rule history shows exclusions accumulating, broadening, and persisting even when maintainers explicitly document that they should be narrowed.

  • Exclusion persistence: 24 of 27 exclusions remain in force at the snapshot, while the three removals replace or re-spell the same exclusion intent.The replacements change representation rather than reconsidering the underlying suppression.
  • Revision activity: Figure 6 plots suppression as a share of revision activity by mechanism, showing raw monthly values behind a twelve-month rolling mean for months with at least 50 revisions.The figure distinguishes predicate-level and value-level suppression.
  • Exclusion history: Figure 7 represents each exclusion as a bar from introduction to last carriage or the still-current snapshot, with directory trees marked by †.The flattened predicate view omits the conjunction requiring the loading process to sit under C:\Windows\.
  • Exclusion broadening: After fifteen months of file-specific exclusions, the rule adds three directory-tree exclusions covering every executable beneath C:\Windows\, C:\Program Files\ and C:\Program Files (x86)\.The latter two create broad exclusions in locations requiring privileged writes.
  • Documented remedy: The rule’s Program Files comment instructs deployers to replace the broad folder with exact applications, but the exclusion remains unchanged for three and a half years.The documented remedy was not applied in the observed history.

5.8. Validation results

Blinded hand labelling supports the detector’s suppression classifications, while also showing that rewrites create the main blind spot and that commit messages independently corroborate the pattern.

  • Detector validation: Precision is 0.828 and estimated recall is approximately 0.911 on the hand-labelled validation sample.The detected arm contributed 58 decided items, of which 48 were confirmed suppressions.
  • Detector validation: 22.5 per cent of validation items could not be decided from the diff alone, indicating that environmental context is often needed to distinguish narrowing from rewriting.This uncertainty limits what diff-only judgement can establish.
  • Blind spot: 89 per cent of estimated false negatives originate in the rewrite stratum, where the paper predicts the detector is blind.Hand inspection and the quantitative estimate agree about the location of failure.
  • Commit corroboration: Of 1,642 suppressions, 1,203 match a unique same-day commit, while 439 occur on days with multiple commits and are reported separately.Commit subjects were withheld from raters and joined by lineage and date rather than position.
  • Commit corroboration: Suppression commits contain reason-language in 30.9 per cent of cases versus 10.8 per cent mechanism-language, supporting a 10.3-fold enrichment claim rather than an absolute percentage interpretation.The detector did not use commit messages, so this evidence is independent of classification.

6. Discussion

The discussion interprets persistent exclusions as a self-reinforcing review problem, identifies discoverable low-cost evasion windows, and proposes prioritising broad, privilege-free exclusions while acknowledging clear scope limits.

  • Why exclusions are not revisited: 52 per cent of removals occur within six months and 76 per cent within a year, after which the removal rate approaches zero.This timing is consistent with review occurring soon after an exclusion is introduced.
  • Why exclusions are not revisited: Persistence does not differ by whether a rule is the only coverage for its ATT&CK technique, indicating review is not triaged by coverage consequence.The paper reports this as consistent with alert-driven review rather than consequence-driven review.
  • Operational implications: The proposed review criterion combines exclusion breadth with whether an unprivileged adversary can satisfy the excluded condition without acquiring privilege.Both properties are computable from rule text before deployment.
  • Operational implications: 657 literals are prioritised for review out of 5,336, while 984 precisely drawn, privilege-costly literals can be left alone under the proposed criterion.The ordering makes a large review task finite.
  • Public discoverability: 64.1 per cent of path-valued literals can be satisfied by an unprivileged process choosing a filename, making those quiet conditions documented and cheap to arrange.The study observes no adversary or deployment and does not show that these windows were used.
  • Implications for measurement: 31 per cent of narrowing occurs by adding values to existing lists, so structural comparison undercounts suppression by roughly a third.The same list-valued-field blind spot applies beyond Sigma to formats including Splunk Security Content, Elastic rules, YARA, and firewall object groups.
  • Operational implications: The proposed interventions include expiry or review dates and surfacing coverage consequences when exclusions are proposed.A corpus example shows five bare filenames replaced by three absolute paths, with two removed.
  • Scope boundary: The method identifies broad, privilege-free, unreviewed exclusions but cannot determine whether an exclusion is wrong without observing its deployment environment.The paper therefore makes no claim about any specific deployment.

7. Threats to validity

The paper reports limitations involving upstream data, corpus scope, semantic coverage, censoring, metadata joins, and validation, while arguing that several biases make the reported narrowing ratio conservative.

  • Upstream dependency: Errors in the released upstream pipeline would be inherited silently, although the prior work was reproduced and four discrepancies were documented without affecting its headline findings.The present analysis depends on that pipeline’s revision alignment and structural records.
  • Corpus scope: The corpus is not a deployed configuration, so findings are scoped to the upstream signal reaching organisations that deploy near-stock rulesets.No claim is made about any specific deployment.
  • Corpus and annotation limits: 64.1 per cent of path-valued literals can be entered by an unprivileged process choosing a filename, but ATT&CK tags are self-reported and only spot-checked.At least one mistag was identified by hand, and coverage is keyed to technique identifiers.
  • Method limitations: Three undetected narrowing mechanisms—generalisation, compound revisions, and case-variant deduplication—bias the reported ratio downward.Value-level suppression is detected and accounts for 31 per cent, but these mechanisms are not.
  • Persistence estimation: Snapshot right-censoring means exclusions still in force may later have been removed, although survival analysis handles censoring and the reported 86.7 per cent is a lower bound.Median follow-up is 1,296 days, exceeding the reported persistence horizon.
  • Representation limits: The canonical form is not always faithful to rule text: 167 of 185 short literals were verified verbatim against the source.This leaves a residual literal-fidelity boundary in the representation.
  • Statistical limits: Months are not independent observations, making rank-correlation significance optimistic; the paper reports effect size alongside it.The limitation concerns significance interpretation rather than the reported effect size.
  • Validation limits: A single rater was used for validation, no inter-rater agreement is available, and 22.5 per cent of items could not be decided from the diff.The paper identifies a second rater on a subset as the most valuable remaining addition.

8. Conclusion

Across nine years, exclusion-based narrowing accumulated, persisted, and was partly invisible to structural comparison. The paper proposes computable signals for prioritizing exclusions while recognizing that benignity remains environment-dependent.

  • 5.4 exclusions were added for every withdrawal, rising to 13:1 per rule, and 87% remained active three years later.Persistence did not differ according to whether the rule uniquely covered its ATT&CK technique.
  • One-third of narrowing escaped structural comparison because appending values to existing lists leaves rule structure unchanged.This issue applies broadly wherever detection logic uses list-valued fields.
  • The paper identifies broad, adversary-cheap, uniquely consequential exclusions as candidates for review, using properties computable from rule text before deployment.Whether an excluded item is genuinely benign still depends on an environment absent from public corpora.
  • The analysis pipeline, derived data, and complete validation record are publicly released under an MIT licence.The exact paper version is archived in a Zenodo release, with development continuing in a public repository.

8.7. Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The first author reports responsibility for the research and describes using Claude for manuscript, code, and limited validation-item discussion assistance. No generative tool produced, selected, or interpreted any reported measurement.

  • The first author states that the conception, methods, methodological decisions, analysis, and interpretation were their work.
  • Claude assisted with drafting and editing, analysis and figure-generation code, and discussion of 15 validation items.
  • No generative tool was used to produce, select, or interpret any measurement reported in the paper.
Loading 2608.31062v1…