PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Risk-Based Alerting Mitigate Cybersecurity Alert Fatigue?

Rafael Uetz, Philipp Bönninghausen, Louis Hackländer-Jansen, Martin Henze

arXiv:2609.02465v1cs.CR

TL;DR

SOCs face mostly false alerts and limited review capacity, motivating evaluation of risk-based alerting. The paper reformulates RBA as continuous prioritization, formalizes five hypotheses in CATS, and tests them across eight datasets. Weighted combinations achieve strong cross-dataset prioritization, supporting RBA as a potential way to reduce false-alert review burden and as a baseline for more resource-intensive methods.

  • Problem

    SOCs face mostly false alerts that can exceed analysts’ review resources and risk missed true alerts, while RBA lacked systematic quantitative evaluation.

  • Method

    The paper reformulates RBA as continuous alert prioritization, formalizes five hypotheses as configurable modules, and evaluates them with CATS across alert datasets.

  • Results

    AUROC μ = 0.92, σ = 0.09 across eight datasets for a weighted combination of the five risk modules, outperforming severity-level prioritization at μ = 0.72, σ = 0.21.

  • Takeaways & Limitations

    RBA can substantially reduce the number of false alerts analysts review and serves as a baseline for more complex, resource-intensive alert prioritization methods.

  • Takeaways & Limitations

    The evaluation excludes subjective entity risk modifiers and assumes analysts require the same time to triage every alert.

Abstract

from arXiv · show

Security operations centers (SOCs) face large numbers of false alerts, making detection of cyberattacks difficult under typical resource constraints. Risk-based alerting (RBA) has been proposed as a means to reduce false alerts and has reportedly succeeded in doing so in various enterprise deployments. However, RBA has not been comprehensively evaluated until now, leaving implementation mostly guesswork based on anecdotal evidence. In this paper, we present the first systematic evaluation of RBA. To this end, we reformulate it as a continuous alert prioritization problem rather than a binary decision problem (i.e., whether an alerting threshold is exceeded), allowing us to evaluate performance across all possible thresholds and thus model SOCs of varying sizes and alert volumes. We distill five fundamental risk hypotheses, formalize them as independently parametrizable modules, and implement them in our novel experimentation suite CATS. We thoroughly assess the hypotheses across eight diverse alert datasets, six of which we created or extended to make such an evaluation possible. Our results show that certain combinations of hypotheses achieve a remarkable alert prioritization performance (AUROC $μ=0.92$, $σ=0.09$ across the eight datasets), outperforming a straightforward prioritization by alert severity level (AUROC $μ=0.72$, $σ=0.21$). We conclude that RBA can substantially reduce the number of false alerts that analysts have to review and thus has the potential to mitigate cybersecurity alert fatigue. In addition, it serves as a strong baseline for more complex, resource-intensive alert triage approaches (e.g., based on large language models).

1 Introduction

SOCs must triage many mostly false alerts, creating alert fatigue and risking missed attacks. This paper systematically evaluates RBA through five hypotheses, CATS, and diverse datasets, finding strong prioritization performance and potential to reduce false-alert burden.

  • Large alert volumes dominated by false alerts can overwhelm SOC analysts and cause them to miss actual attacks.
  • RBA has been widely adopted because it is explainable, requires no training data, and is less computationally expensive than large language model-based methods.
  • RBA had not been systematically evaluated, leaving its generalizability and implementation guidance uncertain.
  • The study formalizes five risk hypotheses covering rule severity and the spatio-temporal accumulation, variety, rarity, and periodicity of alerts.
  • AUROC μ = 0.92, σ = 0.09 across eight datasets for a weighted combination of risk modules, versus μ = 0.72, σ = 0.21 for rule-severity prioritization.
  • The authors conclude that RBA can substantially reduce false alerts requiring review, with almost negligible computational cost, and provide CATS for reproduction and tuning.

2 Background: Risk-Based Alerting

SOCs aggregate alerts from diverse detection systems, but alert volumes can exceed analyst capacity. RBA instead correlates alerts centrally and uses expert-written rules to generate risk-driven items for review.

  • Alert Sources and Triage: SOC alert triage uses centralized SIEM systems to collect alerts from multiple host- and network-based sources.
  • Alert Sources and Triage: The study uses alerts from Falco, Sigma, Suricata, and Wazuh, representing diverse open-source detection systems.
  • Alert Sources and Triage: Figure 1 records 15 060 alerts from six Falco rules during one week in an enterprise network, illustrating potential analyst overload.
  • Risk-Based Alerting: RBA centrally collects alerts in a risk index and assigns scores using expert-written risk rules instead of reviewing alerts individually.
  • Risk-Based Alerting: Risk incident rules correlate normalized alerts and generate a notable for manual review when high-risk activity is found.
  • Risk-Based Alerting: Reported deployments include a true-positive rate increase from 7.07% to 33% and false-positive rate reduction from 78% to 48%, but the evidence is anecdotal.

3 Problem Statement, Goal, and Approach

The paper frames alert triage as prioritizing true alerts under fixed analyst time, addressing the lack of quantitative RBA evaluation. It evaluates configurable risk hypotheses while excluding subjective entity modifiers and assuming equal triage time per alert.

  • Problem Statement, Goal, and Approach: Imperfect detection rules and low attack base rates make alerts mostly false and risk exceeding analysts’ review resources.
  • Problem Statement, Goal, and Approach: The goal is to help analysts find more true alerts with the same time or the same number of alerts in less time.
  • Problem Statement, Goal, and Approach: Continuous prioritization lets analysts review alerts in descending risk order and evaluates all thresholds using AUROC, AP, and Brier score.
  • Problem Statement, Goal, and Approach: Risk hypotheses assign higher scores to true than false alerts using alert-level information, information from other alerts, or both.
  • Problem Statement, Goal, and Approach: Entity risk modifiers are excluded because their definitions are subjective and potentially biased, while triage time is assumed equal for every alert.

4 Risk Hypotheses for Alert Prioritization

The study derives and evaluates five risk hypotheses for distinguishing true from false alerts: accumulation, variety, rarity, rule level, and aperiodicity. CATS implements these hypotheses as configurable modules for systematic testing.

  • Risk Hypotheses for Alert Prioritization: The literature review found 12 risk incident rules and generalized their recurring assumptions into evaluable risk hypotheses.
  • Risk Hypotheses for Alert Prioritization: Accumulation treats many alerts affecting limited entities within a short time span as higher risk.
  • Risk Hypotheses for Alert Prioritization: Variety treats many different triggered detection rules affecting an entity within a time span as higher risk, distinguishing multi-step attacks from narrow benign causes.
  • Risk Hypotheses for Alert Prioritization: Rarity assigns higher risk to alert types occurring infrequently for a rule or entity.
  • Risk Hypotheses for Alert Prioritization: Rule Level uses detection-rule severity as a risk indicator, while Aperiodicity treats irregular alert occurrence as higher risk.
  • Scope: The evaluation excludes organization-specific performance indicators and MITRE ATT&CK labels because datasets do not consistently provide the required information.

5 CATS: Alert Triage Evaluation System

CATS is an experimentation system for visually exploring and automatically evaluating parametrized risk modules and their combinations on alert datasets. It converts module outputs into normalized per-alert risk scores, combines them into pipeline scores, and evaluates prioritization with metrics.

  • CATS supports visual exploration through a web UI and automated evaluation through a batch processor.
  • Each pipeline applies parametrized risk modules to a selected alert dataset, with parameters controlling grouping, temporal windows, alignment, and window shape.
  • Each module assigns every alert a normalized risk score in [0,1], where zero indicates low risk and one indicates high risk.
  • Pipeline operators combine module scores using weighted arithmetic or geometric means.
  • Combined scores feed metric calculations, while batch processing can optimize module weights and parameters for a target metric such as AUROC.

6 Alert Datasets: Existing and Newly Created

The evaluation uses eight alert datasets spanning diverse environments, attacks, and security systems, combining one existing dataset with newly created or extended sources. The datasets contain labeled true and false alerts and include both public and proprietary data.

  • The datasets are designed to include many true and false alerts across diverse environments, attacks, and security systems.
  • Eight alert datasets were assembled because only one publicly available dataset initially met the evaluation requirements.
  • Each alert is stored in JSON Lines format with its log record, triggered rule, parsed features, and a true or false label.
  • The eight datasets derive from five sources and are separated by alert-generating security system because host-based and network-based alerts require different parametrization.
  • AIT-ADS contributes Wazuh and Suricata datasets from a small-company testbed, while SOCBED supplies Sigma and Suricata alerts from a two-hour simulated environment.
  • DEDALE alerts were generated from Windows event logs using Sigma rules after reviewing and correcting labels, while Suricata alerts were omitted because only two of 130,000 were attack-induced.
  • Additional datasets cover APT29S2 emulation, a proprietary ERP-corporation environment, and host-based Falco alerts collected from 44 application servers.

7 Evaluation of Risk Hypotheses

Across eight diverse alert datasets, the evaluation found that several risk modules prioritize alerts above chance, while optimized combinations generalize well to unseen data. Performance depends on module choice and parameterization, and multiple metrics indicate practical prioritization quality.

  • 7.1 Three Modules Prioritize Alerts Well: Variety, Accumulation, and Rule Level prioritized alerts well above chance on almost all datasets, whereas Rarity and Aperiodicity were more mediocre and variable.Variety 1h and 1d exceeded chance on all datasets, followed by Accumulation 1m; Rule Level was near chance on three datasets.
  • 7.1 Three Modules Prioritize Alerts Well: Accumulation performed best with a one-minute window, while Variety was strongest at one hour and Rarity and Aperiodicity improved with longer windows.These patterns are consistent with short attack bursts, multiple attack steps over longer periods, and deviations from long-term benign activity, respectively.
  • 7.2 Combining Modules Works Even Better: AUROC 0.92 mean and 0.09 standard deviation were achieved by leave-one-out pipelines across eight datasets.The module combination performed consistently across diverse environments, alert sources, and attacks.
  • 7.2 Combining Modules Works Even Better: 43 of 44 true alerts entered the highest-prioritized 10% for AIT-ADS Suricata, compared with 28.6 under Rule Level.For DEDALE Sigma, the corresponding counts were 50 of 51 versus 7.7 under Rule Level.
  • 7.2 Combining Modules Works Even Better: Switching from triangle to box windows decreased mean AUROC by 5%, while changing centered to trailing windows caused a 17% drop and greater variability.Using a geometric instead of arithmetic mean had only a marginal effect.
  • 7.3 Digging Deeper with Multiple Metrics: Almost all evaluated metrics exceeded their no-skill levels, with no critical failures across datasets, supporting the optimized pipelines’ fitness for practical application.AUROC and average precision remained high, including on challenging low-base-rate datasets.

8 Discussion and Limitations

The discussion finds that simple risk modules can prioritize alerts effectively, while highlighting operational, adversarial, human-factor, and dataset limitations that constrain interpretation and deployment.

  • Discussion of Key Findings: AUROC 0.5 was exceeded significantly by Rule Level, Accumulation, and Variety, whereas Rarity and Aperiodicity did not exceed chance.The reported p-values were 0.016, 0.012, and 0.0039 for the first three modules, versus 0.63 and 0.42 for the latter two.
  • Discussion of Key Findings: Accumulation and Variety were the strongest performers and the modules most represented in practical RBA literature.
  • Caveats of Our Approach: AUROC summarizes performance across all triage rates, but SOC-specific assessment should use precision and recall at the known triage rate.
  • Caveats of Our Approach: Centered windows improve Accumulation and Variety but delay prioritization, while trailing windows support real-time use at a manageable performance cost.The centered-window delay is half the window length, such as 30 minutes for a one-hour window.
  • Limitations of the Evaluation Datasets: Only one dataset contains real corporate benign activity, while other datasets use simulated users; short-duration datasets also limit evaluation of large windows.The datasets emphasize advanced persistent threat-like attacks and open-source detection systems, reducing diversity in those dimensions.
  • Attacks Against RBA: RBA may be vulnerable to attacks that exploit module assumptions, although multiple rules triggered by attack steps and first periodic occurrences complicate evasion.
  • Human Factors: Analyst trust and initial complexity may limit workload reductions, so user studies are needed to measure RBA’s effect on alert fatigue directly.
  • Use of RBA as a Baseline: RBA has low computational complexity, averaging 413 alerts per second, or 35.7 million per day, on the reported desktop system.The authors argue that parallelization would make it suitable for large enterprise networks and a baseline for costlier methods.

9 Related Work

Related work spans alert-prioritization and false-alert-reduction methods using different data requirements, while this paper emphasizes recent diverse datasets and modular RBA evaluation.

  • Scope: Alert-fatigue research is divided into automation, augmentation, and collaboration, with this paper focusing on alert prioritization and false-alert reduction.
  • Data Requirements: Existing approaches may require telemetry, packet captures, asset information, analyst feedback, or labeled training data, unlike this work’s alert-only input.
  • Prior Methods: Closely related methods include accumulation-like filtering, compromise hypotheses, alert correlation graphs, and dynamically adapted risk thresholds.
  • Evaluation Resources: Many prior evaluations used proprietary or obsolete datasets, whereas this paper evaluates eight recent datasets that were partly created or extended by the authors.

10 Conclusion

The conclusion reports that three of five hypotheses successfully rank true alerts above false alerts, while rarity and periodicity do not significantly exceed chance. A weighted combination performs best overall and supports RBA as a potential way to reduce alert fatigue and as a baseline for more complex methods.

  • Findings: Three hypotheses—spatio-temporal variety, spatio-temporal accumulation, and high rule severity—successfully rank true alerts above false alerts.
  • Findings: Rarity and periodicity of alert types did not significantly exceed chance level.
  • Findings: A weighted combination of all hypotheses achieved higher mean prioritization performance and lower standard deviation than the best single hypothesis.
  • Implications: RBA can substantially reduce false alerts requiring analyst review and appears to generalize well across diverse datasets with relatively robust parametrization.
  • Implications: RBA is recommended as a baseline for more complex, resource-intensive alert-prioritization methods, including large language models.
  • Conclusion: The conclusion states that RBA can mitigate alert fatigue under the assumption that analysts benefit from reviewing substantially fewer false alerts.

Open Science

The authors provide anonymized CATS code, datasets, documentation, and a hosted instance to support reproduction, verification, and extension of the reported results.

  • Repository: An anonymized CATS repository contains code, alert datasets, and documentation for verifying, reproducing, and extending the results.
  • Repository Contents: The repository includes the web UI guidance, Python backend, risk modules, and datasets, though ERPCorp Falco cannot be published because of security and privacy concerns.
  • Hosted Instance: A hosted CATS instance enables alert-prioritization metrics to be reproduced within minutes without local installation.
  • Hosted Instance: The hosted instance is read-only, may fail behind some web proxies, and does not support persistent server-side configuration changes.

Ethical Considerations

The paper documents dataset handling and the CATS tooling used to inspect, configure, and evaluate alert-prioritization pipelines. One enterprise dataset was approved for use, while other datasets contained no personal information to the authors’ knowledge.

  • ERPCorp Falco was recorded in a productive enterprise network with real users, and its collection and use were approved by the corporation’s head of information security.
  • CATS uses human-readable JSON Lines datasets with one alert per line, separating metadata, parsed features, and the original full alert.
  • CATS provides configurable datasets, risk modules, and pipelines, with visual statistics, evaluation curves, plots, and JSON export for further inspection or processing.
  • The example alert format preserves a misuse label indicating whether an alert is true or false, alongside source, identifier, and parsed rule information.

C Discussion of Notable Risk Module Results

The discussion shows that Rarity and Aperiodicity can fail when alert distributions violate their assumptions, while alternative pipelines produce similar overall results. Dataset composition and temporal windows therefore materially affect module behavior and configuration choices.

  • Individual module results: Near-zero AUROC for Rarity and Aperiodicity on AIT-ADS Wazuh and APT29S2 Sigma indicates reversed prioritization.These were the only datasets whose optimized pipelines excluded both modules.
  • Individual module results: In AIT-ADS Wazuh, false authentication-success alerts dominated the data, while true web-server errors formed short attack bursts that Aperiodicity treated as periodic.This distribution made benign alerts appear both rarer and more aperiodic than true alerts.
  • Module limitations: The Rarity and Aperiodicity modules assume that true alerts are rare, but one attack can generate thousands of alerts.The authors recommend combining these modules with others or omitting them rather than relying on them alone.
  • Module limitations: ERPCorp Falco’s Rarity AUROC increased from 0.06 at 1m to 0.69 at 1h, suggesting that longer windows better capture deviations from long-term normal activity.
  • Module limitations: APT29S2 Suricata contains only two alert types with both true and false instances, rendering the Rarity module ineffective and cautioning against single-dataset decisions.
  • Alternative pipelines: 0.93 mean AUROC without Rarity and Aperiodicity versus 0.92 with both modules, but AP falls from 0.90 to 0.88, leaving the better variant inconclusive.The authors advise practitioners to test both variants on their own alert data.
  • Dataset sensitivity: Excluding the short SOCBED- and APT29S2-based datasets barely changes results on the remaining datasets, indicating that their inclusion did not inflate the original evaluation.
Loading 2609.02465v1…