PaperlayerView exact revision on arXiv

Source-linked AI summary

REFINE: A Resilient Evolution Framework for Intelligent Enterprise Alert Triage in Security Operations Centers

Huimin Chen, Quan Long, Yanhao Wang

arXiv:2609.32516v1cs.CRcs.AIcs.LG

TL;DR

SOCs need to reduce benign-alert review while avoiding missed threats, but LLM agents can struggle to remain aligned with changing, organization-specific triage standards. REFINE evolves structured analyst skills using alert dispositions while enforcing recall = 1.0 as a hard constraint. It achieves recall = 1.0 on all four evolution sets and retains it in three of four later test windows.

  • Problem

    LLM agents and static triage policies do not reliably track organization-specific operational standards that evolve with business context and the threat landscape.

    Source: Introduction · p. 1

  • Method

    REFINE encodes analyst expertise as structured skills, uses alert distributions and error boundaries to locate blind spots, and evolves skills under a hard recall constraint.

    Source: Introduction · pp. 1–2

  • Results

    Recall = 1.0 on all four evolution sets; on later test windows REFINE retains recall = 1.0 in three scenarios and reaches 0.807 in the remaining scenario, versus 0.49–0.58 for other self-evolution baselines.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Across four real industrial SOC scenarios, REFINE maintains perfect evolution-set recall while improving benign-alert closure, with later-window results providing evidence of cross-time transfer.

    Source: Conclusion · p. 7

    In the NTLM-Hash scenario, accuracy drops from 0.907 on the evolution set to 0.537 on the test set, with errors concentrated on benign alerts from training-unseen processes.

    Source: Drop · p. 12

Abstract

from arXiv · show

Security Operations Centers (SOCs) process large volumes of alerts daily. Alert triage prioritizes high-risk threats while reducing manual review of benign alerts. LLM agents can reason over logs and threat intelligence, but struggle to keep aligned with organization-specific, rapidly evolving SOC operational standards. We introduce REFINE, an LLM-agent framework for enterprise alert triage. REFINE encodes analyst expertise as structured skills and continuously adapts using analyst disposition feedback. It enforces recall = 1.0 as a hard constraint during evolution to maximize auto-closure of false positives, and identifies judgment blind spots by combining alert distributions with model error boundaries. Evaluated on four real industrial SOC scenarios across four MITRE ATT&CK phases with temporal split: REFINE achieves recall=1.0 on all evolution sets. On future test windows, it retains recall=1.0 in three scenarios; the degraded case reaches 0.807 recall, still outperforming self-evolution baselines (0.49-0.58).

Abstract

SOCs need to reduce benign-alert review without missing threats, but LLM agents and static policies struggle to track organization-specific triage standards that evolve over time. REFINE encodes analyst expertise in skills and adapts them from alert dispositions; across four industrial scenarios, it preserves perfect recall on evolution sets and mostly does so on later test windows.

SOC Alert Triage

  • SOCs need to close benign false positives while avoiding missed threats, because a missed threat may leave an intrusion unhandled whereas an unclosed false positive remains in the human queue.Alert volumes exceed human triage capacity, and alert fatigue weakens threat detection.Source: Introduction · p. 1

Self-Evolution Methods

  • Existing self-evolution methods optimize general objectives, while REFINE targets continuous alignment with organizational judgment and preservation of a no-missed-recall baseline.Conventional SOC classifiers and ranking methods also have difficulty updating organization-specific triage specifications from new disposition feedback.Source: Introduction · p. 1

Introduction

  • REFINE combines structured skills encoding analyst judgment rules, evidence reasoning, and tool protocols with disposition feedback, blind-spot localization, and a hard recall constraint during evolution.Blind spots are localized by combining the global alert distribution with the current skill’s error boundary.Source: Introduction · pp. 1–2
  • Across four industrial SOC scenarios, REFINE reaches recall = 1.0 on every evolution set and retains it in three later test windows; its remaining test recall of 0.807 exceeds self-evolution baselines at 0.49–0.58.The NTLM-Hash evolution-set accuracy rises from 0.217 to 0.907 at recall = 1.0.Source: Introduction · p. 2

Method

REFINE learns a triage skill to maximize accuracy while enforcing perfect recall, then evolves and deploys it with mechanisms for global initialization, constrained reflection, and ongoing realignment. Its search preserves recall-feasible outputs and falls back to human triage when none is found.

Problem Formulation

  • The skill-learning objective is to maximize accuracy subject to a hard recall constraint, reflecting the asymmetric costs of missed threats and unclosed false positives.Missed threats can leave an ongoing intrusion undetected; unclosed false positives remain in the human queue.Source: Problem Formulation · p. 2

REFINE Framework

  • REFINE globally analyzes the full evolution set to create an initial skill, evolves it under recall constraints, and uses sampled safety checks or online accuracy monitoring to trigger rolling post-deployment evolution.Figure 1 depicts the three-stage process, including the recall-safety path.Source: REFINE Framework · p. 2, Figure 1 · p. 3

Global Initial Skill Generation

  • A single global analysis of the complete evolution set generates an initial skill with full-distribution coverage, avoiding a manual seed and letting later evolution repair known defects rather than fill information gaps.The initialization is a one-shot summary of distribution statistics and boundary samples, not per-sample triage.Source: Global Initial Skill Generation · p. 3

Intelligent Sampling

  • Intelligent sampling prioritizes missed recalls, then false positives, while batching related errors for reflection; global generation and candidate evaluation remain anchored to the full evolution set.The priorities are P1 missed recalls, P2 false positives, and P3 correct samples.Source: Intelligent Sampling · p. 3

Evolution Loop

  • Multi-round reflection extends a candidate lineage, accepting a revision only when batch recall does not fall and at least one metric improves; batch and global gates select the highest-accuracy feasible skill.The hard recall constraint is kept separate from scalar objectives, and a failed global gate routes alerts to human triage instead of deploying an infeasible skill.Source: Evolution Loop · pp. 3–4

Continuous Alignment

  • Re-evolution rebuilds the evolution set from the latest production window rather than accumulating historical boundary samples, with hard safety checks or significant online-accuracy drops triggering realignment.Accumulated historical samples could anchor the skill to an outdated operational scale; the timeliness trigger can act before a missed recall is confirmed.Source: Continuous Alignment · p. 4

Experiments

REFINE is evaluated on temporally split, real enterprise SOC alerts, with recall = 1.0 enforced on evolution data and later-window recall used to assess cross-time behavior. Across the reported comparisons and ablations, it achieves recall-safe outputs more consistently than the baselines, while component tests show that the full design is needed for reliable feasible outputs.

Datasets / Evaluation Metrics / Baselines / Implementation Details

  • The evaluation uses four real-world SOC alert scenarios with evolution alerts from an earlier window and test alerts from a strictly later, non-overlapping window; recall = 1.0 is the hard constraint and primary safety metric.The evolution and test sets contain 100 and 150 alerts, respectively. GEPA, ACE, and REFINE share the same LLM and 1,000-inference budget; self-evolution methods are averaged over three runs.Source: Datasets · p. 4, Evaluation Metrics · p. 4, Baselines · p. 5

Evolution Effectiveness

  • Only REFINE reaches recall = 1.0 on all four evolution scenarios; on later windows it retains recall = 1.0 in three, while Host-Process recall is 0.807, highest among methods there.NTLM-Hash evolution accuracy rises from 0.217 (LLM Direct) to 0.907 for REFINE at recall = 1.0.Source: Evolution Effectiveness · p. 5, Table 2 · p. 6

Comparison within Feasible Region

  • ACE’s feasible-pool candidates still fail to match REFINE’s later-window safety in Phishing, Host-Process, and NTLM-Hash; the methods tie only on PROC2_HEYE.ACE has no feasible Phishing candidate; its Host-Process and NTLM-Hash feasible candidates reach test recall 0.575 and 0.921, versus REFINE’s 0.807 and 1.000.Source: Comparison within Feasible Region · p. 5, Table 3 · p. 6

Ablation Study

  • On Host-Process, removing the initial Skill yields recall 0.933 and no feasible output, while removing intelligent sampling also yields recall 0.933 and zero feasibility; removing both lowers recall to 0.817.The initial Skill supplies a full-distribution prior; intelligent sampling directs reflection toward missed-recall and false-positive boundaries while that prior anchors local repairs.Source: Ablation Study · p. 6
  • Removing multi-round reflection and the recall gate together lowers Host-Process recall to 0.975 and produces feasible outputs in only one of three runs; removing all mechanisms lowers recall to 0.808 and feasibility to zero.The joint ablation does not separately quantify the reflection and gate effects; across ablations, accuracy reaches 0.807 while recall remains below 1.0.Source: Ablation Study · pp. 6–7, Table 4 · p. 7

Model Robustness

  • With Kimi-K2.6, REFINE reaches recall = 1.0 on all four evolution sets in all three runs, while LLM Direct misses recall in three scenarios.On NTLM-Hash, REFINE accuracy rises from 0.220 to 0.983 at recall = 1.0; the robustness result is limited to evolution-set behavior.Source: Table 5 · p. 7, Model Robustness · p. 7

Effect of Skill and Context Size

  • ACE’s final playbooks contain 18K–58K tokens, compared with 4.1K–7.6K tokens for REFINE’s Skills, a 2.9×–12.3× size gap.The paper connects ACE’s accumulation of rules with evolution-distribution fit that fails to transfer, while noting the deployment overhead and audit difficulty of feeding the playbook with every alert.Source: Table 6 · p. 7, Effect of Skill and Context Size · p. 7

Conclusion

REFINE evolves structured Skills to align LLM-agent triage with enterprise operational scales, localizes judgment blind spots, and constrains evolution with a recall gate. In four real industrial SOC scenarios, it maintained recall = 1.0 on every evolution set while improving benign-alert closure, with later-window results providing evidence of cross-time transfer.

Conclusion

  • Across four real industrial SOC scenarios, REFINE maintained recall = 1.0 on every evolution set while improving benign-alert closure.Results on later time windows further provide evidence of cross-time transfer.Source: Conclusion · p. 7
  • REFINE localizes judgment blind spots by combining the global alert distribution with the current error boundary.It constrains evolution through a recall gate.Source: Conclusion · p. 7
  • REFINE continuously evolves structured Skills to align LLM-agent alert triage with enterprise operational scales.Source: Conclusion · p. 7

Key Notation

Table 7 is identified as a key-notation table, but the supplied passage does not include its notation entries.

Key Notation

  • Table 7 is identified as the paper’s key-notation table.Source: Table 7 · p. 10

Threat Model and Assumptions

REFINE is framed for enterprise-internal SOC triage under operational, data, and tool assumptions. It learns against organization-specific analyst dispositions and does not model adversarially manipulated alerts or tool outputs.

Threat Model and Assumptions

Deployment Scope

  • REFINE targets enterprise-internal SOC triage, with evolution and test sets drawn from the same alert type in non-overlapping time windows.Source: Deployment Scope · p. 10

Label Source and Quality

  • Analyst escalate/dismiss labels reflect organization-specific operational judgment, not universal security standards, and may legitimately differ across analysts.REFINE evolves against the operational labels available in each evolution window without requiring globally noise-free labels.Source: Label Source and Quality · p. 10

Adversarial Assumptions

  • The threat model assumes genuine production traffic and no adversarial manipulation of the evolution set or labels.Crafted alerts exploiting deployed Skill blind spots are outside scope.Source: Adversarial Assumptions · p. 10

Tool Availability

  • The agent may use a fixed set of read-only security tools, whose outputs are assumed accurate and available.Tool failures and adversarial tool responses are not modeled.Source: Tool Availability · p. 10

Batch Training Procedure

REFINE trains recall-safe Skills through batch reflection, global evolution, and a recall gate; after deployment, safety and timeliness signals trigger re-evolution. If evolution cannot produce a recall-feasible Skill, the algorithm returns null for human-triage fallback.

Batch Training Procedure

  • Within a batch, reflection continues only when recall does not decrease and at least one of recall or accuracy improves; the batch returns its most accurate candidate with recall = 1.0, or null.Source: Batch Training Procedure · p. 10

Deployment Re-Evolution Triggers

Trigger Path 1: Safety Trigger (Hard)

Trigger Path 2: Timeliness Trigger (Soft)

  • A statistically significant online accuracy or takeover-rate drop can trigger re-evolution before any missed recall is confirmed.The stated example threshold is a drop exceeding two standard deviations from the rolling baseline.Source: Trigger Path 2: Timeliness Trigger… · pp. 10–11

The REFINE Algorithmic Procedure

  • The algorithm globally generates an initial Skill, samples alerts by priority, trains batches, evaluates candidates, updates an elite pool, and selects the most accurate recall-feasible Skill.If none is feasible within budget, it returns null for human-triage fallback.Source: The REFINE Algorithmic Procedure · p. 11

Dataset Statistics and Split Rationale

The evaluation uses disjoint earlier evolution and later test windows, with fixed sample sizes and scenario-specific threat availability. Four alert scenarios cover distinct evidence patterns, while recall is enforced on the evolution set because only that window is available during optimization.

Temporal Split Protocol

  • Devo samples 100 earlier-window alerts and Dtest samples 150 strictly later alerts, preventing temporal leakage while keeping evaluation budgets consistent.Alerts are timestamp-ordered and split at a fixed point; the temporal window and sample size are independent choices.Source: Temporal Split Protocol · p. 11
  • Devo alone receives representative aggregation and within-window threat augmentation, capped at 50% of its size and using no test-window alerts.The resulting positive rates vary with available threats: Phishing reaches 50%, while Host-Process is 40%, PROC2_HEYE 9%, and NTLM-Hash 8%.Source: Temporal Split Protocol · p. 11
  • Recall = 1.0 is enforced on Devo because evolution uses only the earlier window; Dtest checks whether the evolved Skill holds on future alerts.Later-window threat counts vary from 5 to 66 and reflect actual threats rather than a fixed target ratio.Source: Temporal Split Protocol · p. 11

Phishing

  • Phishing alerts rely chiefly on sender identity, message wording, and link or credential-request cues, making this the most text-like and balanced scenario.Source: Phishing · p. 12

Host-Process

  • Host-Process alerts hinge on process ancestry, command lines, parent-child relationships, and execution context, testing behavior-chain reasoning.Source: Host-Process · p. 12

NTLM-Hash

  • NTLM-Hash covers suspicious access to LSASS or related credential material and is the paper’s most brittle scenario across time.Source: NTLM-Hash · p. 12

PROC2_HEYE

  • PROC2_HEYE tests recall under the set’s strongest class imbalance, with sparse positive evidence amid predominantly benign traffic.Source: PROC2_HEYE · p. 12

NTLM-Hash Evolution Trajectory

The NTLM-Hash trajectory shows why global evaluation and a separate recall gate matter for text-space Skill evolution. Discrete recall jumps undermine penalty scalarization and constraint-dominance ranking, motivating retention of near-feasible strategies while filtering output for feasibility.

NTLM-Hash Evolution Trajectory

  • Node 7 becomes Sbest with accuracy 0.91 and recall = 1.0 after a trajectory of leap, regression, and convergence; node 5’s higher accuracy 0.94 is rejected at recall 0.875.Nodes 2–3 reach batch accuracy 1.0 on five samples but fall to 0.21–0.22 on full evolution-set evaluation; global evaluation returns the search to S0 before nodes 4–7 converge.Source: NTLM-Hash Evolution Trajectory · p. 12

Drop

  • NTLM-Hash accuracy falls from 0.907 on the evolution set to 0.537 on the test set, with 95.2% of test false positives concentrated among training-unseen processes.The windows share the dominant alert rule; 87.7% of test-window benign alerts come from unseen processes, and correcting their false positives would raise test accuracy to 0.980.Source: Drop · p. 12

Necessity of Decoupling Scoring, Pareto Retention, and the Hard-Constraint Gate

  • The appendix frames batch/global search as a response to the failure of standard ways to collapse constrained multi-objective Skill optimization into one solvable problem.Source: Necessity of Decoupling Scoring,… · p. 13

Discrete-Jump Observation

  • Text edits can cause discrete recall jumps, such as NTLM-Hash dropping from 1.0 to 0.75 in one step, although this is not claimed for all Skill classes.Source: Discrete-Jump Observation · p. 13

Failure Mode 1: Penalty Scalarization

  • Penalty scalarization has no workable λ under discrete recall jumps: small values allow recall trade-offs, while large values collapse to escalating everything.At a 0.25 recall jump, the threshold is a single point rather than an interval.Source: Failure Mode 1: Penalty Scalarization · p. 13

Failure Mode 2: Constraint-Dominance Ranking

  • Constraint-dominance ranking treats all recall-below-1.0 strategies alike, discarding the useful distinction between near-feasible recall 0.875 and 0.75.With discrete jumps, the 0.875 strategy may be one rule from feasibility, whereas the next edit can return recall to 1.0 or drop it to 0.75.Source: Failure Mode 2: Constraint-Dominance… · p. 13

Derivation of the Decoupled Architecture

  • The decoupled design retains feasible and nearly feasible strategies in a Pareto elite pool, then uses an independent recall gate to filter feasible output candidates.This preserves near-feasible evolutionary information while constraining output to deployment-feasible candidates.Source: Derivation of the Decoupled Architecture · p. 13

Global Generation Prompt Template

The initial-generation prompt analyzes the evolution set and produces a deployment-compatible Skill. It specifies the analysis sequence, the structure and rigor expected of the Skill, and the output format.

Global Generation Prompt Template

  • The prompt takes the evolution set, inference template, and available read-only tools as inputs, then produces an initial Skill S0.Its three one-shot steps are sample-feature analysis, detection-intent understanding, and Skill design.Source: Global Generation Prompt Template · p. 14
  • The generated Skill remains compatible with the deployment inference template, and the prompt returns separate thought-process and Skill-content blocks.Source: Global Generation Prompt Template · p. 14
  • The Skill must provide prioritized rules and stepwise triage, distinguish risk evidence from context and exclusion evidence, and bound whitelist-style experience with applicability conditions.Tool use is optional and should not hardcode fixed allow/deny lists or assume external attributes from names alone.Source: Global Generation Prompt Template · p. 14
  • The recommended Skill structure includes core principles, triage strategy, empirical lessons, conflict handling, and edge cases.Source: Global Generation Prompt Template · p. 14

Multi-Round Reflection Prompt Template

The reflection prompt revises a current Skill using batch execution trajectories and feedback. It analyzes errors and correct decisions, compares the Skill against those trajectories, and requires executable, bounded rules in the revised Skill.

Multi-Round Reflection Prompt Template

  • The reflection prompt takes the current Skill, batch trajectories, feedback, inference template, and available read-only tools as inputs for revision.It is intended to improve triage accuracy and consistency while remaining compatible with inference-time invocation.Source: Multi-Round Reflection Prompt Template · p. 15
  • Reflection must explain differing human dispositions by identifying discriminating dimensions, rather than relying on feature similarity or simply lowering thresholds.Source: Multi-Round Reflection Prompt Template · p. 15
  • The analysis identifies error patterns, attributes correct decisions, and compares trajectories with the current Skill to locate gaps.Source: Multi-Round Reflection Prompt Template · p. 15
  • Revisions should encode explicit error handling and reusable correct reasoning while distinguishing risk behavior, context, exclusion evidence, uncertainty, and metadata.Whitelist-style experience must state applicability boundaries; tools remain optional, and fixed trusted lists or inferred external attributes are prohibited.Source: Multi-Round Reflection Prompt Template · pp. 15–16
  • The recommended revised-Skill structure includes core principles, triage strategy, empirical lessons, conflict handling, and edge cases.Source: Multi-Round Reflection Prompt Template · p. 16

Supporting Mechanism Details

The appendix formalizes REFINE’s scoring weights and Pareto elite pool; rationale for roulette parent selection and the per-parent cap κ is covered in the main text.

Scoring Weights

  • With FN = 0 guaranteed at output, feasible strategies are ranked by wTN · |TN| + wTP · |TP| + wFP · |FP|; any weights satisfying wTN > wTP > wFP ≥ 0 yield the same ranking.The specified outcome weights are TP = +1, TN = +3, FN = −1, and FP = 0.Source: Scoring Weights · p. 17
  • Scoring sets parent-selection probabilities and judges the success constraint, while Pareto dominance—not scoring directly—determines which Skills enter the elite pool.The gate’s scoring objective maximizes alert takeover among feasible strategies.Source: Scoring Weights · p. 17

Formal Pareto Elite Pool Definition

  • The elite pool unions candidates that are best on at least one evolution-set sample, then prunes candidates whose champion sets are strict subsets of others.This retains a strategy uniquely best on even one sample, preserving intermediate diversity that coarser accuracy-and-recall dominance would discard.Source: Formal Pareto Elite Pool Definition · p. 17

Related papers

Loading 2609.32516v1…