PaperlayerView exact revision on arXiv

Source-linked AI summary

From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage

Saimon Amanuel Tsegai, Alex Kantchelian, Danfeng, Yao, Peng Gao

arXiv:2610.10608v1cs.CRcs.AIcs.MA

TL;DR

SOC triage must handle many benign alerts without letting true attacks go uninvestigated, but it remains unclear how LLM reasoning strategies should gather evidence and determine when to close an alert. The paper evaluates five approaches with ALERT-BENCH, analyzes their investigation failures, and designs AIDA; AIDA outperforms the studied approaches on the same alerts.

  • Problem

    It remains unclear how LLM agents should organize evidence gathering and decide when an alert investigation is sufficient, despite the need to limit missed attacks and unnecessary escalations.

    Source: §I · p. 1

  • Method

    The paper evaluates five tool-using LLM reasoning approaches on ALERT-BENCH and designs AIDA, which requires a proposed decision before independent challenge and separate adjudication.

    Source: §I · pp. 1–2

  • Results

    AIDA achieves an F1 score of 0.958, compared with 0.371–0.744 for the studied approaches, and reduces the false-negative rate from 40.4% to 3.1%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The study finds that structuring evidence retrieval and decision review can substantially improve agentic SOC triage.

    Source: §VIII · p. 13

    Evaluation uses ALERT-BENCH tasks derived from one dataset family; independently collected SOC telemetry would further test generalization across organizational baselines, detection rules, and logging environments.

    Source: §VII · p. 13

Abstract

from arXiv · show

Security operations centers (SOCs) must triage large volumes of alerts, most of which are benign, while missed attacks can remain uninvestigated. Tool-using large language model (LLM) agents can retrieve evidence during triage, but it remains unclear how reasoning strategies determine what to gather and when an investigation is sufficient to close an alert. We study five representative approaches spanning single-pass tool use, iterative retrieval, sampled investigations, self-review, and explicit verification. To support this study, we build ALERT-BENCH, an interactive benchmark that replays enterprise telemetry through a live SIEM and requires each system to retrieve evidence. Across 1,247 alerts from a multi-stage attack scenario, every approach missed at least 40.4% of attack-related alerts. Trace analysis shows that attack alerts are more likely to be dismissed when searches return no records, same-context review has negative net correction, and dismissal receives no consistently stronger investigation than escalation. Based on these findings, we further design AIDA (Adversarial Investigation and Dialectical Analysis), a multi-agent framework that requires an explicit proposed decision before independent challenge and stronger evidentiary requirements before dismissal. AIDA preserves investigation history in an append-only Investigation Ledger and keeps the challenge in a separate reasoning context. A separate Judge adjudicates the proposed decision and challenge against evidence, resolving the alert or requesting another round when evidence is missing. On the same alerts, AIDA achieves an F1 score of 0.958, compared with 0.371-0.744 for the studied approaches, and reduces the false-negative rate from 40.4% to 3.1% while escalating 18.4% of alerts to analysts. These results show that structuring evidence retrieval and decision review can substantially improve agentic SOC triage.

I. INTRODUCTION

The paper studies how LLM reasoning strategies gather evidence and decide whether SOC alerts can be dismissed or should be escalated. It introduces ALERT-BENCH to evaluate interactive investigations, analyzes recurring failure patterns, and uses them to motivate AIDA.

I. INTRODUCTION

  • SOCs process large alert volumes dominated by benign activity, and dismissing a true attack can leave it uninvestigated.The paper notes that analysts process tens of thousands of alerts per day, with only a negligible fraction corresponding to confirmed compromises.Source: §I · p. 1

A. Alert Triage and Its Automation

  • SOC triage must limit both missed attacks and unnecessary escalations, while prior automated pipelines do not decide what additional evidence to gather when existing evidence is insufficient.Prior systems use fixed analysis procedures, whereas tool-using LLM agents can support iterative investigation.Source: §I · p. 1, A. Alert Triage and Its Automation · p. 3

I. INTRODUCTION

  • The study compares five approaches—Direct-Prompt, ReAct, Self-Consistency, Self-Refine, and Chain-of-Verification—spanning single-pass, iterative, sampled, self-review, and verification strategies.All are evaluated with identical alerts, model choice, evidence access, and tool capabilities.Source: §I · pp. 1, 3
  • Attack alerts are more likely to be dismissed when searches return no records; same-context review can introduce errors, and dismissal receives no consistently stronger investigation than escalation.Self-Refine changes 79 correct decisions into errors, while CoVe changes 132; four approaches retrieve fewer records before dismissal.Source: §I · p. 2
  • AIDA requires an explicit proposed decision before independent challenge, preserves evidence in an append-only Investigation Ledger, and gives a separate Judge authority to request more evidence.The Judge requires stronger evidence before dismissal because closing an alert ends its triage path.Source: §I · p. 2
  • AIDA achieves an F1 score of 0.958, compared with 0.371–0.744 for the studied approaches, and reduces the false-negative rate from 40.4% to 3.1%.AIDA escalates 18.4% of alerts to analysts, compared with 8.5–14.0% for the studied approaches.Source: §I · p. 2
  • ALERT-BENCH starts each system from a detector-generated alert without selected evidence, requiring it to retrieve supporting context before deciding whether to dismiss or escalate.The benchmark replays enterprise telemetry through a live SIEM and keeps ground-truth labels hidden during investigation.Source: §I · p. 2

B. LLM Agents in Security Operations

The paper positions LLM agents as tools for iterative security investigation, then presents ALERT-BENCH as an interactive alert-triage benchmark. The benchmark replays enterprise telemetry and evaluates systems on a reproducible sample of alerts.

B. LLM Agents in Security Operations

  • LLM systems can retrieve and analyze evidence during inference, while multi-agent systems often use shared context that may propagate earlier errors or encourage conformity to prior responses.The paper situates ALERT-BENCH among security benchmarks but emphasizes its focus on defensive alert-triage decisions over enterprise telemetry.Source: B. LLM Agents in Security Operations · p. 3

C. Evaluation and Benchmarking

  • ALERT-BENCH evaluates whether detector-generated alerts should be dismissed or escalated, requiring evidence retrieval from surrounding telemetry through a SIEM rather than a supplied PCAP artifact or investigation question.This distinguishes its task from SIABench and ExCyTIn-Bench.Source: C. Evaluation and Benchmarking · p. 3

III. ALERT-BENCH

  • ALERT-BENCH turns AIT-LDSv2 telemetry and attack annotations, aligned using the AIT-ADS procedure, into an interactive task that starts with an alert and no preselected evidence bundle.Labels remain hidden during investigation and are used for scoring.Source: §III · pp. 3–4

A. Environment Construction and Ground Truth

  • The replay preserves timestamps and host identities so surrounding activity remains searchable across hosts and time, while event-based procedures align Wazuh and Suricata alerts with ground truth.The benchmark replays AIT-LDSv2 logs into Wazuh and ingests Suricata alerts with the host telemetry.Source: A. Environment Construction and Ground… · p. 4

B. Evaluation Corpus

  • The comparison uses 1,247 deterministically sampled alerts from the benign-dominated russellmitchell scenario, selected for tractable repeated evaluations rather than based on method results.The sample contains 225 attack-related alerts and 1,022 benign alerts; attacks concentrate in directory enumeration and WordPress vulnerability scanning.Source: B. Evaluation Corpus · p. 4

C. Interactive Task and Harness

Each system investigates one alert through a fixed SIEM interface, choosing what surrounding telemetry to retrieve and when the evidence is sufficient. The harness records decisions and traces, and scoring covers triage quality, operational load, and computational effort.

C. Interactive Task and Harness

  • Each system receives one alert and independently chooses what to retrieve through a fixed MCP interface, with four investigation operations and a host-inventory lookup.Outputs are restricted to the replayed corpus, and no operation exposes ground-truth labels.Source: C. Interactive Task and Harness · p. 5
  • Investigations end with an escalate-or-dismiss decision and justification, which the harness scores against hidden ground truth after the investigation.The harness records ordered transcripts, tool calls, queries, results, and intermediate or final decisions.Source: C. Interactive Task and Harness · p. 5

D. Scoring

  • Classification metrics include precision, recall, F1, accuracy, and confusion-matrix counts; recall measures attack alerts escalated and precision measures escalations that are attack-related.Source: D. Scoring · p. 5
  • Operational metrics report false-negative rate as the fraction of attack alerts dismissed and Investigation Load as the fraction of alerts sent to an analyst.The measures must be interpreted together: dismissing every alert yields zero Investigation Load while missing every attack.Source: D. Scoring · p. 5
  • The harness records token use, tool-call count, and end-to-end latency separately from triage quality to compare computational effort with decision performance.Source: D. Scoring · p. 5

IV. SYSTEMATIC STUDY

The study evaluates five interactive LLM triage strategies on matched alerts, then analyzes their investigation traces. None achieves high attack coverage, and additional computation does not improve triage quality.

IV. SYSTEMATIC STUDY

  • The study asks how reliably representative reasoning strategies triage SOC alerts through interactive investigation and how they gather and use evidence as investigations progress.The five strategies span single-pass reasoning, iterative tool use, sampled investigations, self-critique, and explicit self-verification.Source: A. Study Design · p. 5

A. Study Design

  • The study treats each approach as a complete strategy, analyzing recurring trace behaviors rather than attributing aggregate differences to a single component.Its scope is strategies that operate directly over the benchmark interface without task-specific training; provenance graphs and deployment-labeled training data are outside this setting.Source: A. Study Design · pp. 5–6
  • All five strategies run on the same 1,247 alerts, model, and investigation interface, supporting matched aggregate and per-alert comparisons.The study reports 6,235 investigations, one run of each approach on every alert.Source: A. Study Design · pp. 5–6

B. Aggregate Results

  • F1 scores range from 0.371 to 0.744, and every approach dismisses at least 40.4% of attack-related alerts.Direct-Prompt has the highest F1 at 0.744 with 0.596 recall; its per-alert correctness differs significantly from each other approach.Source: B. Aggregate Results · p. 6
  • Greater computational cost does not correspond to better triage: Direct-Prompt uses 5,075 tokens per alert, while CoVe uses 33,438 and has the lowest F1, 0.371.The results do not establish that additional computation is harmful by itself; sampling or self-review alone is not sufficient to improve triage.Source: B. Aggregate Results · p. 6

C. Trace Analysis

Trace analysis links empty retrievals and same-context review to missed attacks, and finds no consistent evidence that dismissal receives a more complete investigation than escalation.

Trace analysis method

  • Two independent parsers agree on retrieval states for all 6,235 traces, and parsed decisions reproduce every approach’s aggregate confusion matrix exactly.Retrieval counts cover investigation before review or verification; computational-cost measurements include all tool calls in the complete approach.Source: C. Trace Analysis · p. 6

Empty retrieval and attack dismissal

  • With no records retrieved, ReAct dismisses 98.6% of attacks, Self-Refine 96.2%, and Direct-Prompt 80.0%; across five approaches, 211 of 602 false-negative investigations end without records.For benign alerts, retrieval changes dismissal rates by at most 2.8 percentage points. An empty query can provide negative evidence about its searched predicate, but none of the five approaches checks whether enough relevant context was searched.Source: C. Trace Analysis · p. 6

Effect of continued retrieval

  • Across 1,125 attack investigations, dismissal rises with empty-result count (rank correlation 0.383, p<10−40), while escalation falls as empty results accumulate.ReAct escalation falls from 86.5% with no empty query to 11.9% after two empty results; Self-Refine falls from 84.2% to 49.1% over the same range.Source: C. Trace Analysis · pp. 6–7, Figure 4 · p. 7
  • With at least one successful retrieval, Direct-Prompt and ReAct have similar attack escalation rates, but ReAct reaches the no-record state more often.Direct-Prompt escalates 72.4% and ReAct 69.7% of attacks with records; on 120 shared cases, they escalate 85 and 79, respectively.Source: C. Trace Analysis · p. 7

Same-context self-review

  • Same-context review breaks more correct decisions than it repairs: Self-Refine breaks 79 and repairs 16, while CoVe breaks 132 and repairs 25.Review lowers Self-Refine F1 from 0.665 to 0.550 and CoVe F1 from 0.669 to 0.371; 74 of Self-Refine’s 95 reversals occur without a new record.Source: C. Trace Analysis · p. 7

Investigation effort before dismissal

  • Four approaches obtain fewer successful retrievals before dismissal than escalation, and none explicitly requires a more complete investigation before dismissal.For example, ReAct obtains 0.92 non-empty results before dismissal versus 1.24 before escalation; only Direct-Prompt shows no difference.Source: C. Trace Analysis · p. 8

D. Error Overlap and Interpretation

Attack misses vary across approaches, while AIDA’s design responds to recurring weaknesses with explicit commitment, independent challenge, separate adjudication, and asymmetric evidentiary requirements.

D. Error Overlap and Interpretation

  • Only 26 of 225 attack alerts are missed by all five approaches, while at least one approach escalates 199; 20 of the 26 shared misses are WordPress scanning alerts.The best individual recall is 0.596, compared with combined coverage recall of 0.884; three attack alerts have otherwise identical benign-labeled log lines after timestamp and process-identifier masking.Source: D. Error Overlap and Interpretation · p. 8

A. Design Rationale

  • AIDA requires the Strategist to commit a proposed classification supported by cited evidence before a separate challenge evaluates it.If the per-round tool budget expires first, a forced commitment produces a thesis from evidence already collected.Source: A. Design Rationale · p. 8
  • The Adversary sees the committed thesis and shared evidence without the Strategist’s private reasoning, and the Judge evaluates the thesis and challenge against evidence.The Judge makes the final decision or requests another investigation round.Source: A. Design Rationale · p. 8
  • AIDA requires concrete evidence for escalation, but affirmative benign evidence or a sufficiently complete investigation for dismissal.When a challenge identifies missing evidence that could change the classification, the Judge requests another round.Source: A. Design Rationale · p. 8
  • These three design properties together define the Dialectical Commitment Protocol.The framework uses explicit commitment, independent challenge, and separate adjudication to structure investigation and decision review.Source: A. Design Rationale · p. 8

B. Protocol Overview

AIDA coordinates investigation, challenge, and adjudication through a persistent ledger, with role-specific evidence views and a Strategist that must commit a thesis each round.

B. Protocol Overview

  • The append-only Investigation Ledger records evidence, theses, challenges, continuation directives, and decisions in sequence without overwriting earlier entries.Later rounds can revise a proposed classification while preserving the evidence and decisions that led to revision.Source: B. Protocol Overview · p. 9
  • Each agent sees ledger content needed for its role; the Adversary receives shared evidence but not the Strategist’s private reasoning.The Judge receives the thesis, challenge, and evidence for adjudication, while the Executor performs the Strategist’s tool request and records the observation.Source: B. Protocol Overview · p. 9
  • Each round investigates and records evidence, challenges a committed thesis, then has the Judge decide or issue a continuation directive for another round.A continuation refreshes the Strategist’s tool budget while preserving the existing ledger.Source: B. Protocol Overview · p. 9

1) Strategist:

  • The Strategist emits either a tool request or a thesis that proposes a True Positive or False Positive classification, cites ledger evidence, and reports confidence in [0, 1].It may retrieve within per-round budget B; if the budget expires first, its next turn must produce a thesis from recorded evidence.Source: 1) Strategist · p. 9
  • An empty result is recorded as an unresolved gap when it does not establish a benign explanation, and a continuation directive identifies evidence to pursue next.Earlier evidence is provided in compact summaries to bound context growth.Source: 1) Strategist · p. 9

2) Executor:

AIDA separates evidence retrieval, independent challenge, and adjudication across investigation rounds. The Executor records retrieval outcomes, the Adversary tests the committed thesis against the ledger, and the Judge resolves the alert or requests more evidence.

2) Executor:

  • The Executor is the sole SIEM-facing, non-LLM component and converts requests into typed ledger entries without classifying the alert.Entries distinguish matching records, empty retrievals, and failed calls; transient failures may be retried.Source: 2) Executor · p. 9

3) Adversary:

  • The Adversary evaluates the committed thesis against shared ledger evidence without access to the Strategist’s private reasoning.It flags unsupported claims and issues contradiction or completeness challenges, but does not retrieve evidence itself.Source: 3) Adversary · pp. 9–10

4) Judge:

  • The Judge adjudicates the thesis and challenge against accumulated evidence, resolving the alert or directing another investigation round.Escalation requires concrete evidence of malicious activity; dismissal requires affirmative benign evidence or an investigation complete enough to justify closure.Source: 4) Judge · p. 10
  • The reference configuration fixes phase order and uses a per-round tool budget of five with a maximum of three rounds.The tool budget resets after continuation while the ledger persists; prompts and protocol settings were fixed before evaluation.Source: 4) Judge · pp. 10–11

D. Illustrative Walkthrough

AIDA substantially improves triage on the same alerts, with its Judge correcting unsupported benign-alert escalations in an illustrative trace. Its operating point trades higher analyst load for fewer missed attacks, while incurring relatively high token and tool costs.

D. Illustrative Walkthrough

  • In the illustrative benign-login case, the Adversary challenges an attack thesis unsupported by the recorded evidence, and the Judge correctly closes the alert in round one.The ledger contains seven successful logins from one internal address, with no failed attempt or activity linking the account to an attack; the history is deemed complete enough for closure.Source: D. Illustrative Walkthrough · p. 11

VI. EVALUATION OF AIDA

  • The comparison uses the same 1,247 alerts and GPT-4.1 reference model, while ablation and model experiments use a 300-alert subset containing 54 attacks.The evaluation retains the same evidence access and alert-triage setting as the systematic study.Source: §VI · p. 11

A. RQ3. End-to-End Triage

  • AIDA reaches 0.958 F1 with 0.948 precision, cuts the false-negative rate from 40.4% to 3.1%, and sends 18.4% of alerts to analysts.99.3% of dismissed alerts are benign.Source: A. RQ3. End-to-End Triage · p. 11
  • AIDA recovers 86 of 91 attacks dismissed by Direct-Prompt and escalates 23 of 26 attacks missed by all five studied approaches.It catches 124 of 126 directory-enumeration alerts and 93 of 94 WordPress-scanning alerts; all three service-scanning alerts are missed, but that phase is too small for a conclusion.Source: A. RQ3. End-to-End Triage · p. 11
  • The F1 improvement is statistically significant against every studied approach, including a paired improvement of 0.214 over Direct-Prompt.The paired 95% confidence interval against Direct-Prompt is [0.165, 0.269], and every McNemar test has p < 10^-13.Source: A. RQ3. End-to-End Triage · p. 11
  • AIDA uses 33,942 tokens and 4.3 tool calls per alert on average, with 12.7-second latency and an approximate GPT-4.1 cost of $0.109 per alert.This is at the upper end of studied approaches’ cost range and roughly the same per-alert cost as CoVe, whose F1 is 0.371.Source: A. RQ3. End-to-End Triage · p. 11

B. RQ4. Investigation Behavior

AIDA changes investigation behavior by preventing final decisions with no retrieved records, correcting unsupported benign-alert escalations, and requiring more successful searches before dismissal. Ablations show a measurable contribution from separate adjudication, while the experiment does not isolate a statistically significant contribution from the Adversary.

B. RQ4. Investigation Behavior

  • No attack alert reaches AIDA’s final decision without retrieved records, compared with 97–208 attack investigations per studied approach encountering an empty result.Of AIDA’s ten attack investigations with an empty result, six are escalated and four account for four of its seven false negatives.Source: B. RQ4. Investigation Behavior · p. 12
  • The Judge correctly reverses all 31 thesis decisions it changes, each of which had escalated a benign alert; it reverses no correct commitment.Challenge strength is at most 0.2 on all 1,216 upheld investigations and at least 0.8 on all 31 overturned investigations; all conclude in round one.Source: B. RQ4. Investigation Behavior · p. 12
  • AIDA averages 4.79 tool calls before dismissal versus 2.27 before escalation, and 2.78 versus 1.93 non-empty results, respectively.Both differences are statistically significant; the studied approaches stay within 25% of parity in total calls, and four retrieve fewer non-empty results before dismissal.Source: B. RQ4. Investigation Behavior · p. 12
  • AIDA escalates 23 of the 26 attacks missed by every studied approach, while three of its seven remaining false negatives meet the exact duplicate-ambiguity test.Source: B. RQ4. Investigation Behavior · p. 12

C. RQ5. Contribution of Protocol Roles

  • Removing the Judge reduces F1 from 0.952 to 0.895 and precision from 0.980 to 0.850, while false positives rise from one to nine.The paired 95% interval for the F1 difference is [0.008, 0.131], excluding zero; in No-Judge, the challenge-producing Adversary decides.Source: C. RQ5. Contribution of Protocol Roles · p. 12
  • AIDA exceeds No-Adversary by 0.006 F1 and Only-Strategist by 0.035, but both paired intervals include zero, so the experiment does not isolate a significant Adversary contribution.The full protocol uses 36,988 tokens per alert versus 15,383 for No-Adversary.Source: C. RQ5. Contribution of Protocol Roles · p. 12

D. RQ6. Models and Operating Conditions

AIDA’s effectiveness varies by underlying model and continuation behavior, but it also performs strongly in a different scenario. The paper cautions that evaluation remains bounded by its telemetry and that analyst use of the investigation record remains untested.

D. RQ6. Models and Operating Conditions

  • Qwen3.5 9B reaches 0.952 F1, while the other open-weight models range from 0.098 to 0.777, showing effectiveness depends on the model executing the protocol.The 9B Qwen model outperforms the 70B and 120B alternatives under the same protocol; its subset confusion matrix matches GPT-4.1’s despite two alert-level disagreements.Source: D. RQ6. Models and Operating Conditions · p. 13
  • Additional rounds correct seven initially incorrect commitments for GPT-OSS 120B, but Llama 3.1 8B produces 18 corrections and 33 regressions after continuation.The three-round limit bounds repeated continuation; additional rounds do not reliably improve a model that fails to converge.Source: D. RQ6. Models and Operating Conditions · p. 13
  • On a different AIT-LDSv2 scenario with 337 alerts and 5.93% attack prevalence, AIDA reaches 0.927 F1, with 0.905 precision and 0.950 recall.The single missed attack resulted from retrieval of only the originating alert, leaving the reasoning roles with incomplete context.Source: D. RQ6. Models and Operating Conditions · p. 13

VII. DISCUSSION AND LIMITATIONS

  • Evaluation covers scenarios within the AIT-LDSv2 dataset family; independently collected SOC telemetry is needed to test different organizational baselines, detection rules, and logging environments.AIDA starts from detector-generated alerts and depends on investigation-interface context; experiments do not evaluate how analysts use its ledger or how it affects review effort and overrides.Source: §VII · p. 13

VIII. CONCLUSION

  • The conclusion is that reliable triage depends on how evidence is gathered and conclusions evaluated before closure, not on additional reasoning alone.Source: §VIII · p. 13

APPENDIX A STUDIED REASONING APPROACHES

Table IV summarizes the five reasoning approaches evaluated in the study and the reasoning pattern implemented by each.

APPENDIX A STUDIED REASONING APPROACHES

  • Table IV summarizes the five studied reasoning approaches and the reasoning pattern implemented by each.The passage does not enumerate the approaches or describe their patterns.Source: Appendix A · p. 15

APPENDIX B AGENT PROMPTS

Appendix B reproduces the agent prompts used in the full-protocol configuration. They specify evidence-gathering, challenge, and final decision roles for tier-1 alert triage.

APPENDIX B AGENT PROMPTS

  • The reproduced prompts are for the full-protocol configuration, and Table IV states that the studied approaches share alerts, model, tool operations, and tool descriptions.The table summarizes each approach’s reasoning pattern.Source: Appendix B · p. 15, Table IV · p. 16
  • Investigators gather alert-specific evidence and classify alerts as TRUE_POSITIVE or FALSE_POSITIVE, treating dismissal as the higher-stakes decision.Prompts advise trying alternative searches when initial results are sparse and using payload indicators when corroborating events are absent.Source: Appendix B · pp. 15–16
  • Reviewers test whether the investigator’s thesis is evidence-grounded, apply stronger scrutiny to dismissals, and distinguish contradictory evidence from incomplete evidence.Escalations should be grounded in alert-specific evidence; same-host co-occurrence is informative but not conclusive.Source: Appendix B · pp. 16–17
  • The final decision-maker reviews evidence and both arguments, then concludes or continues investigation, calibrating confidence to evidence quality.High-confidence dismissal requires an affirmative benign explanation or a payload without attacker indicators; absent corroborating searches alone are insufficient.Source: Appendix B · pp. 17–18
Loading 2610.10608v1…