PaperlayerView exact revision on arXiv

Source-linked AI summary

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

Yifan Wang, Dongliang Xu, Xiaoyong Li, Jing Li, Peter S. Pan, Yue Yao

arXiv:2610.31000v1cs.CRcs.AI

TL;DR

Cybersecurity log-analysis datasets often lack reproducible generation, aligned multi-source evidence, and dense reasoning annotations. SecLog-Bench builds an extensible benchmark with scaffolded evidence and structured reference chains, then evaluates attack-stage prediction and reasoning. Models show limited prediction performance, and correct labels do not guarantee reliable reasoning.

  • Problem

    Existing log-analysis resources often lack reproducible generation, aligned multi-source evidence, and dense annotations showing how evidence supports judgments.

    Source: §I · p. 1

  • Method

    SecLog-Bench simulates attack scenarios, aligns host, application, and network logs into evidence windows, and pairs each window with a stage label and reference reasoning chain.

    Source: §I · p. 1, §III · p. 2

  • Results

    The best model reaches 44.44% accuracy, while reasoning remains unreliable and correct predictions may still have flawed reasoning.

    Source: §I · p. 2

  • Takeaways & Limitations

    Cybersecurity log reasoning evaluation should assess evidence-supported reasoning rather than rely solely on final-label accuracy.

    Source: §I · p. 2, A. Discussion · p. 14

    The 216-window evaluation set cannot fully capture enterprise complexity, and its balanced stage-label distribution does not represent operational class prevalence.

    Source: B. Limitations · p. 15

Abstract

from arXiv · show

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

I. INTRODUCTION

Existing cybersecurity log-analysis resources often lack reproducible generation, aligned multi-source evidence, and dense annotations of how evidence supports judgments. SecLog-Bench addresses these gaps with an extensible benchmark and evaluates both attack-stage prediction and reasoning quality.

I. INTRODUCTION

  • Existing datasets often lack reproducible generation, aligned multi-source evidence, and dense annotations explaining how visible evidence supports final judgments.The paper notes that labels alone cannot establish reasoning reliability.Source: §I · p. 1
  • SecLog-Bench uses an extensible simulation platform covering 29 reproducible scenarios and collects host audit logs, application logs, and network traffic for alignment.Lightweight auxiliary indices support cross-source evidence alignment.Source: §I · pp. 1–2
  • The best model reaches 44.44% accuracy and 45.07% Macro-F1, while the highest SRF1 is 54.87%; correct predictions may still have flawed reasoning.Reasoning quality is often misaligned with prediction accuracy, so final-label accuracy alone can overstate capability.Source: §I · p. 2
  • Each evidence window has an attack-stage label and structured reference reasoning chain, enabling evaluation of prediction, reasoning quality, reliability, and consistency.The reference chain is withheld from the evaluated model and used after inference.Source: §I · p. 2

II. MOTIVATING EXAMPLE

A single application log can show a suspicious request without establishing whether it executed or produced output; linked application, audit, and network evidence can support a more advanced judgment. SecLog-Bench formalizes this as window-level reasoning from visible evidence, including an uncertainty label when no concrete stage is supported.

II. MOTIVATING EXAMPLE

  • A request containing cat /etc/passwd and an HTTP 200 status shows suspicious activity, but does not alone establish command execution or returned output.The example's multi-source view links the request to an audit event and a network response containing passwd-like records, jointly supporting RESULT.Source: §II · p. 2

III. PROBLEM DEFINITION

  • SecLog-Bench defines the task as window-level attack-progress reasoning over scaffolded multi-source logs, requiring a stage label and evidence-referenced reasoning chain.The benchmark concerns already identified attack-related evidence, not attack discovery.Source: §III · p. 2
  • An evidence window is selected to capture a stage-relevant portion of the attack chain, and judgments use only evidence visible within its boundaries.When visible evidence supports no concrete stage, the model should output STAGE UNCLOSED rather than force a label.Source: §III · p. 2
  • The four window labels are PREPARATION, EXECUTION, RESULT, and STAGE UNCLOSED, based on the stage supported by visible evidence rather than full-attack progress.STAGE UNCLOSED is neither a benign class nor an actual attack lifecycle stage.Source: §III · pp. 2–3
  • A reasoning chain contains substantive claims with explicit references to indexed evidence units, allowing analysts to trace the evidence cited for each claim.The model receives the evidence view, not the ground-truth label or reference reasoning chain.Source: §III · p. 3

IV. SECLOG-BENCH CONSTRUCTION

SecLog-Bench builds attack-stage reasoning windows from reproducible scenarios and synchronously collected host audit, application, and network logs. It aligns these sources around a core audit sequence, then verifies window-level labels against visible evidence.

A. Workflow Overview

  • The workflow gives models only the model-visible evidence view, then uses the stage label and human-confirmed reference chain for post-hoc evaluation.Each window retains its stage label and a human-confirmed reference reasoning chain.Source: A. Workflow Overview · p. 3

B. Data Collection

  • Across 29 reproducible scenarios, SecLog-Bench collects three log sources and selects stage-labeled local windows from an audit-centered attack-behavior chain.The sources are host audit logs, application logs, and network traffic; the scenarios include real CVE exploits, Sandworm-style simulations, and multi-hop attacks.Source: B. Data Collection · p. 3
  • Host audit logs form the core sequence, while auxiliary indices link application and network observations to corresponding host behaviors without simple timestamp-based concatenation.Indices include log-writer identity, request or processing-unit boundaries, traffic anchors, and NAT or connection aliases.Source: B. Data Collection · pp. 3–4
  • Raw sysdig events become audit triples linking a process, thread, or runtime entity to a file, child process, connection, or remote endpoint.The relation families are NET, FILE, and PROCESS; relation selection also considers scenario coverage, semantic stability, causal or information-flow relevance, and attack-stage reasoning suitability.Source: B. Data Collection · p. 4
  • Stable audit behaviors form the core sequence, useful non-core information is retained as context, and weakly causal, noisy, or unstable events are filtered or kept queryable.Application logs, network traffic, and retained audit context are attached to the core sequence using auxiliary indices and local structural relations.Source: B. Data Collection · p. 4
  • Two reviewers verify or revise each window’s inherited stage label using only visible, traceable evidence and behavioral relations within that window’s boundary.Across 216 windows, reviewers reached 91.67% observed agreement before adjudication, with Cohen’s κ = 0.89; disagreements were resolved through consensus.Source: B. Data Collection · p. 5

C. Reference Reasoning Chain Annotation

The benchmark creates human-confirmed reference reasoning chains from window evidence and stage labels, while keeping those chains hidden from evaluated models. Annotation uses compressed evidence views, structured reasoning steps, independent risk review, and human confirmation.

C. Reference Reasoning Chain Annotation

  • The reference chain supplies the explanation that a stage label alone lacks, and is used after inference to assess whether model reasoning is evidence-supported and covers key judgment conditions.The reference chain is not exposed to evaluated models.Source: C. Reference Reasoning Chain Annotation · p. 5
  • De-identified evidence compression removes answer-revealing cues, then selects and organizes facts, cross-source context, and source coordinates without adding outside facts or rewriting evidence.Annotation and review views are constructed from the window evidence, not exposed to evaluated models, and preserve evidence needed for stage judgment.Source: C. Reference Reasoning Chain Annotation · pp. 5–6
  • Compression prioritizes salient attack behaviors, cross-source continuity, and coverage of window boundaries, transitions, sources, and visible reasons an SU window remains inconclusive.For SU windows, retained evidence must not make a window with attack-related evidence appear evidence-free.Source: C. Reference Reasoning Chain Annotation · p. 6
  • Each reference chain follows four steps: scope, facts, relation, and judgment, moving from evidence boundaries through observable facts and cross-source relations to a stage conclusion.The relation step checks temporal adjacency, subject continuity, object consistency, and cross-source correspondence while retaining weak links and boundary uncertainty.Source: C. Reference Reasoning Chain Annotation · p. 6
  • Reasoning statements follow visible log facts, their behavioral meaning, and their relevance to the stage judgment, avoiding conclusions drawn directly from background or labels.Models need not reproduce the reference chain’s four-step structure or match it verbatim; evaluation checks substantive support, valid references, and coverage.Source: C. Reference Reasoning Chain Annotation · pp. 6–7
  • AI-generated candidate chains undergo independent risk review and become final references only after human confirmation; across 216 windows, candidate decisions had 92.13% observed agreement with human consensus.High-risk candidates receive higher review priority, and samples with unclear boundaries, insufficient support, or high review risk are cross-checked by three reviewers.Source: C. Reference Reasoning Chain Annotation · p. 7

D. Reasoning-Aware Evaluation Protocol

The protocol separates model inference from post-hoc assessment of predictions and evidence-referenced reasoning. It evaluates reasoning-unit support, evidence references, errors, and coverage of key stage conditions using fixed rules and human review for selected disagreements.

D. Reasoning-Aware Evaluation Protocol

  • Models receive the same scaffolded multi-source evidence view and output a stage label plus natural-language reasoning with explicit evidence references; evaluation compares predictions and assesses reasoning post hoc.The model does not receive the stage label or reference reasoning chain during inference.Source: D. Reasoning-Aware Evaluation Protocol · p. 7
  • Two independent assistants score reasoning units and key stage conditions; scalar scores are averaged, disagreements above 0.30 are reviewed, and either assistant’s binary risk flag is retained.Human reviewers adjudicate unclear boundaries or large assistant disagreements without changing the rubric, aggregation rule, or thresholds.Source: D. Reasoning-Aware Evaluation Protocol · p. 8
  • SRP measures precision of generated reasoning units under evidence constraints, while SCC measures coverage of key judgment conditions in the reference chain.For SCC, each condition receives 1.0 for full coverage, 0.5 for partial coverage, or 0 for incorrect or absent coverage.Source: D. Reasoning-Aware Evaluation Protocol · p. 8
  • Critical Error flags severe evidence errors, contradictions with current-window evidence, or out-of-window dependence; Invalid Evidence Reference flags references outside visible scope or irrelevant to their reasoning unit.The sample-level reliability rule also applies fixed thresholds for SRP, SCC, and SRF1.Source: D. Reasoning-Aware Evaluation Protocol · p. 8
  • Human-only review of 44 windows covering 352 outputs from eight models agreed with fixed-pipeline sample-level reliability decisions at 89.20%, with Cohen’s κ = 0.78.The windows represented about 20% of the evaluation set, with 11 windows from each ground-truth label.Source: D. Reasoning-Aware Evaluation Protocol · p. 8

V. BENCHMARK STATISTICS

SecLog-Bench combines reproducible attack scenarios with multi-source evidence windows and human-confirmed reasoning chains. Its statistics describe scenario coverage, evaluation-set composition, construction effort, and the engineering boundary of scenario extension.

Scenario coverage

  • The construction pool spans 29 reproducible scenarios, including 25 CVE-based and four non-CVE scenarios.One multi-hop scenario uses a 26-node service cluster and an attack path with ten inter-host hops.Source: §V · pp. 8–9

ATT&CK mappings

  • Across the 29 scenarios, analysis identified 93 associations with ATT&CK techniques, covering 23 directly assigned IDs and ten tactics.Mean pairwise Jaccard similarity was 0.2904 for direct IDs and 0.3659 at the parent-technique level.Source: §V · p. 9

Evaluation set

  • The final evaluation set contains 216 human-reviewed evidence windows, balanced at 54 per label, with 864 annotated reasoning steps.Candidate reference chains are reviewed against window evidence, and source coordinates retain traceability to raw records.Source: §V · pp. 9–10

Scope and constraints

  • Scenario extension still requires non-trivial engineering effort, despite the benchmark's testbed-based regeneration.The final evaluation-set size reflects a trade-off among annotation depth, manual review, label balance, and sample count—not a raw-log or window-construction limit.Source: §V · pp. 9–10

Evaluation set distribution

  • Table II reports retained windows by scenario group and stage label; the Java group has more windows, but selection is not concentrated on one scenario.The label mix within each group reflects the stage evidence exposed by its attack chains.Source: Table II · p. 9, §V · p. 10

Construction scale

  • Automated construction takes under 70 minutes per scenario, including 20 minutes for raw-log collection; SecLog-Bench combines multi-source logs, scaffolded evidence, extensible generation, and window-level reasoning chains.The timing excludes human review; the combination is unique among the datasets compared in Table III.Source: §V · p. 10, Table III · p. 10

VI. LLM PERFORMANCE

The evaluation examines LLM performance on scaffolded multi-source evidence across attack-stage prediction, reasoning quality, and prediction–reasoning consistency, alongside evidence and sampling variations.

Evaluation dimensions

  • The benchmark evaluates attack-stage prediction and reasoning quality, including consistency between predictions and reasoning.These are two of the four stated evaluation dimensions.Source: §VI · p. 10

Evidence analysis

  • The evaluation also examines failure modes and how evidence organization and source coverage affect performance.These dimensions concern reasoning over scaffolded multi-source evidence.Source: §VI · p. 10

Sensitivity analysis

  • Sampling and voting settings are examined for sensitivity.Source: §VI · p. 10

A. Experimental Setup

The study benchmarks eight representative LLMs under shared samples, evidence views, output requirements, and metrics. The reported cross-model differences reflect available inference interfaces, and results are limited to the fixed evaluation samples.

Experimental setup

  • Eight representative LLMs spanning model families and capability levels are evaluated on the same samples, evidence view, output requirements, and metrics.Model-specific maximum output lengths range from 2,048 to 8,192 tokens; provider-default reasoning behavior is used where applicable.Source: A. Experimental Setup · p. 10
  • Cross-model differences reflect each model’s available inference interface rather than matched reasoning budgets.No explicit reasoning-effort parameter was supplied, and transient failures were retried with the initial configuration.Source: A. Experimental Setup · p. 10

Scope

  • The reported metrics characterize performance on the fixed window-level sample set and do not demonstrate generalization to unseen attack scenarios.The benchmark data and experimental code are publicly available.Source: A. Experimental Setup · pp. 10–11

B. Attack-Stage Prediction Performance

Under the compressed multi-source evidence view, models show limited and uneven attack-stage prediction performance. The strongest overall result still leaves fewer than half of evidence windows correctly labeled.

Overall performance

  • 44.44% Accuracy and 45.07% Macro-F1 are Claude Sonnet 4.6’s best overall results under the compressed multi-source evidence view.The result exceeds the 25% uniform-random reference for four labels, but fewer than half of windows receive the correct stage label.Source: B. Attack-Stage Prediction Performance · p. 11

Per-label performance

  • 25.46% is the mean recall for RESULT across models, the lowest of the four labels.No model consistently distinguishes all four labels; high recall on one label does not ensure balanced performance across the others.Source: B. Attack-Stage Prediction Performance · p. 11
  • 20.03% Macro-F1 is Llama-3.1-8B-Instruct’s result despite its highest Prep. Recall; DeepSeek-R1 has the highest Res. Recall but low Exec. Recall.Source: B. Attack-Stage Prediction Performance · p. 11

Model-scale comparison

  • Qwen3-32B achieves higher Accuracy and Macro-F1 than Qwen3-8B but lower SU recall.The overall advantage is therefore not uniform across labels; the main difficulty is stable discrimination among concrete stages and SU, not attack detection.Source: B. Attack-Stage Prediction Performance · p. 11

C. Reasoning Quality and Prediction–Reasoning Consistency

Correct attack-stage predictions do not guarantee reliable reasoning, and prediction accuracy is only weakly associated with reasoning quality. The section compares reasoning metrics with human judgments and examines efficiency and threshold sensitivity.

C. Reasoning Quality and Prediction–Reasoning Consistency

  • Claude Sonnet 4.6 has the best attack-stage prediction performance but only 38.67% SRF1 and 41.67% TT/Correct, while GPT-5.4 leads SRF1 at 54.87%.GPT-5.4 also has 81.94% TT/Correct and the lowest TF rate, 6.02%; Claude Sonnet 4.6 has 76.08% SCC but 25.93% SRP.Source: C. Reasoning Quality and… · p. 11
  • SRF1 distinguishes human-assessed reliable outputs better than reference-similarity metrics, with AUROC 0.918 and AP 0.933 versus approximately 0.62 and 0.612.The comparison used 352 human-assessed outputs from 44 windows; BERTScore truncated seven candidate outputs and 336 paired reference inputs, while ROUGE-L used full texts.Source: C. Reasoning Quality and… · p. 11
  • Claude Sonnet 4.6 averages 933.89 tokens and has the lowest SRF1/1K Tok. at 41.41, while GPT-5.4 averages 380.39 tokens and leads at 144.25.The Qwen3-32B model is more token-efficient than Qwen3-8B but does not achieve higher reasoning reliability.Source: C. Reasoning Quality and… · p. 12
  • As the SRP threshold rises from 0.30 to 0.36, reliability falls from 37.1% to 31.1%, while mismatch rates remain relatively stable at 32.8% to 31.4%.The study reapplies the rule to 1,728 outputs while holding SCC and SRF1 thresholds and two hard gates fixed.Source: C. Reasoning Quality and… · p. 12
  • At the model level, attack-stage prediction accuracy and reasoning quality show a weak Pearson correlation (r = 0.15, 95% CI [−0.62, 0.77]); each point represents one model.The plot shows a least-squares fit and a 95% bootstrap confidence interval for the fitted trend.Source: Figure 6 · p. 12

D. Failure Modes in Attack-Stage Reasoning

Label-wise and conditional analyses show that recall can mask unreliable reasoning, with failures clustering around missed RESULT evidence and poor calibration of evidence sufficiency.

D. Failure Modes in Attack-Stage Reasoning

  • For EXECUTION, Claude Sonnet 4.6 has 48.15% recall but 0% TT and 48.1% TF, so none of its correct EXECUTION predictions has reliable reasoning.PREPARATION recall for Llama-3.1-8B-Instruct is 62.96%, split into 24.1% TT and 38.9% TF; for RESULT, TT is at most 14.8% for every model.Source: D. Failure Modes in Attack-Stage… · p. 12
  • Result-as-Non-Result errors range from 53.7% to 92.6% across models, and Unclosed-as-Stage errors are at least 50.0% for every model.Llama-3.1-8B-Instruct reaches 92.6% Unclosed-as-Stage, indicating frequent concrete-stage predictions for evidence-insufficient windows.Source: D. Failure Modes in Attack-Stage… · pp. 12–13
  • Fig. 7 groups TT/TF/FT/FF outcomes by ground-truth stage label, while Fig. 8 reports four conditional failure rates: Non-Exec as Exec, Result as Non-Result, Unclosed as Stage, and Stage as Unclosed.The figures structure comparisons around prediction–reasoning consistency by label and recurring conditional error patterns.Source: Figure 7 · p. 13, Figure 8 · p. 13
  • Qwen3-32B has a 64.2% Non-Exec-as-Exec rate, while Qwen3-8B maps 42.0% of concrete-stage windows to SU.Llama-3.1-8B-Instruct shows the opposite SU-calibration pattern, mapping only 3.7% of concrete-stage windows to SU.Source: D. Failure Modes in Attack-Stage… · p. 13

E. Effects of Evidence Organization and Source Coverage

Evidence organization and source coverage affect attack-stage reasoning, while the tested sampling and voting settings do not consistently improve prediction. The discussion recommends traceable, clearly related evidence views and retains human review as necessary.

E. Effects of Evidence Organization and Source Coverage

  • Scaffolded presentation raises Accuracy from 29.17% to 50.00% and SRF1 from 36.59% to 44.09%, while reducing Critical Error from 39.58% to 16.67%.Invalid-reference rates remain similar at 5.21% and 4.17%; the comparison holds the model and underlying compressed multi-source evidence fixed.Source: E. Effects of Evidence Organization… · p. 14
  • Multi-Comp-Scaf. achieves the strongest attack-stage prediction among evaluated views; combining all three sources also gives the best Macro-F1 and 96.33% SRP† and 94.78% SCC†.Compression and scaffolded presentation improve measured performance, while audit, application, and network evidence provide complementary context.Source: E. Effects of Evidence Organization… · p. 14

F. Sensitivity to Sampling and Voting Settings

  • Across tested sampling and voting settings, prediction does not improve consistently; maximum Accuracy is 25.00% for Qwen and 24.07% for Llama.Maximum Macro-F1 is 20.50% for Qwen and 21.32% for Llama; relative to the zero-temperature baseline, maximum Accuracy gain is 1.85 percentage points for Qwen and none for Llama.Source: F. Sensitivity to Sampling and Voting… · p. 14

A. Discussion

  • Current LLM predictions and reasoning outputs still require human review, although models may assist analysts by organizing facts, identifying possible cues, and exposing evidence gaps.The authors state that models do not yet meet the task's reliability requirements.Source: A. Discussion · p. 14
  • For cybersecurity log reasoning, evidence views should prioritize traceability and clear cross-source relations over simply expanding context length.The discussion notes that uncompressed multi-source evidence can expose models to weakly relevant fragments and make key evidence and valid references harder to locate.Source: A. Discussion · p. 14

B. Limitations

The limitations bound the study to post-detection attack-progress reasoning and identify constraints in evaluation scale, label granularity, and reproducibility. The related work spans evidence organization and LLM reasoning, while the conclusion positions SecLog-Bench as an evaluation basis beyond final-label accuracy.

B. Limitations

  • The results address post-detection attack-progress reasoning, not attack discovery, open-stream candidate-window retrieval, benign-versus-malicious discrimination, or operational detection false-positive rates.STAGE UNCLOSED denotes insufficient evidence for a concrete stage judgment rather than benign activity.Source: B. Limitations · pp. 14–15
  • The 216-window, label-balanced evaluation set cannot fully capture enterprise complexity or operational class prevalence, and four window-level labels abstract away finer attack-lifecycle transitions.Potential anchoring bias may remain because human reviewers see AI-generated candidate chains and risk flags; service-model updates, API changes, and inference settings may affect reproducibility.Source: B. Limitations · p. 15

A. Attack Evidence Organization / B. LLM Reasoning

  • Prior evidence-organization work covers provenance, network patterns, log semantics, and source alignment, but not LLM-facing views of correlated evidence for inferring attack progress.Reasoning benchmarks and security benchmarks also rarely meet on cybersecurity log evidence: the former are not designed for security-log analysis, while the latter usually emphasize task performance over reasoning.Source: A. Attack Evidence Organization · p. 15, B. LLM Reasoning · p. 15

IX. CONCLUSION

  • SecLog-Bench evaluates scaffolded audit, application, and network evidence for attack progress, stage prediction, and reasoning consistency; experiments show correct predictions need not imply reliable reasoning.The conclusion presents the benchmark as a systematic basis for evaluating cybersecurity log reasoning beyond final-label accuracy.Source: §IX · p. 15
Loading 2610.31000v1…