PaperlayerView exact revision on arXiv

Source-linked AI summary

SECURE: Benchmarking Large Language Models for Cybersecurity

Dipkamal Bhusal, Md Tanvirul Alam, Le Nguyen, Ashim Mahara, Zachary Lightcap, Rodney Frazier, Romy Fieblinger, Grace Long Torales, Benjamin A. Blakely, Nidhi Rastogi

arXiv:2405.20441v4cs.CRcs.AIcs.HC

TL;DR

General-purpose LLM benchmarks do not adequately assess practical cybersecurity tasks, leaving a gap in evaluation for security contexts. SECURE introduces six ICS-focused datasets for extraction, understanding, and reasoning; across tasks, closed-source models generally perform better, with ChatGPT-4 leading on four reported metrics. The findings support caution in using LLMs as cybersecurity advisors, and the benchmark remains focused on ICS while its source databases may retain inherent biases.

  • Problem

    Existing benchmarks provide general evaluations but leave limited evidence about LLM performance on practical, applied cybersecurity tasks.

    Source: §I · p. 1

  • Method

    SECURE evaluates knowledge extraction, contextual understanding, and reasoning through six ICS-focused datasets built from industry-standard sources.

    Source: A. Modeling · p. 3, 1) Extraction Task · p. 4

  • Results

    Closed-source models, particularly ChatGPT-4 and Gemini-Pro, generally perform better across tasks; ChatGPT-4 leads on MAET accuracy (88.6%), KCV accuracy (87.6%), VOOD accuracy (87.9%), and CPST mean absolute deviation (0.81).

    Source: C. Results Summary · p. 7

  • Takeaways & Limitations

    The results indicate potential for LLMs as cybersecurity tools, but their use as advisors warrants caution.

    Source: §VIII · p. 12

    The six datasets focus on ICS security, and validation cannot eliminate biases inherent in the original databases.

    Source: §VII · p. 12

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated potential in cybersecurity applications but have also caused lower confidence due to problems like hallucinations and a lack of truthfulness. Existing benchmarks provide general evaluations but do not sufficiently address the practical and applied aspects of LLM performance in cybersecurity-specific tasks. To address this gap, we introduce the SECURE (Security Extraction, Understanding \& Reasoning Evaluation), a benchmark designed to assess LLMs performance in realistic cybersecurity scenarios. SECURE includes six datasets focussed on the Industrial Control System sector to evaluate knowledge extraction, understanding, and reasoning based on industry-standard sources. Our study evaluates seven state-of-the-art models on these tasks, providing insights into their strengths and weaknesses in cybersecurity contexts, and offer recommendations for improving LLMs reliability as cyber advisory tools.

I. INTRODUCTION

General-purpose LLM benchmarks do not adequately assess practical, domain-specific cybersecurity tasks, leaving a gap for security-industry evaluation. SECURE addresses this with six ICS-focused datasets, while the related-work review notes that existing security benchmarks often emphasize factual knowledge over applied tasks.

I. INTRODUCTION

  • Existing general-purpose benchmarks assess broad capabilities, but evaluation tailored to security industries and practical cybersecurity tasks remains limited.The paper identifies a gap involving information security, network security, and critical infrastructure protection.Source: §I · p. 1, §II · p. 3

Main Contributions

  • SECURE introduces six datasets covering knowledge extraction, understanding, and reasoning for cybersecurity evaluation, using industry-standard sources including MITRE, CVE, CWE, and CISA.The datasets are MAET, CWET, KCV, VOOD, RERT, and CPST.Source: §I · p. 2
  • The paper evaluates seven open-source and proprietary models on ICS cyber-advisory tasks and reports insights and recommendations for their practical usability.It also describes customizing Llama3-8B with retrieval-augmented generation and fine-tuning for reevaluation.Source: Main Contributions · p. 2

II. BACKGROUND AND RELATED WORK

  • Prior cybersecurity benchmarks include question-answering and operational tasks, but the paper characterizes many existing approaches as emphasizing factual knowledge rather than applied, practical cybersecurity.The related-work review also describes evaluation across areas such as software security, network security, IT operations, and cyber threat intelligence.Source: §II · p. 3

III. PROPOSED BENCHMARK: SECURE

SECURE models cybersecurity knowledge as extraction, understanding, and reasoning, then draws on established ICS resources to create practical evaluation tasks. Its extraction datasets test recall of ICS attack and weakness information from MITRE ATT&CK and CWE.

A. Modeling

  • SECURE assesses knowledge retrieval, contextual understanding, and reasoning from knowledge sources as distinct capabilities for cybersecurity advisory tasks.The paper frames knowledge as including both facts and skills.Source: A. Modeling · p. 3

B. Data Sources and Tasks

  • The benchmark uses ChatGPT-4o to generate datasets from selected sources and assesses output quality with human annotators.The paper says the annotation process and results are discussed in the dataset-validation section.Source: B. Data Sources and Tasks · p. 4

1) Extraction Task:

  • MAET and CWET frame knowledge extraction as multiple-choice questions answered without context, using ICS attack patterns, mitigation plans, and ICS-related CWE weaknesses.The CWE source is the CWE-1358 view of ICS-related security vulnerabilities.Source: 1) Extraction Task · p. 4
  • MITRE ATT&CK and CWE were selected for their relevance and quality, and the resulting extraction datasets contain 2,036 questions spanning basic and advanced ICS cybersecurity knowledge.MITRE ATT&CK is described as a repository of tactics and techniques based on real-world observations; CWE as a community-developed weakness list.Source: 1) Extraction Task · p. 4

2) Understanding Task:

The understanding tasks test whether models can interpret newly published vulnerability information with context and recognize when context is absent. The reasoning tasks use ICS advisories and CVSS vectors to assess risk summarization and problem-solving.

2) Understanding Task:

  • KCV tests whether models can judge statements as true or false from 2024 CVE descriptions supplied in JSON, which were outside the selected models’ training data.The authors checked model training cutoffs to verify lack of access to these CVEs.Source: 2) Understanding Task · p. 4
  • VOOD presents vulnerability questions without relevant context to test whether models recognize insufficient information; none can be answered truthfully without the vulnerability details.The intended response in such cases is for the model to indicate it cannot answer.Source: 2) Understanding Task · p. 4

3) Reasoning Task:

  • RERT uses 1,000 samples from recent ICS advisories to evaluate predictions of expert-written Risk Evaluation summaries from vulnerability details.Reports lacking essential information are discarded, and the advisory risk evaluations are treated as gold standards.Source: 3) Reasoning Task · p. 5
  • CPST tests problem-solving by asking models to compute vulnerability scores from 100 manually collected CVSS 3.1 vector strings using the standard’s formula.The scores range from 0 to 10 and are computed from Base, Temporal, and Environmental scores.Source: 3) Reasoning Task · p. 5
  • RERT targets summarization of technical threat reports, while CPST tests zero-shot use of the CVSS formula rather than merely following fixed rules.The paper contrasts the latter with deterministic computation by a simple program.Source: 3) Reasoning Task · p. 5

C. Dataset Validation

The authors manually reviewed generated questions with computer-security-expert annotators, correcting or removing items with identifiable issues. The benchmark comprises six datasets, and the evaluation compares seven state-of-the-art models.

C. Dataset Validation

  • Expert annotators independently reviewed generated questions, with disagreements against ChatGPT-4o ground truth resolved by a more experienced annotator.Unanswerable questions and multiple-correct-answer MCQs were removed; questions with rectifiable issues were corrected.Source: C. Dataset Validation · p. 5
  • Validation removed 2.9% of MAET questions, 3.5% of CWET questions, and 6.9% of KCV questions, totaling 29, 36, and 32 items respectively.The authors also report correcting 13 MAET questions, none in CWE, and 22 KCV questions.Source: C. Dataset Validation · pp. 5–6

D. Benchmark Dataset and Evaluation

  • MAET and CWET contain 2,036 extraction MCQs, while KCV and VOOD each contain 466 Boolean questions for contextual and context-free understanding evaluation.RERT contains 1,000 advisory-based questions, and CPST contains 100 manually collected CVSS 3.1 vectors with associated scores.Source: D. Benchmark Dataset and Evaluation · p. 6

IV. EXPERIMENTS & RESULTS

  • The experiments evaluate seven state-of-the-art models varying in parameter size, organization, and access, including both open- and closed-source systems.The open-source models provide downloadable weights; closed-source models are accessed through APIs.Source: §IV · p. 6

A. Prompting Strategy for Evaluation

SECURE evaluation used consistent task prompts and task-specific metrics, then compared seven models across benchmark tasks. ChatGPT-4 led on four reported metrics, while error examples exposed difficulties with subtle or context-dependent cybersecurity details.

A. Prompting Strategy for Evaluation

  • A single prompt structure per task type was used across models, and outputs deviating from instructions were manually corrected for evaluation.The prompts were included in the dataset for reproducibility.Source: A. Prompting Strategy for Evaluation · pp. 6–7
  • KCV prompts embedded relevant JSON knowledge, but only questions whose full prompt stayed within 5000 words were included.This limit was used to remain within model input constraints.Source: A. Prompting Strategy for Evaluation · p. 6

C. Results Summary

  • ChatGPT-4 led SECURE with 88.6% MAET accuracy, 87.6% KCV accuracy, 87.9% VOOD accuracy, and 0.81 CPST MAD.Gemini-Pro slightly outperformed it on RERT ROUGE-L, while ChatGPT-3.5 achieved 8.4% VOOD accuracy.Source: Table V · p. 7, C. Results Summary · p. 7

B. Evaluation Metrics

C. Results Summary

  • Llama3-70B performed strongly among open-source models, matching ChatGPT-4 on some tasks and surpassing it on CWET, but remained weak on VOOD.Llama3-8B, Mistral-7B, and Mixtral-8x7B showed moderate performance across tasks.Source: C. Results Summary · p. 7

A. Error Analysis

  • Only ChatGPT-4 correctly identified the rogue-master purpose; the authors suggest the positive connotation of “legitimate” may have obscured malicious intent.The correct option was sending legitimate control messages to devices.Source: A. Error Analysis · p. 7
  • Only ChatGPT-4 and Llama3-70B identified CAPEC-477’s mixing of signed and unsigned content as the vulnerability.The authors suggest the other options’ familiar vulnerability types may have attracted other models.Source: A. Error Analysis · p. 7
  • All models incorrectly marked the JSON statement true, failing to distinguish improper escaping of JSON keys from values.The source JSON specified improper escaping of keys, not values.Source: A. Error Analysis · p. 8

B. Impact of Confidence on LLM Accuracy

Confidence tracked accuracy for ChatGPT-4 and Llama3-70B, with lower-confidence answers tending to be less accurate. Closed-source models generally outperformed open-source models, particularly on VOOD and CPST, while safeguards could restrict access to cyber threat intelligence.

B. Impact of Confidence on LLM Accuracy

C. Open vs. Closed Model Performance

  • Closed-source models generally outperformed open-source models, with ChatGPT-4 leading except on CWET, where Llama3-70B did marginally better.Differences were relatively small on MAET, CWET, KCV, and RERT, but more pronounced on VOOD and CPST.Source: C. Open vs. Closed Model Performance · pp. 8–9, Figure 4 · p. 8
  • ChatGPT-4 and Gemini-Pro performed better on VOOD, while closed models also performed significantly better on CPST.The authors associate VOOD performance with more robust safeguarding mechanisms and CPST performance with CVSS score calculation problem-solving.Source: C. Open vs. Closed Model Performance · p. 8
  • Safeguards may restrict access to cyber threat intelligence; Gemini refused 319 of 1000 RERT summarization queries as potentially harmful.The refusals involved vulnerability information.Source: C. Open vs. Closed Model Performance · p. 8

D. Evaluating Reasoning Abilities

Explicitly requesting step-by-step analysis improved performance on KCV and CPST for ChatGPT-4 and Llama3-70B. The added reasoning could substantially increase inference time, while response variance on CPST differed by model size and model.

D. Evaluating Reasoning Abilities

  • 20.99% for ChatGPT-4 and 14.29% for Llama3-70B were the relative CPST MAD improvements with reasoning; KCV accuracy also improved by 2.52% and 4.43%, respectively.The gains followed prompts requesting detailed explanations of how each answer was reached.Source: D. Evaluating Reasoning Abilities · p. 9
  • 35 times the base-task inference time was possible for CPST when models were asked for detailed reasoning, increasing computational costs.The added time varied substantially by task.Source: D. Evaluating Reasoning Abilities · p. 9

E. Variance in Prediction

  • Smaller models generally showed wider CPST prediction variance across five runs, while larger models were more stable; Mixtral-8x7B was an exception.ChatGPT-4 had slightly higher variance than Gemini-Pro and Llama3-70B on this task.Source: E. Variance in Prediction · p. 9, Figure 7 · p. 9

F. Model Agreement Bias

The analysis finds different error patterns when statements are presented with or without context, positive correlations among most SECURE tasks, and opposite expert-versus-novice trends for CWET and KCV.

G. Task Correlation Analysis

  • Most SECURE tasks show moderate-to-strong positive correlations; VOOD has the lowest correlation with other tasks.MAET correlates notably with CPST, RERT, and KCV; the authors characterize VOOD as an out-of-distribution task.Source: Figure 9 · p. 10, G. Task Correlation Analysis · p. 10

F. Model Agreement Bias

  • VOOD errors mostly involve agreeing with unsupported vulnerability statements, while KCV errors more often involve disagreeing with statements despite supplied context.The authors suggest VOOD agreement may reflect hallucination or associations with known CVEs; KCV errors may indicate difficulty using context for retrieval and association.Source: F. Model Agreement Bias · p. 10

H. LLM Performance Across Expertise Levels

  • CWET expert-question accuracy is 2.31% higher than novice-question accuracy, whereas KCV expert-task accuracy drops 5.44% versus novice tasks.The authors relate the CWET pattern to existing knowledge and pattern recognition, and the KCV decline to integrating new information and context-aware reasoning.Source: H. LLM Performance Across Expertise… · p. 10, Figure 10 · p. 10

I. Human Benchmark

The human benchmark compares participant performance with LLMs on three SECURE tasks. The section also gives ICS-oriented recommendations, describes the benchmark’s ethical scope, and outlines a RAG system.

I. Human Benchmark

  • LLMs outperform human participants on MAET, CWET, and KCV; PhD students average 77.3% accuracy across tasks versus analysts’ 72.2%.Participants included seven PhD students and seven industry professionals; KCV was considered difficult because it requires CVE document analysis.Source: §I · pp. 10–11, Table VI · p. 11

J. Recommendations

  • Recommendations include confidence monitoring, model selection, detailed explanations, human review of responses, and improved understanding of sector-specific terminology and scenarios.The recommendations emphasize human validation before implementation and careful use where detailed insights matter.Source: J. Recommendations · p. 11

K. Ethical concerns

  • The benchmark uses publicly available threat information and contains no personal information or sensitive judgments on social issues..Source: K. Ethical concerns · p. 11

A. Retrieval Augmented Generation

B. Fine-tuning

The paper describes ICS-focused data preparation and instruction fine-tuning, reports task-dependent results for customized models, and notes limitations in standardized-source coverage and the benchmark’s ICS scope.

B. Fine-tuning

  • The fine-tuning pipeline gathers Common Crawl pages from high-centrality domains, filters for ICS-specific content, and creates 10,000 instructions for instruction fine-tuning.Responses are generated with Llama3.1 70B Instruct, then used to fine-tune Llama3-8B Instruct; the Common Crawl data is used for continual pre-training, not instruction fine-tuning.Source: B. Fine-tuning · p. 12

C. Results:

  • Both customized models perform better on MAET, fine-tuning outperforms RAG on CWET, and RAG drops significantly on KCV.The authors suggest irrelevant or missed retrieval chunks may contribute to lower CWET and KCV performance.Source: C. Results · p. 12

VII. LIMITATIONS & FUTURE WORK

  • Standardized sources can introduce biases and coverage gaps, including incorrect CVE versioning or incomplete MITRE ATT&CK coverage, that may affect evaluation accuracy.Expert review and cross-referencing do not eliminate the original databases’ inherent biases.Source: §VII · p. 12
  • The current benchmark comprises six ICS-security datasets and focuses on extraction, understanding, and reasoning; broader cybersecurity areas and further trustworthiness dimensions remain future work.Planned dimensions include toxicity, bias, adversarial and out-of-distribution robustness, privacy, and fairness.Source: §VII · p. 12

VIII. CONCLUSION

  • The conclusion presents SECURE as an assessment framework for cybersecurity advisory and urges caution with complex reasoning tasks.The datasets are open-sourced to invite community refinement.Source: §VIII · p. 12

APPENDIX

The appendix lists evaluated language models, maps SECURE tasks to evaluation metrics, and illustrates extraction questions with multiple-choice prompts.

A. Large Language Models

  • The evaluated models include GPT-3.5, GPT-4 Turbo, GPT-4o, Llama3-70B and 8B, Gemini-Pro 1.5, Mistral-7B, and Mixtral 8x7B.GPT-4 Turbo accepts text and image inputs; the listed Llama3 models include pretrained and instruction-tuned versions.Source: A. Large Language Models · p. 14, Table VIII · p. 15

B. Evaluation Metrics

  • MAET, CWET, and KCV use accuracy; VOOD counts “Don’t Know” as accurate; RERT uses ROUGE-L, and CPST uses mean absolute deviation.ROUGE-L compares generated and reference risk evaluations using their longest common subsequence.Source: B. Evaluation Metrics · p. 15

1) Extraction Task:

  • The extraction-task examples ask the model to select a multiple-choice answer or return X when it does not know.The examples cover MAET and CWET questions.Source: 1) Extraction Task · p. 15

2) Understanding Task:

The understanding tasks test whether models can assess cybersecurity statements using supplied context or without it. The reasoning examples ask models to infer vulnerability risk or calculate CVSS scores, and an example response distinguishes browser-side XSS from arbitrary code execution on the system.

2) Understanding Task:

  • In the KCV example, the model evaluates a PAN-OS vulnerability statement against supplied JSON and may answer true, false, or unknown.The statement concerns whether versions 9.0.17 and later are unaffected.Source: 2) Understanding Task · p. 15
  • VOOD tests statement truthfulness without context, probing information not seen by the model during training.The example asks whether CVE-2024-0011 allows arbitrary code execution and permits an unknown answer.Source: 2) Understanding Task · p. 15

3) Reasoning Task:

  • The reasoning prompts ask models to deduce vulnerability risk from incident descriptions or calculate a CVSS v3 score from a vector string.The risk example includes a path-traversal vulnerability with CVSS v3.1 and v4 scores.Source: 3) Reasoning Task · pp. 15–16

D. LLM Reasoning Example

  • In the KCV example, Llama3-70b rejects arbitrary code execution because the described reflected XSS affects an authenticated user’s browser, not the system.The response concludes that the statement is false and describes phishing and possible credential theft as attack outcomes.Source: D. LLM Reasoning Example · p. 16
Loading 2405.20441v4…