PaperlayerView exact revision on arXiv

Source-linked AI summary

CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge

Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, Merouane Debbah

arXiv:2402.07688v2cs.AIcs.CR

TL;DR

Because cybersecurity spans diverse, demanding topics, the paper builds CyberMetric-10000 from generated and reviewed questions to examine cybersecurity knowledge. On CyberMetric-80, GPT-4o outperformed all human experts in a closed-book test.

  • Problem

    Cybersecurity spans diverse topics that challenge human mastery, raising whether machine intelligence surpasses humans across cybersecurity knowledge in a closed-book test.

    Source: §I · p. 1

  • Method

    CyberMetric-10000 comprises 10,000 cybersecurity questions generated by GPT-3.5-turbo from text chunks and reviewed by Falcon-180B and human validators.

    Source: §I · p. 1, B. ➁Question Generation · p. 2

  • Results

    GPT-4o outperformed all human experts on CyberMetric-80; expert humans averaged 72% accuracy, comparable to Llama 2-70B.

    Source: §V · p. 5, §VII · p. 6

  • Takeaways & Limitations

    The authors conclude that machine intelligence surpassed human performance on knowledge-based cybersecurity questions in a closed-book scenario.

    Source: §VII · p. 6

    CyberMetric focuses exclusively on cybersecurity questions and knowledge, so it is unsuitable for general conclusions about machine versus human intelligence, intuition, or problem-solving.

    Source: §VII · p. 7

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used across various domains, from software development to cyber threat intelligence. Understanding all the different fields of cybersecurity, which includes topics such as cryptography, reverse engineering, and risk assessment, poses a challenge even for human experts. To accurately test the general knowledge of LLMs in cybersecurity, the research community needs a diverse, accurate, and up-to-date dataset. To address this gap, we present CyberMetric-80, CyberMetric-500, CyberMetric-2000, and CyberMetric-10000, which are multiple-choice Q&A benchmark datasets comprising 80, 500, 2000, and 10,000 questions respectively. By utilizing GPT-3.5 and Retrieval-Augmented Generation (RAG), we collected documents, including NIST standards, research papers, publicly accessible books, RFCs, and other publications in the cybersecurity domain, to generate questions, each with four possible answers. The results underwent several rounds of error checking and refinement. Human experts invested over 200 hours validating the questions and solutions to ensure their accuracy and relevance, and to filter out any questions unrelated to cybersecurity. We have evaluated and compared 25 state-of-the-art LLM models on the CyberMetric datasets. In addition to our primary goal of evaluating LLMs, we involved 30 human participants to solve CyberMetric-80 in a closed-book scenario. The results can serve as a reference for comparing the general cybersecurity knowledge of humans and LLMs. The findings revealed that GPT-4o, GPT-4-turbo, Mixtral-8x7B-Instruct, Falcon-180B-Chat, and GEMINI-pro 1.0 were the best-performing LLMs. Additionally, the top LLMs were more accurate than humans on CyberMetric-80, although highly experienced human experts still outperformed small models such as Llama-3-8B, Phi-2 or Gemma-7b.

I. INTRODUCTION

Cybersecurity’s breadth makes comprehensive knowledge difficult to master and motivates tests of both model ranking and human-versus-machine performance. The paper introduces CyberMetric’s broad cybersecurity benchmark and a human comparison on its smaller subset.

I. INTRODUCTION

  • The paper asks whether machine intelligence has surpassed humans across cybersecurity knowledge and which available model achieves the highest accuracy.The questions focus on closed-book performance across diverse cybersecurity domains.Source: §I · p. 1
  • CyberMetric covers nine cybersecurity domains with 10,000 questions and answers drawn from hundreds of guidelines, standards, books, and research papers.The source material totals over 100,000 pages.Source: §I · p. 1
  • The benchmark includes CyberMetric-80, CyberMetric-500, and CyberMetric-2000 subsets, with the 80- and 500-question sets fully validated by human experts.The datasets are made available to the research community as a reference for cybersecurity knowledge.Source: §I · p. 2
  • A study of 30 participants answering CyberMetric-80 compares human and LLM cybersecurity knowledge in a closed-book multiple-choice setting.The study is framed around performance across the breadth of cybersecurity knowledge.Source: §I · pp. 1–2

II. RELATED WORK

Prior multiple-choice benchmarks cover other domains, while the paper identifies a lack of large-scale datasets for broad cybersecurity knowledge. Existing cybersecurity assessment is described as narrower in scope or validation.

II. RELATED WORK

  • Large-scale datasets for assessing broad cybersecurity knowledge had not been developed prior to this research, despite datasets for math, coding, and reasoning.Related multiple-choice datasets include QASC and OpenBookQA for elementary science, alongside multilingual QA datasets.Source: §II · p. 2
  • CodeApex [8] assesses programming through comprehension, generation, and correction tasks; GPT-4 led its tested LLMs, while human performance varied by exam setting.The reported human comparison found novices similar to GPT-4 in closed-book tests after learning, while humans did better in open-book exams.Source: §II · p. 2
  • Liu introduced a cybersecurity dataset designed to evaluate LLM capabilities, based entirely on one material; it included around two hundred questions, and human comparison and validation were not part of its development process.Source: §II · p. 2

III. METHODOLOGY

CyberMetric’s methodology uses a phased workflow to build a reference dataset, beginning with source collection and RAG-based question generation.

III. METHODOLOGY

  • The dataset workflow has five phases: data collection, question generation, post-processing, validation, and reference-dataset creation.Source: §III · p. 2

A. ➀Data Collection

  • GPT-3.5-turbo generated questions using RAG over cybersecurity standards, research papers, public books, RFCs, and other publications.The source corpus exceeded 100,000 pages; PDF text was split into chunks of 8,000 tokens after irrelevant sections were excluded.Source: A. ➀Data Collection · p. 2
  • PDF text was extracted and divided into 8,000-token chunks, with tables of contents, prefaces, acknowledgements, pictures, references, and appendices excluded.Source: A. ➀Data Collection · p. 2

B. ➁Question Generation

Question generation and successive review stages filtered, corrected, and validated the generated items. More than 200 hours of expert review produced the final 10,000-question dataset.

B. ➁Question Generation

  • GPT-3.5-turbo generated ten multiple-choice questions per 8,000 tokens, producing 11,000 questions before review by Falcon-180B and human validators.Falcon review removed 1.7% for grammatical or semantic errors, and human validators removed another 2.3%; solution accuracy was not yet checked at this stage.Source: B. ➁Question Generation · p. 2

C. ➂Question Post-processing

  • T5-base flagged 230 questions for grammar improvement, and GPT-4, GPT-4o, and Mistral7x8b assessed answer correctness across the remaining 10,470 questions.Falcon-180B also filtered questions that could not be understood without external references.Source: C. ➂Question Post-processing · p. 3

D. ➃Test Phase

  • Expert review addressed multiple correct answers, outdated information, errors in source material, and missing references before reducing the dataset to 10,000 questions.The final validation stage required more than 200 hours of expert effort.Source: D. ➃Test Phase · p. 3

E. ➂Reference Dataset Creation

The authors created CyberMetric-80 and CyberMetric-500 for human testing and expert validation, recruited participants for a closed-book comparison, and statistically checked answer-option consistency and balance.

E. ➂Reference Dataset Creation

  • CyberMetric-80 was created for human participants, while CyberMetric-500 answers were verified by experienced cybersecurity experts with recognized certifications.The expert panel members had at least 10 years of experience.Source: E. ➂Reference Dataset Creation · p. 4
  • Thirty volunteers spanning beginners to experts were recruited to complete the questionnaire without additional help.Recruitment used social networks and contacts at universities, research institutions, and Big4 consulting companies.Source: E. ➂Reference Dataset Creation · p. 4
  • Automated statistical analysis checked that answers were consistent and evenly distributed across options A, B, C, and D.Source: E. ➂Reference Dataset Creation · p. 4

IV. EXPERIMENTAL RESULTS

The experiments compared 25 LLMs across four CyberMetric datasets and assessed human performance on CyberMetric-80. GPT-4o and GPT-4-turbo led proprietary models, while the human results varied by experience.

IV. EXPERIMENTAL RESULTS

  • The experiments compared 25 LLMs on all four CyberMetric datasets, compared humans with LLMs on CyberMetric-80, and assessed dataset accuracy.Source: §IV · p. 4

A. Assessing LLMs Performance

  • The 25 LLMs were tested on an AWS instance with eight NVIDIA Tesla A100 GPUs using temperature 1.0, top p 0.9, and top k 50.Source: A. Assessing LLMs Performance · p. 4
  • GPT-4o and GPT-4-turbo were the most accurate proprietary models; Mixtral-8x7B-Instruct and Falcon-180B were the top open-source models.The reported rankings can vary by 3–4 percentage points across runs because model outputs are probabilistic.Source: A. Assessing LLMs Performance · p. 4, Table III · p. 5

B. Human Performance on CyberMetric-80

  • Two participants who lacked cybersecurity experience were excluded after their answers closely matched GPT-3.5 output.Their accuracy and 92% of their incorrect responses matched GPT-3.5.Source: B. Human Performance on CyberMetric-80 · p. 4
  • The mean accuracy for experienced participants was 72.24%, while beginners’ scores ranged from 21.25% to 63.75%.The highest individual score was 71/80, approximately 88.75%; the participant median score was 56.Source: B. Human Performance on CyberMetric-80 · p. 4

C. Dataset Accuracy

The authors use fully validated CyberMetric-80 and CyberMetric-500 as reference points when assessing the accuracy of CyberMetric-10000, while noting residual errors and content differences.

C. Dataset Accuracy

  • CyberMetric-80 and CyberMetric-500 are fully validated and serve as reference points for assessing the larger dataset.Source: C. Dataset Accuracy · p. 5
  • The authors estimate that 2–3% of CyberMetric-10000 questions still have issues.They invite reports of inadequate or questionable answers through the dataset’s GitHub repository.Source: C. Dataset Accuracy · p. 5
  • Lower accuracy on CyberMetric-10000 may reflect its larger proportion of recent recommendations, guidelines, and research that many models do not know, as well as question issues.Source: C. Dataset Accuracy · p. 5

V. DISCUSSION - HUMAN VS MACHINE

Expert humans averaged performance comparable to Llama 2-70B, while GPT-4o, Mixtral-8x7B-Instruct, and GPT-4-turbo significantly outperformed humans on CyberMetric-80. The discussion also identifies difficult questions and domains.

V. DISCUSSION - HUMAN VS MACHINE

  • GPT-4o, Mixtral-8x7B-Instruct, and GPT-4-turbo significantly outperformed humans on CyberMetric-80; expert humans averaged 72% accuracy, comparable to Llama 2-70B.Top human performers came close to Falcon-180B, GPT-3.5, and GEMINI-pro 1.0.Source: §V · p. 5

A. Most Difficult Questions for Humans on CyberMetric-80

B. Most Difficult Questions for LLMs

1) Difficulty 1: Cutting Edge Research:

The examples show that cybersecurity questions can test both awareness of recent guidance and precise calculations. Models may struggle with each, while tool access can help with calculation tasks.

1) Difficulty 1: Cutting Edge Research:

  • Outdated training data can lead models to recommend periodic password changes, contrary to current NIST guidance to change passwords only upon evidence of compromise.The passage says only a few models, including GPT-4o and GPT-4, may answer such questions accurately, possibly because of internet access or recent training data.Source: 1) Difficulty 1: Cutting Edge Research · p. 6
  • The January 2023 BSI guideline sets DLIES prime p at no less than 3000 bits and prime q at a minimum of 250 bits, a question even expert analysts find challenging.The passage describes the choice of a 3000-bit p and a 256-bit q as an apparently unbalanced prime scheme.Source: 1) Difficulty 1: Cutting Edge Research · p. 6

2) Difficulty 2: Complex Computations:

  • Without external tools, many models struggle with precise calculations; Mixtral-8x7B-Instruct and GPT-3.5 may fail a binary XOR question, while GPT-4o handles it easily with embedded Python.Source: 2) Difficulty 2: Complex Computations · p. 6
  • Without external assistance, many models struggle with precise calculations such as converting subnet mask 255.255.248.0 to CIDR notation.The passage says complex models often answer such calculations incorrectly.Source: 2) Difficulty 2: Complex Computations · p. 6

VI. LIMITATIONS AND ETHICAL CONSIDERATIONS

CyberMetric-10000 may still contain incorrect validations or irrelevant questions. The study used publicly accessible cybersecurity documents and sought to exclude non-public content during validation.

A. Limitations and Threats to Validity

  • CyberMetric-10000 may still contain incorrectly validated or irrelevant questions, and the authors say they will announce updates and corrections on the project website.The passage says most questions have been validated by human experts.Source: A. Limitations and Threats to Validity · p. 6

B. Ethical Considerations

  • The study drew on publicly accessible security standards, research papers, NIST publications, BSI guidelines, and RFCs, and validators sought to remove or verify non-public content.RAG was used to generate questions from sources, with source verification during human validation.Source: B. Ethical Considerations · p. 6

VII. CONCLUSION

On CyberMetric-80, GPT-4o outperformed all human experts, while expert participants averaged 72% accuracy and top performers nearly matched leading models. GPT-4o and GPT-4 were the top proprietary models, and Mixtral-8x7B-Instruct and Falcon-180B tied among open-source models on CyberMetric-10000.

VII. CONCLUSION

  • GPT-4o outperformed all human experts on CyberMetric-80; expert humans averaged 72% accuracy and beginners were outperformed by 18 of the 25 tested models.The passage says top human performers nearly matched the highest-performing LLMs.Source: §VII · p. 6
  • GPT-4o and GPT-4 were the top proprietary models, while Mixtral-8x7B-Instruct and Falcon-180B tied for the best open-source scores on CyberMetric-10000.The passage also notes that most models remain limited in complex calculations and reasoning.Source: §VII · pp. 6–7
  • The findings concern cybersecurity knowledge questions and do not support general conclusions about machine versus human intelligence, intuition, or problem-solving.Source: §VII · p. 7

Related papers

Loading 2402.07688v2…