PaperlayerView exact revision on arXiv

Source-linked AI summary

When LLMs Meet Cybersecurity: A Systematic Literature Review

Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, Dan Meng

arXiv:2405.03644v2cs.CRcs.AI

TL;DR

Existing cybersecurity approaches face an evolving threat landscape, while research on LLM applications lacks a comprehensive overview. This paper systematically reviews LLM construction, cybersecurity task applications, and associated challenges, finding substantial potential alongside inherent security risks.

  • Problem

    Cybersecurity faces continuously adapting adversaries, while existing LLM research lacks a comprehensive overview of construction methods, applications, and challenges.

  • Method

    The paper conducts an extensive systematic review of cybersecurity-oriented LLM construction techniques and LLM usage across more than 10 cybersecurity tasks.

  • Results

    LLMs show potential across cybersecurity tasks including vulnerability detection, program repair, honeypots, and secure code generation, though detection can have high false-positive rates and security risks remain.

  • Takeaways & Limitations

    Applying LLMs in cybersecurity requires both task-specific construction and mitigation of vulnerabilities such as jailbreaking before deployment in sensitive environments.

  • Takeaways & Limitations

    LLM-agent deployment faces indirect prompt-injection risks when malicious instructions in web content or external-tool inputs manipulate agent actions.

Abstract

from arXiv · show

The rapid development of large language models (LLMs) has opened new avenues across various fields, including cybersecurity, which faces an evolving threat landscape and demand for innovative technologies. Despite initial explorations into the application of LLMs in cybersecurity, there is a lack of a comprehensive overview of this research area. This paper addresses this gap by providing a systematic literature review, covering the analysis of over 300 works, encompassing 25 LLMs and more than 10 downstream scenarios. Our comprehensive overview addresses three key research questions: the construction of cybersecurity-oriented LLMs, the application of LLMs to various cybersecurity tasks, the challenges and further research in this area. This study aims to shed light on the extensive potential of LLMs in enhancing cybersecurity practices and serve as a valuable resource for applying LLMs in this field. We also maintain and regularly update a list of practical guides on LLMs for cybersecurity at https://github.com/tmylla/Awesome-LLM4Cybersecurity.

1 Introduction

This introduction positions LLMs as promising cybersecurity tools amid evolving threats, while identifying a lack of comprehensive synthesis and defining three review questions.

  • Motivation: Evolving cyber threats challenge traditional signature-based and rule-based defenses, while LLMs offer promising support through domain models and advanced prompting.Examples include RepairLlama and Hackmentor, alongside ChatGPT-based prompt engineering, in-context learning, and chains-of-thought.
  • Research gap: Existing cybersecurity LLM studies often rely on case studies without comprehensive methodology, limiting connectivity, scalability, and reproducibility across the field.The review argues that a systematic overview is needed as research rapidly expands.
  • Research questions: The survey addresses how to construct cybersecurity-oriented LLMs, where LLMs can be applied, and which challenges and research directions remain.The questions cover model construction, cybersecurity applications, and future research.
  • Scope: The reviewed application scope includes vulnerability detection, secure code generation, program repair, binary analysis, IT operations, threat intelligence, anomaly detection, and LLM-assisted attacks.These applications are organized among the main cybersecurity tasks surveyed in the paper.
  • Review contribution: The review summarizes construction principles, techniques, training data, and specialized domain models, while analyzing LLM strengths across cybersecurity tasks and identifying deployment challenges.It also proposes future research directions for applying LLMs in cybersecurity.

2 Preliminary

The preliminary section frames the review as a systematic examination of LLM use in cybersecurity, spanning model types and diverse security tasks.

  • Review scope: The review covers more than 300 academic papers since 2023 to characterize LLM progress, current applications, challenges, and future directions in cybersecurity.The surveyed literature is summarized by the LLMs used and the cybersecurity task categories addressed.
  • LLM categories: Open-source LLMs expose model weights for cybersecurity fine-tuning and customization, whereas closed-source models may offer greater performance and scale.Open-source adaptability is particularly relevant to private data and customized cybersecurity needs.
  • LLM categories: Code-based LLMs are designed for programming languages and are suited to cybersecurity needs involving code understanding, analysis, and secure code generation.Examples include CodeLlama and StarCoder.
  • Application categories: The surveyed cybersecurity applications include threat intelligence, vulnerability detection, malware detection, anomaly detection, fuzzing, program repair, and LLM-assisted attacks.These categories address both defensive analysis and attack-oriented uses of LLMs.
  • Application categories: Secure and insecure code generation is treated as a cybersecurity question concerning risks in generated code and strategies for correcting it.The category explicitly asks whether generated code is risky and whether LLMs can repair it.

3 RQ1: How to construct cybersecurity-oriented domain LLMs?

The review finds that cybersecurity-oriented LLMs are generally built by adapting general-purpose models with cybersecurity data, while specialized architectures and independent pretraining offer additional routes. Model selection and training choices depend on cybersecurity capability, task requirements, available resources, and desired performance.

  • Base-model selection: Cybersecurity capability evaluations cover domain knowledge, secure code generation, and IT operations.These evaluations help identify suitable base models and reveal both general cybersecurity capabilities and remaining limitations.
  • Training methods: CPT further trains a pretrained model on large unlabeled cybersecurity data, while SFT uses human-labeled prompt-response data for task-specific optimization.SFT refines model weights using a task-specific loss and is also called instruction fine-tuning.
  • Implementation pathways: Full-parameter fine-tuning updates all model parameters but requires considerable computing power and time, whereas PEFT freezes most parameters and reduces computational costs.PEFT can produce tiny checkpoints while achieving comparable performance to full fine-tuning.
  • Specialized models: Specialized security-oriented LLMs can also be independently pretrained using innovative architectures and proprietary large-scale datasets.Examples include multimodal machine-language modeling and traffic-specific pretraining designed to improve security-related representations and detection.
  • Construction approaches: General-purpose LLMs can be adapted into cybersecurity domain models using continual pre-training or supervised fine-tuning.CPT uses unlabeled domain data for broader domain understanding, whereas SFT uses labeled data to optimize specific cybersecurity tasks.

4 RQ2: What are the potential applications of LLMs in cybersecurity?

The review organizes cybersecurity applications of LLMs across defense, software testing, attack assistance, code generation and analysis, and other security-related tasks. These applications span offline and online defense settings as well as broader security workflows.

  • Application scope: LLMs are applied to offline defense, online defense, software testing, attack assistance, code generation and analysis, and other security-related applications.Examples include threat intelligence, vulnerability and malware detection, anomaly detection, fuzzing, program repair, and honeypots.

4.1 Threat Intelligence

Threat-intelligence studies use LLMs to extract, structure, summarize, assess, and reason over cyber-threat information. Approaches combine prompting, fine-tuning, retrieval, and knowledge graphs to improve usefulness and reliability.

  • Extraction and structuring: LLMs extract and summarize cyber-threat intelligence from unstructured reports and knowledge sources.Frameworks structure reports using STIX, summarize global and organization-specific databases, and construct queryable attack knowledge graphs.
  • Attack description: Fine-tuning on ATT&CK and CAPEC information is compared with directly using GPT-3.5 for generating cyberattack descriptions.This work examines whether domain-specific training improves attack-description generation.
  • Threat reasoning: Retrieval-augmented generation improves LLM explanations of MITRE ATT&CK tactics, techniques, and procedures by supplying relevant context.The study compares encoder-only and decoder-only models and enhances decoder-only models without fine-tuning.
  • Operational assistance: LLMs are also used for forensic-report generation, bug-report deduplication, vulnerability mitigation suggestions, and proactive security strategy support.Crimson combines real-time retrieval and domain-specific fine-tuning to reduce errors and hallucinations in cybersecurity strategies.
  • Quality assessment: Knowledge graphs combined with LLMs significantly improve threat-intelligence quality assessment through automated claim extraction and fact-checking.The framework verifies extracted OSCTI claims against a paragraph-based knowledge graph.

4.2 Vulnerability Detection

Vulnerability-detection research generally finds LLMs promising, but performance varies by model, dataset, prompting, and task complexity. Researchers therefore augment code with structural or retrieved context, while evaluations continue to expose false positives and weaknesses on nuanced real-world tasks.

  • Evidence of capability: LLMs are generally promising for vulnerability detection, but reported results vary across datasets and studies.Some evaluations find poor performance or high false-positive rates, while fine-tuned models can outperform static-analysis and deep-learning tools; synthetic-data performance may degrade on real-world datasets.
  • Context augmentation: Researchers improve vulnerability reasoning by preprocessing code or supplying structural information such as ASTs, data-flow graphs, control-flow graphs, API sequences, and diagrams.These strategies address the limitation that directly providing code may be insufficient for vulnerability analysis.
  • Advanced strategies: Retrieval-augmented and adversarial-agent frameworks provide additional knowledge or verification stages for vulnerability detection.Examples include mutation, injection, and extension strategies, knowledge-level Vul-RAG, and GPTLENS auditor–critic roles.
  • Reasoning control: LLM4Vuln separates vulnerability reasoning from other capabilities and lets the model request additional contextual information about target code.Its findings indicate that more input is not necessarily better for vulnerability reasoning.
  • Vulnerability management: ChatGPT can assist with security-bug prediction, severity evaluation, vulnerability repair, and patch verification, but needs improvement for prioritization and patch validation.The study specifically identifies nuanced vulnerability-management tasks as remaining weaknesses.
  • Datasets and limitations: DiverseVul contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable C/C++ functions.Experiments across 11 deep-learning architectures still report high false-positive rates, low F1 scores, and difficulty with complex CWEs.

4.3 Malware Detection

LLMs support malware detection through static code analysis, dynamic debugging, and malware summarization. The surveyed methods use decompiled code, API behavior, iterative prompting, and dedicated datasets to assist analysis.

  • Static analysis: LLMs analyze decompiled C code from reverse-engineering tools such as Ghidra to understand software functionality and extract information.These systems act as static analysis assistants for reverse engineering.
  • Static analysis: Fuzz testing exposes diverse program behaviors and inputs, helping LLMs improve their semantic understanding of programs.The approach targets functions and subroutines whose behavior varies across inputs.
  • Dynamic debugging: GPT-4 can generate explanatory text for API calls and use BERT-generated API sequences to support dynamic malware analysis.The approach aims to represent API-call behavior without training the dataset during generation.
  • Dynamic debugging: SocketAI Scanner identifies malicious intent in JavaScript packages using self-refinement, zero-shot role-playing, and chain-of-thought prompting.Compared with static analysis tools, it achieves higher precision and lower false positive rates.
  • Malware summarization: MALSIGHT iteratively summarizes binary malware by exploring malicious source code and benign pseudocode, alongside the MalS and MalP datasets.The datasets support further research on human-readable descriptions of malware behavior.

4.4 Anomaly Detection

The surveyed anomaly-detection applications span logs, web content, URLs, network traffic, DDoS attacks, and digital forensics. LLMs are adapted through fine-tuning, distillation, specialized encoding, and task-specific use.

  • Log-based anomaly detection: 60 fine-tuned language models, including BERT, RoBERTa, GPT-2, and GPT-Neo, are evaluated for log analysis and domain adaptation.The findings emphasize effectiveness for specific log types.
  • Web content security: DistilBERT- and RoBERTa-based models are applied to phishing and spam detection, while GPT-4 is also evaluated for phishing-email generation.GPT, Claude, PaLM, and LLaMA are reported to identify malicious intent, sometimes surpassing human detection rates.
  • Threat detection: Knowledge distillation assigns labels to unlabeled URLs through a teacher model, allowing a smaller student model to improve accuracy with fewer parameters.The resulting model is positioned for malicious-URL detection.
  • Threat detection: LLMs are fine-tuned for DDoS and network-traffic classification on CICIDS 2017, Urban IoT, and privacy-preserving encoded traffic.SecurityBERT uses Privacy-Preserving Fixed-Length Encoding for network-traffic classification.
  • Digital forensics: ChatGPT is assessed for digital-forensic tasks including downloaded-file and execution checks, browser-history analysis, event-log analysis, and cloud-machine interactions.The surveyed use covers multiple forensic evidence sources.

4.5 Fuzz

LLMs shift fuzzing toward guided generation of test inputs and can outperform traditional methods when supplied with vulnerability descriptions, exploit information, and code context. Strategies vary by testing target, including APIs, deep-learning libraries, and protocols.

  • Advantages over traditional fuzzing: LLMs outperform SIEGE and TRANSFER in generating test cases when given detailed vulnerability descriptions, possible exploits, and code context.The comparison evaluates ChatGPT without tuning against two traditional testing tools.
  • Advantages over traditional fuzzing: LLM-based fuzzing shifts seed generation from random mutation toward guided mutation through GPT-based seed variants.A GPT-based seed mutator selects seeds and requests higher-quality input variants from ChatGPT.
  • Target-specific strategies: For general APIs, LLMs can automatically generate many effective fuzzing drivers while reducing human intervention.The approach is compared with traditional program-analysis methods.
  • Target-specific strategies: TitanFuzz targets deep-learning libraries with training data containing code snippets that call their APIs.The strategy accounts for strict tensor-dimension requirements that can otherwise produce meaningless tests.
  • Target-specific strategies: Protocol fuzzing can use LLMs trained on human-readable protocol documents to mutate interactive messages without machine-readable specifications.HTTP is given as an example protocol target.

4.6 Program Repairing

LLM-based program repair is evaluated across benchmarks and real-world vulnerabilities, with results showing strong promise on some cases but difficulty on complex examples. Researchers combine LLMs with analysis feedback, prompting strategies, patch ranking, and fine-tuning.

  • Evaluation of existing LLMs: Codex outperforms numerous existing automatic program-repair methods on QuixBugs without retraining.QuixBugs contains 40 Python and Java bugs; other studies evaluate ChatGPT and Gemini on repair tasks.
  • Evaluation of existing LLMs: LLMs show promise for simple vulnerability repairs but struggle with more complex real-world examples.The evidence spans synthetic, artifactual, and real-world security scenarios.
  • Tool-assisted repair: Feedback-Driven Security Patching passes Bandit feedback to an LLM to generate potential solutions for security vulnerabilities.Static analysis feedback is used to support patch generation.
  • Repair strategies: Prompting and reasoning strategies include whole-program refinement, self-debugging, self-consistency, chain-of-thought reasoning, and patch-validation feedback.These methods generate or select repair candidates using execution results, reasoning paths, or validation signals.
  • Repair strategies: Entropy-delta patch ranking and classification can rank correct patches more effectively than state-of-the-art machine-learning tools.The method combines LLM entropy with prior automatic program-repair tools.
  • Fine-tuning: MORepair uses multi-objective fine-tuning to model code-transformation syntax and the logical reasoning behind code changes.The paper presents fine-tuning as necessary for state-of-the-art program-repair performance.
  • Target-specific repair: ZeroLeak generates patches for side-channel vulnerabilities through zero-shot learning and checks them with dynamic analysis tools.The checks assess both functional correctness and information-leakage prevention.

4.7 LLM Assisted Attack

LLMs support both automated cyberattacks and offensive assistance, spanning penetration testing, phishing, privilege escalation, payload generation, and attack-graph construction. The reviewed studies show useful but incomplete capabilities, including executable and more targeted payloads and partial help with CTF challenges.

  • Automated penetration testing: LLMs automate penetration testing by supporting inference, generation, parsing, task planning, and execution workflows.PentestGPT uses three modules reflecting penetration-testing roles, while related work studies high-level planning and automated testing.
  • Attack construction: LLMs generate multi-stage attacks, attack graphs, phishing websites and emails, privilege-escalation steps, and executable attack commands.These systems can chain CVEs, clone websites, capture credentials, register domains, deploy scripts, and construct complex attack sequences.
  • Payload generation: LLM-generated payloads efficiently produce executable code for the top 10 MITRE weaknesses observed in 2022 and tend to be more complex and targeted than manually crafted payloads.The comparison covers ChatGPT and Bard-generated payloads against manually crafted payloads.
  • CTF challenges: LLMs help with CTF challenges to a certain extent but do not solve them comprehensively.The evaluation included GPT-3.5, PaLM2, and Prometheus across representative challenges from common CTF categories.
  • Attack proxies: LLMs can serve as attack proxies, allowing remote victim control without direct attacker-to-victim communication and making attribution more difficult.ChatGPT is described as an intermediary between the victim and attacker-controlled command-and-control infrastructure.

4.8 (In)secure Code Generation

The reviewed work examines both the security of LLM-generated code and methods for improving it. Models may produce unsafe code unless explicitly prompted otherwise, while security-focused fine-tuning and analyzer integration can improve secure generation.

  • Generated-code security: Current models frequently overlook security issues during code generation and repair, leading to vulnerable code.CodeSecEval evaluates 44 vulnerability types using 180 curated samples.
  • Evaluation methods: Security evaluation frameworks combine prompt datasets, code-generation environments, execution, and systematic vulnerability assessment.SALLM uses Docker to execute generated code, while other work designs broad execution scenarios and curated benchmarks.
  • Generated-code security: LLMs may identify vulnerabilities during prompted code review yet still generate unsafe code unless explicitly instructed otherwise.The studies also identify deep-neural-network uninterpretability as a challenge to understanding this behavior.
  • Security improvements: Security-specific fine-tuning and instruction tuning can enhance generated-code security without compromising utility across various tasks.SafeCoder combines traditional instruction tuning with security-specific fine-tuning using an automatically collected GitHub dataset.

4.9 Others

Beyond the main cybersecurity tasks, LLMs are studied for incident response, network management, vulnerability reproduction, security analysis, honeypots, and domain expertise support. These applications show operational promise but also expose reliability concerns such as misleading or excessive answers.

  • Other applications: LLM applications include IoT fingerprint generation, security-patch identification, SoC security verification, binary taint analysis, and input-output risk detection.The reviewed methods use embeddings, synthetic patch data, security evaluations, dependency reasoning, and instruction-tuned safeguards.
  • Honeypots: Dynamic LLM-generated honeypots aim to reduce recognizability, and related work demonstrates broad honeytoken generation across 210 prompt structures.LLMPot extends the approach to realistic, vendor-agnostic ICS honeypots.
  • Incident response: LLMs can draft incident-response plans, recommend practices, and identify documentation gaps, with human oversight needed for accuracy and relevance.The cited work frames LLMs as tools for streamlining incident-response planning rather than replacing review.
  • Network management: NeMoEval benchmarks natural-language generation of task-specific code for network traffic analysis and network lifecycle management.The benchmark covers two network-management applications.
  • Vulnerability reproduction: AdbGPT automatically reproduces vulnerabilities from vulnerability reports through prompting without training or hard coding.The approach targets automated vulnerability reproduction.
  • Cybersecurity expertise Q&A: LLM-generated answers to cybersecurity expertise questions can contain fake information and excessive length, making misleading information difficult to recognize.Some testers nevertheless valued the answers’ comprehensiveness and language style.

5 RQ3: What are the challenge and further research for the application of LLMs in cybersecurity?

The review identifies vulnerabilities in LLMs and agent systems, especially jailbreaks, prompt injection, and backdoors, while proposing expanded tool use and autonomous agents as future directions. Agents show promise for both attacks and defense, but their security risks remain unresolved.

  • Challenges: LLMs are vulnerable to backdoor and prompt-injection attacks that manipulate outputs or compel actions aligned with attackers’ intentions.The review discusses triggers embedded in models or inputs and malicious commands inserted into inputs or external content.
  • Challenges: Jailbreak prompts can induce harmful content despite model safeguards, demonstrating persistent vulnerabilities across violation categories.One study analyzed 6,387 jailbreak prompts to characterize attack strategies.
  • Challenges: Simple data tuning can manipulate securely aligned LLMs into harmful content, while LLM-integrated applications face data pollution, database-control, data-leakage, and reproducibility risks.The review also emphasizes the dual-edged nature of generative AI and the risks associated with closed-source models.
  • Further research: GPT-4-based LLM agents achieve up to 73.3% success in hacking attempts and can autonomously discover vulnerabilities in real-world websites.Related studies report autonomous exploitation of zero-day vulnerabilities and coordinated exploitation by teams of agents.
  • Further research: LLM agents can support cyber defense by interpreting threats, executing instructed security actions, generating mitigation plans, and accelerating CTI analysis.Nissist, Cyber Sentinel, PhishAgent, and a CTI-analysis agent illustrate these defensive uses.
  • Further research: Tool use, API calls, and autonomous agents are identified as directions for advancing AI utilization in cybersecurity.The proposed direction centers on agents that understand, plan, and execute complex cybersecurity tasks.

6 Conclusion

The paper identifies broad potential for LLMs across cybersecurity tasks while emphasizing deployment risks such as jailbreaking. It lays groundwork for future cybersecurity strategies through expanded tool use, API calls, and autonomous agents.

  • LLMs show great potential across cybersecurity tasks, including threat intelligence, vulnerability detection, and secure code generation.
  • Jailbreaking exposes inherent LLM vulnerabilities that create significant security risks in sensitive cybersecurity environments.
  • Future research should extend LLM tool-use and API-call capabilities and develop autonomous intelligent agents for complex cybersecurity operations.
  • The review lays groundwork for researchers and practitioners to harness LLMs while addressing challenges posed by increasingly complex cyber threats.
Loading 2405.03644v2…