PaperlayerView exact revision on arXiv

Source-linked AI summary

ORCAGen: Orchestrating Context-Aware Malware Deception with RAG-Guided Generative AI

Shihab Ahmed, Md Sajidul Islam Sajid, Teryl Taylor, Frederico Araujo, Tariqul Islam

arXiv:2610.12415v1cs.CR

TL;DR

Malware containment can cost defenders visibility into attacker behavior, while static deception strategies are difficult to adapt at scale. ORCAGen uses RAG and structured prompting to generate and validate malware-specific playbooks offline; its evaluated comparisons show better deception generation than direct and RAG-enhanced prompting and SPADE.

  • Problem

    Rapid malware removal can limit visibility into attacker behavior, while manually configured deception strategies are difficult to scale as malware changes.

    Source: §I · p. 1

  • Method

    ORCAGen uses a curated malware-and-defense knowledge base, RAG, and structured prompts to generate PoC malware and matching deception logic, validating it before runtime deployment.

    Source: §I · pp. 1–2, B. Knowledge Base Creation for RAG · p. 4

  • Results

    ORCAGen achieved 100% zero-shot deception success, improved over direct and RAG-enhanced prompting, and outperformed SPADE across evaluated models and refinement thresholds.

    Source: C. Ablation and Baseline Comparison · p. 9, §VII · p. 10

  • Takeaways & Limitations

    The framework enforces only pre-validated deception logic without live LLM inference, with GPT-5.5 requiring the fewest refinements and Gemini 3.5 Flash having the lowest response time and runtime overhead.

    Source: §VII · p. 10

    Output quality depends on knowledge-base coverage and model reliability; the prototype focuses on user-space malware and controlled virtual environments.

    Source: §VII · p. 10

Abstract

from arXiv · show

Malware defenses often remove or isolate suspicious programs as quickly as possible. While effective for containment, this approach can also waste an opportunity to observe attacker behavior and deploy targeted countermeasures. ORCAGen takes a different approach: it uses GenAI to build malware-specific deception playbooks offline, validates them before deployment, and enforces only the verified logic at runtime. ORCAGen combines Retrieval-Augmented Generation (RAG) with structured prompt engineering to generate both proof-of-concept (PoC) malware and corresponding deception orchestration code. A curated knowledge base (KB) of malware procedures and active defense strategies grounds the generation process, helping the LLM produce threat-specific and executable deception logic rather than generic or hallucinated outputs. The generated PoC malware provides a safe and reproducible way to test whether a deception strategy can disrupt, redirect, or suppress targeted malware behavior before the strategy is added to the runtime playbook. We evaluate ORCAGen across GPT-4o, GPT-5.5, Gemini 3.5 Flash, Qwen3-Coder, and Claude Sonnet 4.5 using execution success, hallucination rate, refinement effort, deception effectiveness, and runtime overhead. The evaluation covers synthesized malware scenarios and 150 real-world malware samples across keyloggers, information stealers, and ransomware. Across the evaluated scenarios, GPT-5.5 required the fewest refinements and produced no observed hallucinated APIs, while Gemini 3.5 Flash achieved the lowest response time and runtime overhead. The results show that RAG-guided structured prompting can support the scalable construction of malware-specific deception playbooks that remain lightweight and deterministic during runtime enforcement.

I. INTRODUCTION

ORCAGen addresses the tension between rapid malware containment and preserving visibility into attacker behavior. It generates malware-specific deception playbooks offline, tests them against PoC malware, and enforces only verified logic at runtime.

I. INTRODUCTION

  • Immediate malware removal can erase visibility into attacker behavior, while static deception rules are difficult to scale as malware changes.The paper identifies a need for threat-specific responses that can be validated before runtime deployment.Source: §I · p. 1
  • ORCAGen uses offline RAG and structured prompting to generate PoC malware alongside corresponding deception orchestration code.The generated logic is validated against the PoC before only verified code is added to a deployable playbook.Source: §I · p. 1
  • A curated knowledge base maps malware procedures to active defenses, grounding generation in target behavior rather than generic outputs.RAG retrieves relevant procedures and strategies for structured prompts.Source: §I · p. 2
  • The workflow creates PoC malware, generates matching deception logic, and tests it in a controlled environment before adding it to the runtime playbook.Runtime enforcement uses validated logic rather than live LLM inference.Source: §I · pp. 1–2
  • The evaluation compares multiple LLMs using execution success, hallucination rate, refinement effort, deception effectiveness, and runtime overhead.The paper evaluates ORCAGen across multiple LLMs.Source: §I · p. 2

II. BACKGROUND

The background motivates using retrieval and structured prompts to control security-oriented generation, then frames ORCAGen as offline construction and runtime enforcement for a bounded endpoint threat model.

A. Generative AI in Security Contexts

  • Structured prompts and RAG address incomplete or hallucinated model outputs by specifying task constraints and supplying external domain knowledge without fine-tuning.ORCAGen uses RAG with structured prompting to ground generation in malware procedures and active defense strategies.Source: A. Generative AI in Security Contexts · p. 2

B. Deception-Oriented System Orchestration

  • ORCAGen builds malware-specific deception logic offline, validates it in a controlled setting, and deploys verified playbooks for deterministic runtime enforcement.This middle path contrasts static playbooks with fully adaptive runtime logic, which may introduce latency, unpredictability, or operational risk.Source: B. Deception-Oriented System… · p. 2

III. THREAT MODEL

  • The threat model assumes externally identified suspicious processes and endpoint control, focusing on user-space malware interactions with monitored user-level resources.Kernel-level behavior and activity outside monitored user-space interactions are out of scope; ORCAGen does not replace detection, access control, backups, or removal.Source: §III · pp. 2–3

IV. ORCAGEN: SYSTEM OVERVIEW

ORCAGen separates offline playbook construction from runtime enforcement and grounds generation in structured malware-procedure and active-defense knowledge bases. Its KB workflow combines search, manual review, and entries that pair behavioral detail with defensive responses.

A. System Architecture

  • ORCAGen generates malware-specific deception logic offline, validates its interaction with PoC malware, and enforces only pre-tested responses at runtime.The runtime module injects a validated Super DLL into suspicious or policy-selected processes without querying an LLM.Source: A. System Architecture · p. 3

B. Knowledge Base Creation for RAG

  • Two knowledge bases separately capture malware procedures and map those behaviors to deception strategies.The target families are keyloggers, information stealers, and ransomware, whose behaviors expose user-level interactions.Source: B. Knowledge Base Creation for RAG · p. 3
  • Figure 2’s KB collection process combines lexical search, semantic search, and manual validation, using curated seeds from malware families, ATT&CK techniques, and associated APIs or artifacts.Candidate sources include academic papers, threat-intelligence reports, technical documentation, and malware blogs; sources found through both retrieval modes were prioritized for review.Source: B. Knowledge Base Creation for RAG · pp. 3–4, Figure 2 · p. 4
  • Manually filtered documents become structured entries pairing malware family, attack technique, behavioral description, and deception strategy.Procedure-level entries capture APIs, artifacts, and execution patterns; response-level entries capture intervention objectives, triggers, safety constraints, and expected responses.Source: B. Knowledge Base Creation for RAG · p. 4

C. RAG Implementation

ORCAGen retrieves task-relevant malware and defense knowledge to ground offline generation, then uses structured prompts to produce PoCs and orchestration code. Its prompts scaffold the project, separate generation into two stages, and support targeted refinement.

C. RAG Implementation

  • RAG retrieves relevant malware procedures and active-defense entries as structured records, then supplies task-specific context for PoC and orchestration generation.The prototype KB contains 8 malware procedure entries and 19 active-defense entries covering 3 malware families, 8 techniques, and 28 API-level or file-system indicators.Source: C. RAG Implementation · p. 4

D. Prompt Engineering

  • An initial project-level scaffold specifies a PoC, DLL-based API hooking module, and injector, constraining component interfaces and execution flow.The scaffold is intended to prevent incomplete projects, mismatched signatures, incompatible encoding, and untestable fragments.Source: D. Prompt Engineering · p. 5
  • Each prompt combines an objective, output format, retrieved or user-provided context, and examples to guide compatible, executable code generation.The output format specifies constraints such as language, libraries, function definitions, and file structure.Source: D. Prompt Engineering · p. 5
  • Generation proceeds in two stages: malware-procedure context guides PoC creation, then the PoC and retrieved defense entries guide orchestration-code generation.This keeps emulation and defense generation distinct while letting the orchestration prompt reason over the behavior exposed by the PoC.Source: D. Prompt Engineering · p. 5
  • Targeted refinement prompts use compile errors or runtime observations to correct failed outputs without human source-code edits, stopping at success or the refinement limit.The number of refinements is recorded as an evaluation metric.Source: D. Prompt Engineering · p. 5

E. Code Generation

ORCAGen generates controlled PoC malware and corresponding deception orchestration code offline. It compiles and validates these artifacts before eligible orchestration logic can enter the playbook.

E. Code Generation

  • ORCAGen generates both controlled PoC malware and corresponding deception orchestration code offline for constructing a deception playbook.The PoC is intended to reproduce target behavior safely and reproducibly for validation, not to create deployable malware.Source: E. Code Generation · p. 5
  • For PoC generation, retrieved procedure entries guide code that emulates behaviors such as keyboard capture, credential-file access, file scanning, or ransomware-style modification.Generated PoCs are compiled and executed in a controlled environment; those that fail to compile or reproduce the behavior are refined or discarded.Source: E. Code Generation · p. 5
  • For orchestration generation, the model combines the PoC with retrieved defense context to produce intervention code, and only compiled, reviewed, validated variants are eligible for the playbook.Possible interventions include API hooks, file-system redirection, decoy credential redirection, and simulated success or failure responses.Source: E. Code Generation · pp. 5–6

F. Offline Validation and Runtime Enforcement

ORCAGen validates generated deception logic against controlled PoC behavior before adding passing strategies to the runtime playbook. Runtime enforcement uses validated logic in flagged or policy-selected processes, without live LLM inference.

F. Offline Validation and Runtime Enforcement

  • Validation executes the PoC, then tests whether orchestration redirects, delays, suppresses, or misleads its behavior without disrupting benign activity in the same environment.ORCAGen records traces, forensic logs, behavioral changes, false positives, and false negatives to assess reliability.Source: F. Offline Validation and Runtime… · p. 6
  • Only strategies that pass validation enter the reusable Super DLL playbook, which is injected at runtime into a process already flagged or selected by policy.Trust status and behavior-specific triggers gate responses so benign or whitelisted processes can proceed normally.Source: F. Offline Validation and Runtime… · p. 6
  • Runtime evaluation measures the playbook’s behavior, not live LLM inference, because generation, refinement, and validation are completed offline.The Super DLL monitors covered API calls and behaviors and triggers the corresponding validated response when a match occurs.Source: F. Offline Validation and Runtime… · p. 6

V. EVALUATION

  • The evaluation used five LLM backends and an isolated Windows 7 virtual machine for code compilation and testing.The Windows 11 Enterprise host had an Intel Core i7-13700 CPU, 32 GB RAM, and an NVIDIA GeForce RTX 3070 GPU; the VM had 8 GB RAM.Source: §V · p. 6

A. Functionality Validation

Functionality varied by model and task: GPT-5.5 and GPT-4o showed the strongest overall PoC reliability, while GPT-5.5 and Gemini 3.5 Flash led orchestration execution. These differences support offline validation before generated logic enters a runtime playbook.

A. Functionality Validation

  • The evaluation measured PoC execution as compiling, running, and reproducing expected behavior, and orchestration execution as compiling, running, and intervening against the PoC.It covered fifteen scenarios across keyloggers, information stealers, and ransomware.Source: A. Functionality Validation · pp. 6–7
  • GPT-5.5 and GPT-4o achieved 93.33% zero-shot PoC execution and only one minor correction, while GPT-5.5 and Gemini 3.5 Flash reached 100% zero-shot orchestration execution across categories.Overall, the results show that reliability differs substantially across models and tasks, reinforcing the need for offline validation before playbook inclusion.Source: A. Functionality Validation · p. 7
  • Gemini 3.5 Flash had 100% zero-shot PoC success for keyloggers but 60% for information stealers and ransomware; Qwen3-Coder reached 60%, 100%, and 80%, respectively.Claude Sonnet 4.5 did not generate PoC malware because of content safety restrictions and was evaluated only for orchestration using GPT-5.5-generated PoCs.Source: A. Functionality Validation · p. 7
  • GPT-4o achieved 66.67% zero-shot orchestration execution overall; Qwen3-Coder reached 40% for keyloggers and information stealers and needed additional prompting for every ransomware case.Claude Sonnet 4.5 achieved 80% zero-shot orchestration execution for keyloggers and information stealers, but 20% for ransomware.Source: A. Functionality Validation · p. 7
  • GPT-5.5 produced one compatibility-related correction and no hallucinated APIs, while Qwen3-Coder had the most errors, including four hallucinated API or data references.Other backends mostly produced minor implementation errors such as missing headers, encoding mismatches, type-casting issues, or C++ compatibility problems.Source: A. Functionality Validation · p. 7

1) Against Synthesized Malware Samples:

ORCAGen was evaluated first against synthesized malware scenarios and then against real-world malware samples. Results show that validated playbooks could disrupt the tested behaviors, with success and limits varying by model and malware behavior.

1) Against Synthesized Malware Samples:

  • The synthesized evaluation tested 15 scenarios across three malware families, measuring deception success, refinements, and policy-related false positives and false negatives.Each scenario allowed up to three prompt refinements and was run in a controlled virtual environment.Source: 1) Against Synthesized Malware Samples · p. 7
  • 100% zero-shot deception success came from GPT-5.5 and Gemini 3.5 Flash across the synthesized scenarios.GPT-4o, Claude Sonnet 4.5, and Qwen3-Coder reached 66.67% (10/15), 60% (9/15), and 26.67% (4/15), respectively.Source: 1) Against Synthesized Malware Samples · pp. 7–8
  • After refinement, all models reached 100% success; GPT-4o and Claude Sonnet 4.5 averaged two refinements, while Qwen3-Coder required three.No false positives or false negatives were observed after validation in the synthesized experiments.Source: 1) Against Synthesized Malware Samples · pp. 7–8

2) Against Real-World Malware Samples:

  • Playbooks redirected information stealers to honey credentials, baited ransomware while preserving originals, and replaced or redirected keylogger captures for suspicious processes.The real-world evaluation covered 150 samples, with 50 each from keyloggers, information stealers, and ransomware.Source: 2) Against Real-World Malware Samples · p. 8
  • 92% of keyloggers, 100% of information stealers, and 96% of ransomware samples were completely neutralized; keylogger failures involved direct keyboard-state polling.One ransomware sample was classified as partial disruption because its command-and-control server was unavailable during testing.Source: 2) Against Real-World Malware Samples · pp. 8–9

C. Ablation and Baseline Comparison

The ablation and baseline comparisons test whether ORCAGen’s performance reflects more than repeated prompting. ORCAGen outperformed direct and RAG-enhanced prompting, as well as SPADE, in the reported comparisons.

C. Ablation and Baseline Comparison

  • The prompting ablation compared direct prompting, RAG-enhanced prompting, and ORCAGen using GPT-5.5 across 15 scenarios, with up to five refinements per scenario.ORCAGen combines retrieved defense knowledge with structured prompts and a validation-oriented project scaffold.Source: C. Ablation and Baseline Comparison · p. 9
  • ORCAGen achieved 100% deception success zero-shot, while RAG-enhanced prompting reached 73.33% after two refinements and direct prompting reached 46.67% after five.Direct prompting failed in zero-shot execution; RAG-enhanced prompting did not improve beyond 73.33%.Source: Figure 6 · p. 9, C. Ablation and Baseline Comparison · p. 9
  • ORCAGen outperformed SPADE across all evaluated models and refinement thresholds.For GPT-4o, ORCAGen reached 100% after five refinements while SPADE reached 7.14% after five and 50% after ten; SPADE had no successful deception for Gemini 2.5 Flash.Source: Figure 7 · p. 9, C. Ablation and Baseline Comparison · p. 9

D. Performance Measurement

Performance measurement compares generation reliability and runtime cost across five models. GPT-5.5 required the fewest refinements, while Gemini 3.5 Flash had the fastest response time and lowest runtime overhead.

D. Performance Measurement

  • The evaluation measured refinements, hallucination rate, runtime overhead, and response time across GPT-4o, GPT-5.5, Gemini 3.5 Flash, Qwen3-Coder, and Claude Sonnet 4.5.Runtime overhead was averaged across ten independent trials after injecting the validated orchestration DLL into a target process.Source: D. Performance Measurement · pp. 9–10
  • GPT-5.5 required 0.07 refinements per task and had no observed hallucinations; Gemini 3.5 Flash had a 34.92s response time and 18.32ms runtime overhead.GPT-5.5’s runtime overhead was 27.69ms, and its response time was 54.21s.Source: Table III · p. 10, D. Performance Measurement · p. 10
  • Qwen3-Coder required 1.73 refinements, had a 0.27 hallucination rate, and introduced 183.56ms runtime overhead.The authors report these as the weakest reliability and highest overhead among the evaluated models.Source: Table III · p. 10, D. Performance Measurement · p. 10

VI. RELATED WORK

Related work spans static decoys, adaptive deception, and GenAI-generated artifacts, while ORCAGen focuses on malware-specific orchestration validated against behavior-specific PoCs. The conclusion reports performance gains and notes boundaries in model, threat, and evaluation scope.

VI. RELATED WORK

  • Prior cyber-deception systems often focus on static decoys or deception selection rather than executable, malware-specific intervention logic.GenAI deception systems also commonly generate artifacts or interactive responses rather than logic tested against executable adversarial behavior.Source: §VI · p. 10
  • ORCAGen uses RAG and structured prompting to generate malware-specific orchestration code, validate it offline against behavior-specific PoCs, and enforce prevalidated logic without live LLM inference.The paper distinguishes this offline construction and deterministic runtime enforcement from LLM-based deception against attack agents.Source: §VI · p. 10

VII. DISCUSSION AND CONCLUSION

  • The conclusion reports that GPT-5.5 required the fewest refinements with no observed hallucinated APIs, while Gemini 3.5 Flash had the fastest response time and lowest runtime overhead.The ablation favored ORCAGen over direct and RAG-enhanced prompting; real-world transfer was reported for samples interacting with monitored targets.Source: §VII · p. 10
  • Output quality depends on KB coverage and model reliability; the prototype targets user-space malware, and controlled virtual environments do not capture all adversarial adaptation or enterprise complexity.The authors identify kernel-level, firmware-level, and hardware-assisted threats as outside the current prototype’s focus.Source: §VII · p. 10

APPENDIX A REPRESENTATIVE KNOWLEDGE BASE ENTRIES

The appendix describes the fields represented in ORCAGen’s knowledge-base entries and the collection, screening, and extraction of the documents used to build the corpus.

APPENDIX A REPRESENTATIVE KNOWLEDGE BASE ENTRIES

  • Each representative knowledge-base entry records a malware family, attack technique, behavioral description, and corresponding deception strategy.Source: Appendix A · p. 12

APPENDIX B KNOWLEDGE BASE SOURCE DISTRIBUTION

  • Table V summarizes the distribution of collected and retained documents.Source: Table V · p. 12
  • The document collection draws on academic articles and industry-oriented sources, including threat-intelligence reports, malware-analysis blogs, and technical documentation.Named sources include Mandiant, Malwarebytes, SANS, Microsoft, Android, and MITRE technical documentation.Source: Appendix B · p. 12
  • After duplicate removal, documents were screened for technical relevance to malware behaviors or API usage patterns, then retained for structured knowledge extraction.Source: Appendix B · p. 12
Loading 2610.12415v1…