KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

1 Introduction

• KaliBench targets a gap between knowledge-based and end-to-end agentic evaluations: neither directly measures whether models can generate correct invocations of concrete cybersecurity tools.The benchmark focuses on a central challenge for cybersecurity workflows: translating high-level analyst goals into precise commands in unforgiving CLIs.Source: §1 · p. 2
• A multi-stage verification pipeline combines LLM-based validation, sandboxed execution, and human review to produce commands that are semantically correct and practically executable.The paper presents this as a dataset-quality measure and a core contribution.Source: §1 · p. 2
• KaliBench contains 8,504 query–command pairs across 23 capability dimensions and five security phases, with 5,000 evaluation and 3,504 training samples after deduplication.Its pipeline includes LLM-based data generation, multi-round verification-regeneration, representative-sample selection, and alias-aware, fine-grained scoring.Source: §1 · p. 2, Figure 2 · p. 3
• Deterministic CLI structure enables verifiable reward signals without runtime execution, supporting reinforcement-learning training.The paper reports improved performance for an 8B model and results competitive with substantially larger models.Source: §1 · p. 2
• Argument construction remains the primary bottleneck, while SFT and RLVR improve an 8B model’s mean Total Score across three evaluation modes by 7.5 percentage points.The benchmark also provides fine-grained supervision over tool selection and argument construction.Source: §1 · p. 2

LLMs and Cybersecurity Benchmarks

• Knowledge-focused cybersecurity benchmarks assess conceptual understanding through question-answering and classification tasks, rather than concrete operational behavior.The cited examples include multiple-choice, true/false, and short-answer formats.Source: §2 · p. 3
• End-to-end CTF and enterprise-style evaluations combine planning, environment interaction, observation, tool use, and recovery, making individual failure sources difficult to isolate.KaliBench complements them by separately evaluating tool selection, optional arguments, and positional arguments.Source: §2 · p. 3

Function-Calling Benchmarks and Evaluations

• Function-calling benchmarks generally use schema-defined APIs, whereas KaliBench addresses the unstudied setting of inferring tools and arguments from a shared CLI.The cited work also identifies a gap in fine-grained, alias-aware assessment of real-world cybersecurity commands.Source: §2 · p. 3
• KaliBench organizes its tools across five security phases and 23 functional dimensions, with examples pairing natural-language queries and decomposed ground-truth commands.The phases follow the security attack lifecycle.Source: Figure 3 · p. 4

3 KaliBench

• Generation begins from 2,372 Kali Linux tool documents and produces 27.7K query–command pairs with tool, alias-aware optional-argument, and positional-argument annotations.Each tool receives 10–13 generated examples, with the official documentation and taxonomy guiding the process.Source: §3 · p. 4
• Initial inspection found hallucinated or misspelled options in over 50% of generated commands; LLM verification filtered 53.8% in its first round and iterative regeneration addressed uncovered tools.The resulting LLM-validated dataset contained 14.5K pairs before further verification.Source: §3 · pp. 4–5
• Sandbox execution filters commands with errors indicating hallucinated tools or invalid arguments, and human review identifies additional malformed options, omissions, and semantic mismatches.Terminal verification filters 11.95% of LLM-verified commands; human review finds another 4.9% of the verified dataset invalid.Source: §3 · p. 5
• The final benchmark contains 8,504 verified query–command pairs; its 5,000-example evaluation set covers all 1,642 tools, while the 3,504-example training set covers 962 tools.The dataset is a translation layer between natural-language cybersecurity queries and executable Kali Linux commands.Source: §3 · pp. 4–5
• Evaluation varies tool information across unrestricted, restricted, and hinted settings, from inferring tools and arguments to receiving candidate tools and their documentation.The restricted setting supplies 20 candidate tool names, including the correct tool.Source: §3 · p. 5
• Shell-style outputs are parsed without execution; component scoring checks tool selection, alias-aware optional arguments, and ordered positional arguments, alongside total and exact-command scores.Optional-argument order is flexible, but positional arguments must remain in a fixed order.Source: §3 · pp. 5–6
• KaliBench groups 23 Kali Linux capability dimensions into five security phases, assigning each dimension a primary functional role to support coherent performance analysis.The phase taxonomy spans the workflow from reconnaissance and initial access to defensive analysis and reporting.Source: §3 · p. 6
• Training combines all three evaluation settings into 10,512 samples; SFT targets ground-truth commands, while RLVR rewards correct components and exact matches without executing model outputs.The component rewards cover tool names, positional arguments, and alias-aware optional arguments, with an additional formatting reward.Source: §3 · p. 6

4 Experiments

• Experiments evaluate general-purpose and cybersecurity-oriented models, with three RedSage-K post-training variants trained using LoRA-based SFT, GRPO, or both.Models with 72B or more parameters use the OpenRouter API; RedSage-K variants use the RedSage-Ins 8B backbone.Source: §4 · pp. 6–7

4.1 Model Performance Across Three Modes

• 72.0%→95.2% average Tool Accuracy from U to R contrasts with modest argument-level gains, while H raises Exact Correct from 22.3% to 73.1%.Optional-argument prediction is the most challenging; tool hints produce the largest gains in command construction.Source: §4.1 · p. 7
• Cybersecurity-oriented models do not consistently outperform general-purpose models: they fare better in less-guided settings, while general-purpose models exploit hints more effectively.Reasoning-enabled variants improve argument-level performance, and scale alone does not explain model performance.Source: §4.1 · p. 8
• 61.68% full-set Exact Correct makes GPT-5.6-Sol the leading proprietary system, ahead of Codex at 51.68% and Claude Opus 5 at 44.02%.The strongest evaluated open-weight model, GLM-5.2, scores 41.32%; GPT-5.6-Sol answers 4,988 of 5,000 queries.Source: §4.1 · p. 8, Table 2 · p. 8
• RedSage-K (SFT+GRPO) reaches 79.2 Avg and the best compact-model U/R Total Score and Exact Correct, but GRPO variants slightly reduce H-mode Total Score.SFT and GRPO show complementary strengths; SFT initialization mitigates regressions on previously solved examples.Source: §4.1 · p. 8

4.2 Fine-Grained Performance and Robustness Analysis

• Performance varies across dimensions, but relative difficulty broadly persists across modes; added tool information narrows Total Score dispersion without eliminating dimension-level gaps.Figure 5 reports dimension and phase distributions alongside mode-level scores; U has the widest spread, while H stabilizes performance across most dimensions.Source: §4.2 · pp. 9–10, Figure 5 · p. 9
• Tools with simple invocation patterns and clear defaults are handled more reliably than interfaces with irregular options, where flag–value binding and strict syntax remain challenging.Recurring errors include option misuse, hallucinated options or aliases, and singular–plural option-name confusion.Source: §4.2 · pp. 9–10
• Messy queries cause the largest performance drop across wording variations, while Hinted mode declines by only 1.07 percentage points.Paraphrases have little overall effect; informal wording introduces a small but consistent degradation.Source: §4.2 · p. 10, Table 3 · p. 10

• Restricting candidate tools largely resolves tool-selection errors, but argument correctness—especially optional flags and flag–value bindings—remains the primary bottleneck.The discussion frames command-structure evaluation as important in schema-free settings where tools and syntax must be inferred without explicit APIs.Source: §5 · p. 10
• KaliBench supports model improvement through fine-grained runtime-free rewards, and SFT and RLVR improve compact models toward larger systems.The paper describes its framework as supporting reliable, executable use of LLM-based cybersecurity tools.Source: §5 · p. 10
• The study focuses on single-turn command generation and documentation-grounded data; future work includes multi-step workflows, retrieval-augmented tool use, and environment-aware evaluation.Source: §5 · p. 10

• The authors answer yes, stating that the abstract and introduction clearly define the article’s contributions and scope.Source: NeurIPS Paper Checklist · p. 17
• The checklist asks that claims match the results and communicate important assumptions and limits on generalization.It also says aspirational goals may be included when clearly distinguished from attained results.Source: NeurIPS Paper Checklist · p. 17

2. Limitations

• The authors say KaliBench’s contribution is experimental rather than theoretical and refer readers to Appendix O for the limitations discussion.The paper marks theoretical assumptions and proofs as not applicable because it focuses on experimental results.Source: §2 · pp. 17–18
• Experiments are supported by released code and data, with prompts, terminal-verification rules, metric calculations, and training details documented for reproduction.The paper also reports dataset splits, evaluation protocols, and training methods in Section 3, with model configurations and hyperparameters in Appendix G.Source: §2 · pp. 18–19
• Figure 5 reports box plots with Q1–Q3 and min/max whiskers across models, security dimensions, and mode settings to show variability.Source: §2 · p. 20

A Ethical Statement

• The paper discusses KaliBench’s dual-use risks and release precautions, and releases it for research under CC BY-NC 4.0 with responsible-use documentation.The dataset derives from public Kali Linux documentation, contains no personally identifiable information, and involved no human subjects or external annotators.Source: §2 · p. 21, A Ethical Statement · pp. 23–24
• KaliBench supports reliability evaluation for defensive cybersecurity uses but adds no attack techniques or end-to-end attack automation.Its data primarily come from public Kali Linux documentation, and its scope excludes multi-step attack chains and environment-adaptive procedures.Source: A Ethical Statement · p. 24

B CLI Command Generation

• Canonicalization fixes command ordering, flag-value syntax, quoting, and alias handling to enable reproducible parse-tree matching.Commands use a canonical form, preferably long flags; predictions are isolated with standardized output tags.Source: B CLI Command Generation · p. 24

C.1 Prompt for Generating the Query-Command Pairs

• Generation prompts require documentation-faithful, uniquely determined queries and canonical commands with explicit parameters and structured argument annotations.They prohibit invented flags, pipelines, subshells, and implicit defaults.Source: §C.1 · p. 25

C.2 Prompt for LLM-assisted Data Verification

• The verification prompt checks documented flag existence, argument correctness, and whether a command can fulfill the query, labeling unsupported commands as hallucinated.It returns a binary verdict and concise justification for filtering erroneous annotations.Source: §C.2 · pp. 25–26

C.3 Prompt for Model Evaluation

• Evaluation prompts require one executable command with correct tool selection and complete arguments; model coverage spans 8B–753B parameters.Restricted and Hinted modes add tool names or names with usage documentation; evaluation used temperature 0.2 and a 16,384-token context limit.Source: §C.3 · p. 26, §G.1 · p. 29

D KaliBench Samples in Detail

• Benchmark instances pair natural-language queries with tool invocations and ground-truth commands across five security phases and 23 evaluation dimensions.The dimensions and phases are illustrated through representative samples in Figure 6.Source: D KaliBench Samples in Detail · p. 27, Figure 6 · p. 27

E Terminal verification Details

• Terminal verification executes each ground-truth command in an isolated Kali Linux Docker sandbox and uses output patterns to flag hallucinated tools or parameters.Commands run with timeouts and resource limits; syntactically valid commands that fail at runtime on placeholder inputs can be retained.Source: E Terminal verification Details · pp. 27–28

F Representative Invalid Cases Identified During Manual Review

• Manual review identifies malformed flags, omitted query-required arguments, and semantic errors involving flag function, operand order, or argument type.Examples include using --g instead of -g, omitting kustomize’s bash argument, and supplying a filesystem path where svnsync requires a URL.Source: F Representative Invalid Cases… · pp. 28–29

G.2 Training Setup.

• Both SFT and GRPO used LoRA rank 64 and alpha 128, zero dropout, bfloat16, gradient checkpointing, and 2 training epochs on one NVIDIA H200 GPU.Adapters covered attention and MLP projection layers; the optimizer was 8-bit AdamW with a 5 × 10−6 learning rate.Source: §G.2 · p. 29
• SFT used batch size 4 per device with gradient accumulation of 8, yielding an effective batch size of 32; training took about 32 minutes.The setup used a 16,384-token maximum sequence length and standard negative log-likelihood loss.Source: §G.2 · p. 29
• GRPO used 8 generations per prompt and required about 28 hours from the original checkpoint or 17 hours when initialized from the SFT checkpoint.It used a BNPO-style loss with clipping parameter ϵ = 0.2 and KL coefficient β = 0.001.Source: §G.2 · p. 29

H Model Performance Difference across Execution Modes

• Restricted mode improved Tool ∆1 by 23.2 pp on average, but Exact ∆1 rose only 5.9 pp, as argument selection, value binding, and formatting remained errors.The analysis says restricted mode mainly reduces tool-choice ambiguity without substantially solving command construction.Source: H Model Performance Difference across… · p. 30
• Hinted versus restricted mode raised Optional F1 by about 39.1 pp and Positional F1 by about 21.9 pp on average, while Tool Accuracy fell for 11 of 24 models.Gemma-3-it-27B’s Tool Accuracy dropped by 10.3 pp; the passage attributes the argument-metric gains to resolving optional-argument, flag, value, and ordering ambiguities.Source: H Model Performance Difference across… · p. 30
• 44.8 pp average Exact ∆2 was nearly eight times Exact ∆1, indicating that argument correction outperformed tool disambiguation for producing fully correct commands.Larger Exact gains were more closely associated with Optional ∆2 than Tool ∆1.Source: H Model Performance Difference across… · p. 30
• RedSage-K variants showed smaller mode-switching gains than strong general-purpose models, consistent with weaker Hinted-mode gains despite improved intrinsic command understanding after post-training.Across models, the main differentiator was how effectively documentation repaired argument errors and supported exact commands.Source: H Model Performance Difference across… · pp. 30–31

I RLVR Failure Analysis

• Across 5,000 evaluation examples, recurring RLVR failures included invalid optional arguments or option–value pairs, incorrect positional arguments, and regressions on examples already solved by the baseline.One example uses msfvenom –list-formats instead of msfvenom –list formats.Source: I RLVR Failure Analysis · p. 31
• In H mode, GRPO and SFT+GRPO reduced Total Score by 0.76 and 0.19 percentage points, respectively, versus RedSage-Ins; GRPO regressed on 562 baseline successes.Its degradation concentrated in commands requiring optional arguments or a mixture of optional and positional arguments.Source: I RLVR Failure Analysis · p. 31
• Compared with GRPO alone, SFT+GRPO improved Total Score by 2.91 points in U and 3.28 points in R, and reduced H-mode exact-match regressions from 562 to 488.The passage identifies SFT initialization as mitigating these regressions.Source: I RLVR Failure Analysis · p. 31
• Across security dimensions, both RLVR variants broadly improved performance in U and R, with no robust evidence of a dimension where RLVR systematically failed to improve.Small H-mode variations did not form a consistent dimension-specific pattern.Source: I RLVR Failure Analysis · p. 32

J Analysis of CLI Option Distribution

• The evaluation set contains 3,074 optional arguments (2,739 train; 4,378 full), spanning common meta-flags and security-relevant network, target, credential, and output controls.The reported distribution is presented as evidence that the dataset includes functional command patterns beyond generic flags.Source: J Analysis of CLI Option Distribution · pp. 32–33

K Complete List of Best-Worst Performed Tools

• Across 24 configurations and three modes, d2j-std-apk, rfdump, and truecrypt2john averaged perfect scores, while nikto paired 0.986 avg_tool with 0.000 avg_exact.Low-performing tools’ weaker argument scores show that strong tool identification alone can coexist with poor exact correctness.Source: K Complete List of Best-Worst… · p. 33

L.1 Command-level Scoring for Evaluating Models

• Commands are parsed into tool, alias-grouped optional flags, and positional arguments; component F1 scores contribute to Total score, while Exact match requires a correct canonicalized command.The scoring descriptions specify separate matching for optional flags and positional arguments.Source: §L.1 · pp. 33–34

L.2 Dimensional Score Calculation

• Each instance’s scalar score is assigned to every capability dimension associated with its tool, and aggregating these projections yields per-dimension performance scores.The tool’s taxonomy assignment is represented by a multi-hot vector across 23 dimensions.Source: §L.2 · p. 34

M Detailed Results for Proprietary Models and Agentic Scaffold

• Both standalone proprietary models score lower on Pos-F1 than Opt-F1 despite high tool-selection accuracy, a pattern also seen with Codex.Exact Correct requires correct tool selection, flag–value bindings, and positional arguments; component errors accumulate, leaving EC below the lowest component score in all three modes.Source: M Detailed Results for Proprietary… · p. 34, Table 8 · p. 35

N Model-wise Sensitivity to Messy Queries

• Every tested configuration loses Total Score on messy queries: most drop about 3 percentage points, two lose more than 4, and the smallest drop is 1.98 percentage points.The comparison uses each configuration’s Unrestricted-setting score on messy inputs against its score on corresponding clean inputs.Source: N Model-wise Sensitivity to Messy… · p. 35, Figure 8 · p. 35

O Limitations

• Ground-truth commands reflect Kali manuals available at dataset construction, so later flag additions, deprecations, renamings, or reinterpretations can make canonicalized commands outdated.The passage says these effects are limited in practice because tools often maintain backward compatibility, and hinted evaluation is insulated by providing tool documentation.Source: O Limitations · p. 35
• Conflicting manual conventions, such as regtree’s conflicting -V descriptions and Impacket’s atypical short-option forms, make deterministic canonicalization approximate when documentation diverges from behavior or variants coexist.The authors characterize these cases as reflecting CLI diversity and historical evolution rather than issues specific to KaliBench.Source: O Limitations · pp. 35–36
• The manual-grounded LLM verifier remains imperfect, and semi-automatically extracted alias groups miss some historical, distribution-specific, or community-adopted variants.Human review of a random subset indicates residual verifier errors are limited.Source: O Limitations · p. 36
• KaliBench is a controlled, well-defined abstraction for systematic fine-grained NL-to-CLI evaluation that complements execution-based or agentic evaluations in operational cybersecurity environments.Source: O Limitations · p. 36