HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation

I. INTRODUCTION

• Existing tools rely on static rule content and manually maintained fixes, leaving work to administrators when no suitable fix exists, configurations diverge, or guidance changes.HardEval isolates code-generation capability from retrieval context, post-processing, and orchestration.Source: §I · p. 1
• Existing code benchmarks omit persistent system state, repository and agent benchmarks target application code, and security evaluations leave OS hardening unmeasured.Even SHIELDS [1] evaluates an end-to-end remediation loop rather than isolating the required code-generation capability.Source: §I · p. 1
• HardEval withholds the implementation, executes model-generated Bash scripts on virtual machines, and grades each task by whether the corresponding OpenSCAP/OVAL rule flips from fail to pass.The hidden answer key contains reference remediation and the OVAL check identifier.Source: §I · pp. 1–2, Figure 1 · p. 2
• HardEval formalizes live-system remediation as an evaluation problem and provides a reusable benchmark built from DISA STIG failures and ComplianceAsCode content.The authors report cross-author prompt robustness and a zero-code Ubuntu 24.04 port, with model ordering stable across prompt authors.Source: §I · p. 2
• Across fourteen configurations, frontier models solve many hardening tasks but still fail on scanner-sensitive details that text-based evaluation would miss.The recurring failure patterns include scanner-specific value or mechanism mismatches, despite plausible-looking security code.Source: §I · p. 2

II. RELATED WORK

• Code-generation suites test isolated snippets without persistent system state or OS-configuration concerns.They include HumanEval [2], MBPP [3], APPS, MultiPL-E, BigCodeBench, and LiveCodeBench [7].Source: §II · p. 2
• Repository and agent benchmarks evaluate realistic application tasks, but none tests OS-hardening shell scripts against domain-specific compliance checks.Many permit tool loops that could reveal the fix; HardEval withholds the implementation.Source: §II · p. 2
• Security evaluations target insecure-code avoidance, offensive capability, or hardware and firmware; SHIELDS [1] instead uses an iterative scanner-feedback loop, while HardEval tests single-shot remediation-code generation.SHIELDS [1] resolves up to 73% of findings; HardEval uses no agent orchestration or scanner feedback.Source: §II · p. 3

III. THE HARDEVAL BENCHMARK

• The benchmark pipeline combines source-data construction, implementation-withheld prompt authoring, execution-based grading, and model evaluation to isolate hardening capability.The design aims to separate capability from retrieval, orchestration, and leakage confounds.Source: §III · p. 3

A. Source Material and Dataset Construction

• An OpenSCAP scan under the DISA STIG profile supplies failing rules, expert-authored descriptions, reference Bash remediations, and OVAL grading identifiers.Of 233 initially failing rules, 215 have reference Bash remediations and 200 automatable rules become graded tasks; the fixed cross-model manifest contains 151 verified-applicable rules, or 137 excluding SSH.Source: A. Source Material and Dataset… · p. 3
• Only the prompt p_i is exposed to the model; the reference remediation r_i and OVAL identifier o_i are withheld at inference and used for grading.Dataset extraction parses scan XML into reference data and grading metadata, while prompt generation is a decoupled stage.Source: A. Source Material and Dataset… · p. 3

B. Implementation-Withheld Prompt Authoring

• Prompt authors use rule titles, severity, rationale, and descriptions—but not reference scripts—to generate implementation-withheld tasks.Source: B. Implementation-Withheld Prompt… · p. 3
• Prompts state a precise objective and rationale, omit implementation mechanisms, preserve mechanism-independent constraints, and require a single Bash script.A lexical scan checks prompts for hidden-reference paths, directive names, and other mechanism tokens; two of 215 prompts were flagged, both benign.Source: B. Implementation-Withheld Prompt… · p. 3
• The example prompt discloses a 15-character minimum while withholding the target file, directive, and enforcement mechanism.The hidden reference sets minlen = 15 in /etc/security/pwquality.conf, which the OVAL oracle checks.Source: B. Implementation-Withheld Prompt… · p. 3, Figure 2 · p. 4

C. Evaluation Protocol

• Each model receives one prompt and submits one script without retries, tool access, or live system inspection.Each configuration runs on its own throwaway AlmaLinux 8 virtual machine restored to the same failing baseline.Source: C. Evaluation Protocol · p. 3
• The benchmark comparison table organizes representative benchmarks by output execution, execution environment, and external tool or agent access.Source: Table I · p. 4
• Scripts run with reboot-related commands neutralized, and only an OpenSCAP rule verdict of literal pass earns credit.A full rescan after the final script agreed with rule-level outcomes in all retained RHEL runs.Source: C. Evaluation Protocol · p. 4
• Because scripts persist across a run, aggregate scores measure cumulative batch hardening rather than always attributing each rule pass to its task’s script.The harness assumes non-adversarial models; runs disabling a scoring subsystem are excluded as invalid.Source: C. Evaluation Protocol · p. 4
• HardEval measures satisfaction of the DISA STIG compliance oracle, not unrestricted security, and a target-rule pass does not rule out collateral host effects.An alternative mechanism may improve security yet fail the probe.Source: C. Evaluation Protocol · p. 4

• The evaluation includes seven frontier closed models and seven open-weight code-specialized configurations, including GLM-4-9B at FP16 and 4-bit precision.Source: D. Models and Experimental Setup · p. 4
• Every configuration receives the same prompts and greedy decoding at temperature 0, with scripts graded on identical AlmaLinux 8 hosts using the same model-agnostic grader.The grader uses OpenSCAP 1.3.14, SCAP Security Guide 0.1.81, and the STIG profile.Source: D. Models and Experimental Setup · p. 4
• The grader extracts scripts from a Bash fence, any fenced block, or raw text as fallback, so output formatting does not penalize models.Source: D. Models and Experimental Setup · p. 5
• The construction pipeline is implemented in benchmark/build_dataset.py in the supplementary code and data folder.Source: D. Models and Experimental Setup · p. 5

• The evaluation reports cross-model HardEval results.Source: §IV · p. 5
• The evaluation examines run-to-run variance.Source: §IV · p. 5
• The evaluation includes cross-author prompt validation and cross-operating-system generalizability.Source: §IV · p. 5
• The evaluation includes failure analysis.Source: §IV · p. 5

A. Cross-Model HardEval Results

• 81.0% verified success on the combined server-safe bucket puts Claude Opus 4.7 first, while Claude Opus 4.8 scores 77.4%; the single-run gap is not treated as an ordering.The seven frontier models occupy the top seven positions, from 81.0% down to 51.1%; DeepSeek-Coder-33B, the strongest open-weight configuration, scores 35.8%.Source: A. Cross-Model HardEval Results · p. 5
• 33.6% for Qwen2.5-Coder-14B exceeds CodeLlama-34B’s 21.2%, while DeepSeek-Coder-33B reaches 35.8% despite more than twice Qwen’s parameters.The Qwen-14B and CodeLlama-34B pooled multi-run confidence intervals do not overlap; model family, instruction tuning, and decoding remain uncontrolled confounders.Source: A. Cross-Model HardEval Results · p. 5
• 1.7% on audit rules versus 24.1% on server configuration for Qwen 7B illustrates domain variation, while Sonnet 4.5 leads audit rules at 81.0%.GPT-4o scores 62.1% on audit rules versus 70.7% for Claude Opus 4.8 and 81.0% for Claude Sonnet 4.5; SSH comparisons carry a measurement-channel confound.Source: A. Cross-Model HardEval Results · p. 5
• 13.1% aggregate accuracy for GLM-4-9B FP16 and 12.4% for its Q4_K_M 4-bit version have overlapping pooled confidence intervals.The multi-run pooled estimates are 14.1% (FP16) and 12.2% (4-bit); pooled audit-rule estimates of 1.3% and 0.4% also overlap.Source: A. Cross-Model HardEval Results · p. 5

B. Run-to-Run Variance

• Qwen-14B and CodeLlama-34B have separated 95% Wilson intervals, while the two GLM-4-9B precisions overlap on the combined metric and audit rules.The pooled intervals combine one greedy and three seeded runs; the authors treat them as descriptive because decoding settings are mixed and trials on each rule are correlated.Source: B. Run-to-Run Variance · p. 5, Figure 3 · p. 6

C. Cross-Author Prompt Validation

• GPT-4o authored a parallel prompt set from the same raw STIG metadata, and both prompt sets were scored with the same harness, AlmaLinux 8 host, and OVAL grading.Six of 13 possible leakage flags in GPT-4o’s initial pass were confirmed as real mechanism leaks and regenerated.Source: C. Cross-Author Prompt Validation · pp. 5–6
• Claude Opus 4.8 falls from 77.4% to 73.7%, and GPT-4o from 65.7% to 53.3% on GPT-4o-authored prompts; Claude remains ahead on either author’s set.The lower scores do not remove the model ordering; the authors interpret the results as author-robust ordering on these two prompt sets, not prompt-invariant absolute scores.Source: Table III · p. 6, C. Cross-Author Prompt Validation · p. 6

D. Cross-OS Portability

• HardEval ports from RHEL 8 to Ubuntu 24.04 without code changes, changing only the OVAL datastream and OS name in the prompt.Models inferred the appropriate tooling from the OS name alone, using apt/dpkg rather than yum/rpm.Source: D. Cross-OS Portability · p. 6
• Two exploratory runs were excluded after a script broke the audit subsystem and caused 47 later audit checks to fail as measurement artifacts.The triggering script’s failure on its own rule was genuine, but the poisoned later verdicts were not scored.Source: D. Cross-OS Portability · pp. 6–7
• The ported pipeline produces non-degenerate, well-separated pass rates across three valid runs on 88 scoreable concepts shared by the two STIGs.The remaining concepts depend on mechanisms absent from Ubuntu or lack an Ubuntu counterpart; the 88 rules are not a hand-picked subset.Source: D. Cross-OS Portability · p. 7
• Ubuntu results support pipeline portability, not invariant task difficulty: audit rules remain highly solvable for the strongest models, while server and kernel configuration become harder.HardEval’s domain labels are operational categories rather than difficulty classes that hold across operating systems.Source: D. Cross-OS Portability · p. 7

E. Failure Analysis

• Across models, plausible hardening can still fail because the script misses the exact file, directive, or value checked by the OVAL probe.For example, Qwen2.5-Coder-7B writes PASS_MIN_LEN to /etc/login.defs, while the probe checks /etc/security/pwquality.conf.Source: E. Failure Analysis · p. 7
• Audit-rule construction is the sharpest discriminator, while SSH scores are conservative because hardening can sever the grading channel; the suggested SSH inversion did not reproduce in the manifest re-run.GPT-4o and Claude Opus 4.8 both score 10/14 on SSH in that re-run.Source: E. Failure Analysis · p. 7
• Package and service rules show the shallowest performance gradient, likely because common dnf and systemctl idioms are well represented in pretraining corpora.Source: E. Failure Analysis · p. 7

V. CONCLUSION AND LIMITATIONS

• Scores span 81.0% (Claude Opus 4.7) to 12.4% (4-bit GLM-4-9B), with every frontier model above every open-weight model on combined server-safe controls.The benchmark uses a fixed 151-rule cross-model manifest, with 137 rules excluding SSH.Source: §V · p. 7
• Audit-rule construction most sharply separates models; parameter count and release recency weakly predict within tiers, and failures center on value, mechanism, and audit-key mismatches.GLM-4-9B showed no detectable four-bit quantization penalty; a GPT-4o-authored prompt set lowered both solvers’ scores, and the Ubuntu port covered 88 shared rules.Source: §V · p. 7
• Tasks cover a curated, automatable subset of RHEL 8 DISA STIG controls; generalization to RHEL 9, Windows, and other profiles remains unshown.The Ubuntu port demonstrates methodological portability, not broader performance generalization.Source: §V · p. 7
• Ten configurations had one greedy run; variance excludes closed models, SSH scores are conservative lower bounds, and training-data contamination from public STIG content cannot be ruled out.SSH is excluded from the headline metric because the harness grades over the same channel the scripts harden.Source: §V · p. 7
• The results establish scanner-verified compliance remediation as a first-class language-model evaluation axis.Future work includes extending HardEval to RHEL 9 and Windows STIG profiles; the open-source benchmark lets the community track capability as models evolve.Source: §V · p. 7