PaperlayerView exact revision on arXiv

Source-linked AI summary

SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

Hanna Kim, Jian Cui, Minkyoo Song, Hwanjo Heo, Seungwon Shin, Kimin Lee, Xiaojing Liao

arXiv:2609.06149v1cs.CRcs.AI

TL;DR

Static IOC recovery from malicious scripts is difficult because indicators may be transformed or dispersed, while existing evidence lacks statically restricted, recovery-aware ground truth. SCRIPTIOC-BENCH evaluates LLMs on manually verified script samples and finds that recovery remains challenging, with mitigations improving grounding and precision.

  • Problem

    Static recovery of actionable IOCs from malicious scripts is underexplored, and existing datasets do not provide ground truth restricted to statically obtainable indicators with recovery-level annotations.

  • Method

    SCRIPTIOC-BENCH evaluates LLM extraction on 634 manually verified JavaScript, PowerShell, and VBScript samples covering four IOC types and recovery levels.

  • Results

    Static IOC extraction remains challenging across model scales: the strongest model reaches 65.4 F1, while models perform better on direct indicators than deeper reconstruction.

  • Takeaways & Limitations

    Deterministic string utilities and task-specific adaptation improve grounding and precision, but Program-dependent recovery remains a key challenge.

  • Takeaways & Limitations

    The benchmark covers only JavaScript, PowerShell, and VBScript and four IOC categories, leaving other artifact types for future extensions.

Abstract

from arXiv · show

Script-based malware remains a prevalent attack technique. These scripts often contain indicators of compromise (IOCs) that provide actionable threat intelligence. However, statically recovering such indicators is challenging, as relevant values may be dispersed or transformed within code. Although large language models (LLMs) have shown promise in security analysis, their ability to recover IOCs from malicious scripts remains underexplored. We present SCRIPTIOC-BENCH, a benchmark for measuring static IOC extraction capability on real-world malicious scripts. The benchmark comprises 634 manually verified JavaScript, PowerShell, and VBScript malware samples covering four IOC types (URLs, domains, IP addresses, and filesystem artifacts). We further stratify ground-truth IOCs by recovery level, distinguishing directly exposed indicators from those requiring decoding or reconstruction. Using this benchmark, we evaluate a broad range of proprietary and open-weight LLMs and show that IOC recovery without execution remains challenging across model scales: the strongest model reaches only 65.4 F1. To characterize how recovery fails, we introduce a false-positive taxonomy and use it to compare the error profiles of the evaluated models. We further study two mitigations on a small open-weight model, deterministic string utilities and task-specific adaptation, finding that they provide complementary recovery gains, raise precision, and shift errors toward sample-grounded mismatches.

1 Introduction

SCRIPTIOC-BENCH addresses the underexplored problem of statically recovering actionable IOCs from real-world script-based malware. It benchmarks LLMs across recovery difficulty and input complexity, while analyzing failure modes and mitigation strategies.

  • Script-based malware enables malicious execution, payload retrieval, and host interaction through built-in commands and scripting interpreters.
  • The benchmark addresses the lack of ground truth restricted to statically obtainable IOCs and recovery-level annotations.
  • 634 manually verified JavaScript, PowerShell, and VBScript samples cover URLs, domains, IP addresses, and filesystem artifacts.
  • The evaluation compares open-weight and proprietary LLMs across recovery difficulty, input complexity, and false-positive failure modes.
  • 65.4 F1 is achieved by the strongest model, while performance drops sharply when IOC recovery requires deeper reconstruction.
  • Deterministic string utilities and task-specific adaptation shift a small model’s errors toward sample-grounded errors.

2 Background and Related Work

The paper situates static IOC recovery within the operational importance and obfuscation of script-based malware. It distinguishes this benchmark from prior work through statically recoverable, indicator-level ground truth and broader evaluation coverage.

  • Script-based Malware: Script-based malware commonly uses JavaScript, PowerShell, and VBScript to execute commands, download payloads, and reduce reliance on custom binaries.
  • Script-based Malware: Obfuscation techniques such as string encoding, control-flow flattening, and runtime evaluation complicate IOC recovery.
  • IOC Extraction: Static analysis avoids execution, scales to large corpora, and remains applicable when scripts refuse to execute, unlike costly and evadable sandboxing.
  • Related Work: Prior work spans obfuscation detection, program analysis, deobfuscation, and LLM-based code or malware reasoning.
  • Problem Scope: SCRIPTIOC-BENCH restricts labels to IOC values fully determined by script source and embedded data without execution or runtime input.
  • Problem Scope: The benchmark covers URLs, domains, IP addresses, and filesystem artifacts across JavaScript, PowerShell, and VBScript.

3 SCRIPTIOC-BENCH

SCRIPTIOC-BENCH gives an LLM a task description and malicious script, then evaluates structured IOC predictions against recovery-level ground truth. The benchmark covers three scripting languages, four IOC types, and static extraction under context constraints.

  • Overview: The framework comprises task input, LLM-based IOC extraction, and evaluation against labeled IOC ground truth with recovery-level annotations.
  • Task Formulation: Given a task description and malicious script, the model must statically extract typed IOCs without executing the sample.
  • Inference Setup: Static extraction may require resolving indirect references, deobfuscating values, and reconstructing artifacts split across strings.
  • Task Formulation: Predicted indicators are represented as value–type pairs, with types URL, DOMAIN, IP, or FILE.
  • Benchmark Composition: The benchmark targets JavaScript, PowerShell, and VBScript, which support distinct malware execution and delivery settings.
  • Benchmark Composition: The four IOC categories capture network infrastructure and host-side traces appearing in payload delivery, command-and-control, persistence, and file-write operations.
  • Inference Setup: Long scripts are split into token-bounded chunks, processed independently, and aggregated when they exceed the model context window.
  • Evaluation: Models are evaluated with per-IOC precision, recall, and F1, including analyses by recovery level and input complexity.

4 Datasets

SCRIPTIOC-BENCH is constructed from manually verified script malware with statically recoverable IOC labels and recovery-level annotations. Its analysis shows substantial variation in input complexity and IOC distributions across script subsets.

  • Dataset construction: The dataset construction combines malware samples from Filescan.io and the JS Malware Collection, with IOC labels verified for static recoverability.Annotation used malware-analysis reports only as auxiliary cross-checks because those reports can mix static and dynamic evidence.
  • IOC annotation: Recovery levels are assigned per IOC: Direct values are explicit, while fixed-transform values use deterministic reassembly or transcoding without executing the script.Program-dependent indicators are not recovered by the fixed transform set and may still be statically recoverable through script-specific logic.
  • Input complexity: Code token lengths are heavy-tailed, with mean lengths of 325.6K, 25.4K, 58.0K, and 13.3K for JSf, PSf, VBSf, and JShp, respectively.PSf and VBSf also retain long upper tails, creating pressure for long-context reasoning and evidence aggregation.
  • Input complexity: Gzip compression ratios span a wide range, indicating structural redundancy differences that can either dilute IOCs in filler code or conceal them in encoded content.Low-ratio samples may reduce signal-to-noise, whereas high-ratio samples may hide indicators inside Base64 or XOR content.
  • IOC label distribution: URLs and domains predominate overall, but IOC distributions vary by script type and reflect distinct malware behaviors.Recent JS is sparse and URL-centric, while historical JS is domain-dense because downloader logic cycles through multiple fallback hosts.
  • IOC label distribution: PSf has the broadest URL distribution and resembles multi-stage droppers that retrieve components through multiple paths and legitimate platforms such as GitHub, Dropbox, and Telegram.VBS samples often follow a two-URL pattern, with 42% containing an initial retrieval URL and a subsequent stage.

5 Experiments

The experiments evaluate LLM-based IOC extraction across model scales, recovery difficulty, input complexity, and false-positive provenance. Stronger models lead overall, but reconstruction, long inputs, and precise grounding remain difficult.

  • Overall performance: GPT-Terra and Gemini-Pro lead at around 65 F1, while 8B models remain well behind larger systems.Qwen3.6-35B nearly matches Qwen3-C480B, whereas scaling otherwise improves recall and reduces response failures within the Qwen family.
  • IOC recovery level: Direct IOCs are substantially easier: GPT-Terra reaches 91.7% recall and Gemini-Pro reaches 89.4%.Smaller models still show substantially lower recall even when indicators appear explicitly in the source.
  • IOC recovery level: GPT-Terra’s recall falls from 90.9% for single reassembly and 82.4% for single transcode to 19.0% for three or more recovery operations.Recall declines as Fixed-transform recovery requires more operations, with single transcoding harder than single reassembly.
  • IOC recovery level: Program-dependent recovery remains a major limitation, with Qwen3-C480B reaching only 13.8% recall at this level.These indicators expose differences in models’ ability to reason over program-specific logic.
  • Input complexity: Longer inputs generally increase response failures and reduce Regex-missed Recall, even when a model maintains low failure rates.Qwen3-C480B illustrates that reliable completion and recovery beyond surface matching can degrade separately with input length.
  • False-positive analysis: False-positive profiles differ by scale: smaller models often echo prompt examples, whereas frontier models more often produce sample-grounded Near-copy or Granularity errors.Within Qwen3-Coder, Prompt echo falls from 53% of false positives for C80B to about 8% for C480B; remaining errors shift toward Granularity and Unattributed categories.
  • False-positive analysis: Greater capacity shifts errors toward malware-grounded mismatches, reducing obviously spurious indicators but leaving residual predictions that require closer analyst verification.Frontier models fail less through prompt examples but still struggle with precise reconstruction and maintaining grounding.

6 Improving Small-Model IOC Recovery with Tools and Adaptation

The study tests deterministic string utilities and task-specific adaptation on Qwen3-8B to improve IOC recovery and reduce false positives. The interventions provide complementary gains, but neither materially improves program-dependent recovery.

  • Study design: The case study compares base, tool-augmented, adapted, and adapted-plus-tool Qwen3-8B variants under a shared inference setup.Deterministic utilities cover common decoding, string reconstruction, and reversible transformations.
  • String-utility tool augmentation: Tool access increases Direct recall from 67% to 73% but decreases Fixed-transform recall, partly because added interactions increase parsing failures.Tool augmentation primarily improves grounding and recovery of indicators directly present in scripts.
  • False-positive behavior: Both approaches improve precision while shifting errors away from prompt echo and toward more sample-grounded mismatches.Tool access suppresses Prompt echo more sharply, while adaptation also reduces it but less strongly.
  • Task-specific adaptation: Adaptation increases Fixed-transform recall from 17% to 22%, with single-step reassembly improving from 28% to 38%.Single-step transcode changes little, indicating that adaptation helps recurring reassembly patterns more than reliable transcoding.
  • Remaining bottleneck: Program-dependent recall remains only 1–3% across all four variants, so the approaches do not materially extend recovery through sample-specific program logic.This remains the hardest recovery level even for frontier models.

7 Discussion

The benchmark shows that current LLMs are not yet reliable for standalone static IOC extraction, especially when indicators require deeper reconstruction. The study also identifies scope boundaries and contamination concerns that qualify interpretation of results.

  • Implications for Analyst Workflows: 65.4 F1 is the strongest-model result, while recall drops sharply for Program-dependent indicators.These results indicate that static IOC extraction remains unreliable without analyst or tool support.
  • Implications for Analyst Workflows: Models exhibit different failure modes, so their outputs require verification tailored to each error profile.The discussion links varied model errors to the need for workflow-level checking rather than unreviewed deployment.
  • Implications for Analyst Workflows: Deterministic string utilities and task-specific adaptation improve grounding and precision in a controlled study on a small open-weight model.The two mitigations provide complementary gains and shift errors toward sample-grounded mismatches.
  • Ground-Truth Scope: The benchmark covers URLs, domains, IP addresses, and filesystem artifacts, while registry artifacts remain outside its current scope.Registry artifacts require additional annotation and matching rules because their threat-relevant unit can include paths, values, and operations.
  • Data Contamination: GPT-Terra is the only evaluated model for which pretraining contamination may overlap with the recent datasets.Its recent-subset results remain far from saturated, reaching 65.2, 57.0, and 55.0 F1 on JS_f, PS_f, and VBS_f, respectively.

8 Conclusion

SCRIPTIOC-BENCH evaluates static recovery of concrete IOCs from malicious scripts and provides a shared testbed for reproducible comparison. The conclusion finds that direct indicators are easier than deeper reconstruction, while tool augmentation and adaptation improve grounding and precision.

  • 8 Conclusion: SCRIPTIOC-BENCH measures static recovery of URLs, domains, IP addresses, and filesystem artifacts from JavaScript, PowerShell, and VBScript malware.Its labels are manually verified and restricted to indicators recoverable from the script file alone.
  • 8 Conclusion: LLMs perform well on directly exposed IOCs but struggle with deeper reconstruction.Program-dependent recovery remains a key challenge even for frontier models.
  • 8 Conclusion: False-positive profiles vary across model capabilities, motivating fine-grained error analysis.The benchmark uses this analysis to characterize how recovery fails across evaluated systems.
  • 8 Conclusion: Tool augmentation and task-specific adaptation improve grounding and precision.The conclusion presents these interventions as mitigations rather than complete solutions to Program-dependent recovery.
  • 8 Conclusion: The public leaderboard enables reproducible comparisons among LLMs, tool-augmented agents, and rule-based baselines.It is intended to track progress and expose remaining failure modes under a shared protocol.

C Training Details

The training procedure combines supervised fine-tuning and reinforcement learning with task-specific rewards, structured prompts, deterministic tools, and long-context model configurations. Its reward design emphasizes IOC quality while discouraging fabrication and mass enumeration.

  • Training Procedure: Qwen3-8B is adapted through supervised fine-tuning followed by reinforcement learning with verifiable rewards.Both stages train LoRA adapters over the open-weight base model.
  • Reward Function: The GRPO reward combines five terms because IOC extraction requires both classification and typed indicators that may be only partially correct.The design addresses distinct failure modes of malicious-code analysis.
  • Reward Function: Format, classification, and status terms reward parseability, verdict correctness, and consistency between declared status and emitted IOCs.Status mismatches include declaring absence while emitting IOCs or declaring presence while emitting none.
  • Reward Function: False-positive-only samples receive a negative precision reward, and recall is dampened when unmatched IOCs are emitted.These countermeasures penalize fabrication and mass enumeration; IOC precision and recall receive the largest reward weights.
  • Prompting and Inference: The prompt pipeline uses single-pass extraction for short samples and map–reduce processing for long samples.Tool-augmented variants append a tool-usage block, while map outputs are synthesized by a reduce call.
  • Prompting and Inference: Cross-chunk synthesis completes unresolved findings only when another chunk supplies the missing evidence, and prohibits unsupported values.The tool-augmented workflow calls deterministic utilities before producing tagged sections and marks unresolved dependencies rather than guessing.
  • Models and Tools: The evaluation compares 8B-scale and larger open-weight model groups under model-specific context and generation budgets.Agent settings expose deterministic utilities through a tool-calling wrapper with a per-sample limit of ten model calls.

F Map–reduce Long-context Handling

Long samples are handled by splitting code into chunks, summarizing partial findings, and aggregating them in a final reduce call. URL performance is also evaluated under increasingly strict matching criteria, which lowers measured F1.

  • Map–reduce Long-context Handling: Samples exceeding the code budget use map–reduce, with each chunk producing partial findings for a final structured-output aggregation.Map outputs are truncated to 1,500 tokens, contributing approximately N ·1,500+1,000 tokens to the reduce prompt for N chunks.
  • Map–reduce Long-context Handling: The per-chunk code budget is determined from served context length, prompt overhead, and reserved output space.The margin accounts for tokenizer drift and decoded-length expansion.
  • Matching Strictness: URL F1 decreases from endpoint-level matching to endpoint-plusquery-key and full-URL matching.Models more often recover the correct endpoint than the exact query structure or complete URL string.

H False-positive Taxonomy: Labelling Rules

The taxonomy deterministically assigns each false positive to one provenance category using ordered rules, then characterizes operational severity and threshold sensitivity.

  • Closest-reference matching: Each false positive is paired with its closest reference IOC using sequence similarity, common-substring length, and edit distance.The paired reference supports downstream provenance classification.
  • Precedence: Rules are evaluated top to bottom, and the first matching predicate assigns exactly one label to each false positive.The first two rules handle malformed values; the remaining rules address well-formed values.
  • Category rules: The taxonomy distinguishes normalization errors, ungrounded invalid values, benign shared services, type mismatches, granularity errors, near-copies, prompt echoes, source non-IOCs, placeholders, and unattributed values.Granularity captures the right entity at a different altitude, while near-copy captures boundary cuts or fuzzy distortions.
  • Operational handling: The Disposition column gives a rough operational-severity ordering for errors that survive triage, with UNATTRIBUTED spanning review cost and possible misdirection.The ordering is not a single-property severity axis.
  • Threshold sensitivity: ±0.05 shifts in either NEAR-COPY ratio threshold move at most 2% of reported false positives between adjacent IOC-related categories and never across the non-IOC / IOC boundary.Thresholds were tuned on a heldout development split and reported for reproducibility rather than theoretical motivation.

I IOC-type-wise analysis

IOC extraction varies substantially by script and IOC type: frontier models are relatively stable on URLs and domains, whereas filesystem artifacts remain consistently difficult.

  • URLs and domains: URL and domain extraction is relatively stable for frontier models across scripts and IOC types.This pattern suggests greater robustness to obfuscation and contextual ambiguity for these IOC categories.
  • URLs and domains: Open-weight models show larger performance variation, including notable drops on JShp.Their performance depends more strongly on indicators remaining close to surface form.
  • Filesystem artifacts: Filesystem artifacts are the most challenging IOC category overall, with consistently low F1 across models and datasets.Files combine filenames, paths, extensions, temporary artifacts, payload names, and benign-looking strings that require contextual interpretation.
  • Filesystem artifacts: Directory-level scoring yields substantially higher F1 than full-path scoring because filenames are often sample-specific and constructed through concatenation.Directories more often follow familiar conventions, while basenames are more frequently split, transformed, or assembled.

J IOC Recovery-Level Labeling

The benchmark labels each statically obtainable IOC by the recovery process required, separating direct exposure, fixed transformations, and program-dependent reconstruction. Harder levels require reasoning beyond the fixed decoder and remain unevenly recovered across transformations and mechanisms.

  • Recovery-level labeling: Each ground-truth IOC is graded against a fixed decoder that provides a reproducible reference for measuring LLM recovery gaps.The decoder covers reversible, program-independent transformations documented in prior work.
  • Recovery-level labeling: The benchmark excludes loops, arithmetic, and control-flow-driven decoders because they require executing attacker-controlled logic rather than inverting a fixed codec.This exclusion defines the Program-dependent boundary.
  • Recovery levels: Direct values appear verbatim, Fixed-transform values appear in decoder-produced blobs, and Program-dependent values require reasoning over the script’s own decode logic.These three levels define progressively more involved recovery requirements.
  • Recovery levels: Program-dependent is a one-sided label: it identifies values not reached by the fixed decoder, not values that are impossible to recover statically.Composite indicators receive the highest recovery level among their components.
  • Recovery performance: Frontier models are generally strongest on split/join and literal concatenation, while decoding-heavy transformations such as Base64 are less reliable.Performance varies substantially by operator and model, including across Program-dependent construction mechanisms.
  • Decoder design: The fixed decoder recursively applies its transform set to depth 4, deduplicates by hash, caps blobs at 4 MB, and limits each operator to 60 matches.These constraints yield candidate decoded blobs per script.
Loading 2609.06149v1…