PaperlayerView exact revision on arXiv

Source-linked AI summary

MiST: Mid-Training LLMs for Cybersecurity

Oded Ovadia, Elad Ben Zaken, Elad Guttman, Orly Moreno Kadosh

arXiv:2609.18496v1cs.CRcs.AI

TL;DR

Cybersecurity LLMs need to handle specialized, context-dependent security knowledge, while general-purpose training may not supply it and high-quality cybersecurity corpora are scarce. MiST adapts Qwen models using synthetic data generated from a compact expert-vetted seed corpus during mid-training, followed by SFT and preference tuning. The resulting models improve cybersecurity benchmark performance and provide a stronger starting point for downstream adaptation.

  • Problem

    General-purpose LLMs may not meet cybersecurity demands for specialized terminology, changing attack techniques, and operational knowledge, while high-quality cybersecurity pre-training corpora remain scarce.

    Source: §1 · p. 1

  • Method

    MiST transforms a compact expert-vetted cybersecurity seed corpus into synthetic mid-training data, then applies SFT and general preference tuning to adapted Qwen checkpoints.

    Source: §1 · p. 2

  • Results

    MiST substantially improves over corresponding Qwen baselines on public cybersecurity benchmarks, outperforms similarly sized cybersecurity models, and is competitive with larger LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MiST provides a stronger starting point than original Qwen models for task-specific SFT and reinforcement learning, with higher RL validation accuracy at lower KL movement.

    Source: §6 · p. 9

    The evaluation focuses on knowledge-intensive security understanding, cyber threat intelligence, vulnerability reasoning, and benchmark-style analysis, not operational settings such as incident triage or tool use.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Cybersecurity combines high-stakes analysis with complex technical language, making it an impactful and challenging domain for LLMs. We present MiST (Mid-trained Security Transformer), a suite of 8B and 32B models that achieve strong performance on public cybersecurity benchmarks. We use mid-training as an intermediate adaptation stage between general pre-training and cybersecurity training. Rather than performing continual pre-training over large volumes of raw domain text, we curate a compact, expert-vetted seed corpus, and transform it into high-quality domain-specific synthetic training data. The final MiST checkpoints improve mean cybersecurity accuracy by +13.1 and +8.6 absolute percentage points over the corresponding Qwen baselines for 8B and 32B, respectively, corresponding to relative gains of +27.0% and +15.8%. Ablation results further show that these cybersecurity gains arise in the mid-training and supervised fine-tuning stages through a combination of the synthetic data generation flows. Furthermore, we show that MiST provides a stronger initialization for downstream task-specific fine-tuning adaptation and reinforcement learning.

1 Introduction

Cybersecurity LLMs must interpret specialized, changing, context-dependent security information, yet general-purpose training may not provide these capabilities and high-quality domain corpora are scarce. MiST uses a compact expert-vetted seed corpus and synthetic data for mid-training, and reports strong benchmark results for its 8B and 32B models.

1 Introduction

  • General-purpose LLMs may struggle with specialized terminology, changing attack techniques, and context-dependent operational knowledge; high-quality cybersecurity pre-training corpora remain scarce.Privacy-sensitive environments may also require on-premises deployment, making reliance on external API models impractical.Source: §1 · p. 1
  • MiST treats mid-training as a domain-specific bridge between pre-training and post-training, built from a small, high-quality seed dataset extended through diverse synthetic pipelines.This emphasizes training-data composition rather than simply scaling the number of raw domain tokens.Source: §1 · pp. 1–2
  • MiST models substantially improve over their Qwen base models, outperform similarly sized cybersecurity models, and are competitive with larger LLMs on public cybersecurity benchmarks.Figure 1 compares MiST and corresponding Qwen accuracy at 8B and 32B, with relative improvements annotated.Source: §1 · p. 2, Figure 1 · p. 2
  • MiST frames mid-training as a corpus-design problem, transforming compact expert artifacts into an intermediate training distribution rather than relying on more raw domain text.Relevant artifacts include vulnerability records, taxonomies, detection rules, and threat-intelligence reports.Source: §1 · p. 2

2 Related Work

Prior cybersecurity LLMs have used raw-domain continued pre-training, cybersecurity-focused instruction tuning, and synthetic supervision. MiST instead transforms a compact expert-vetted corpus into synthetic mid-training data before SFT and preference tuning.

Cybersecurity LLMs

  • MiST uses synthetic transformations of a compact expert-vetted corpus as mid-training data before SFT and preference tuning, unlike prior approaches centered on raw-corpus CPT or cybersecurity SFT.The related-work passage contrasts MiST with PRIMUS and Foundation-Sec-8B, community models such as DeepHat-V1 and Lily-Cybersecurity, and other synthetic-data efforts.Source: §2 · p. 2

Mid-training

  • Mid-training is described as a bridge that moves the model toward the target post-training distribution, with intermediate data closer to the target domain associated with stronger gains.This view treats intermediate-corpus construction as part of what defines mid-training.Source: §2 · pp. 2–3

Cybersecurity LLMs

  • The training pipeline proceeds from general pre-training through cybersecurity mid-training and SFT, then applies DPO for general alignment.The pipeline figure assigns mid-training and SFT cybersecurity-capability roles, while DPO is the final general-alignment stage.Source: Figure 2 · p. 3

Mid-training

  • Prior mid-training work includes synthetic sources and filtered, rewritten pre-training data; MiST similarly rewrites authoritative cybersecurity sources to expose concepts, relations, and security reasoning.Olmo et al. (2025) is described as constructing a corpus of approximately 100B tokens.Source: §2 · p. 3

3 Methodology

MiST builds cybersecurity mid-training data by transforming a compact, varied expert corpus into synthetic learning material, verifying its quality, and mixing in a small amount of general instruction data. The training pipeline then adapts Qwen checkpoints through mid-training, SFT, and preference tuning.

3 Methodology

  • The methodology uses a cybersecurity seed corpus as source material for synthetic data emphasizing knowledge acquisition, concise reasoning, and multi-turn conversation, alongside general data and staged training.Dataset composition and statistics across training stages are reported in Table 12.Source: §3 · p. 4

3.1 Seed Data Curation

  • The seed corpus spans vulnerability and threat intelligence, security knowledge bases and taxonomies, operational artifacts, and defensive guidance and platform documentation.Source-specific transformations address differences in structure, granularity, audience, and information density.Source: §3.1 · p. 4

3.2 Synthetic Data Generation

  • Synthetic generation flows are grouped by stage: mid-training flows target knowledge acquisition and foundational reasoning, while post-training flows target conversational cybersecurity interactions.The flows are grounded in the curated seed corpus and were manually reviewed and refined.Source: §3.2 · p. 4

3.2.1 Flows for Mid-Training Data

  • Mid-training flows produce semantically varied paraphrases, educational artifacts, iterative factual QA, and explanations of structured cybersecurity metadata.The QA process adds two follow-up rounds, and metadata explanations use supplied fields to avoid unguided generation.Source: §3.2.1 · p. 5

3.2.2 Flows for Post-Training Data

  • Post-training flows generate user-assistant dialogues of up to seven turns and persona-based conversations between cybersecurity experts and stakeholders.Personas include security officers, software engineers, red-team operators, and students.Source: §3.2.2 · p. 5

3.2.3 Data Verification and Filtering

  • Generated samples are checked by an LLM across six quality dimensions and removed when their overall score is below 8.The sub-scores cover instruction adherence, task completion, factuality, format and style, relevance, and logical consistency and coherence.Source: §3.2.3 · p. 5

3.3 General Data

  • A 50K-sample subset of general Olmo 3 Dolci SFT data is replayed during mid-training to avoid a domain-only corpus and introduce instruction formatting.The remaining general instruction data is used during post-training.Source: §3.3 · p. 5

3.4 Training

  • MiST adapts Qwen checkpoints through cybersecurity mid-training, SFT, and DPO; MiST-8B initializes from Qwen3-8B-Base, while MiST-32B uses a post-trained Qwen3-32B checkpoint.Mid-training runs for two epochs with a maximum sequence length of 16,384 tokens; Figure 4 reports MiST-8B greatly outperforms other models in its weight class.Source: §3.4 · pp. 5–6, Figure 4 · p. 6

4 Evaluation

The evaluation uses cybersecurity benchmarks spanning security knowledge and applied tasks, alongside general-domain benchmarks to assess forgetting. The authors apply multiple contamination controls and report mean performance across five runs.

4 Evaluation

  • The cybersecurity evaluation covers threat intelligence, advisory settings, security exams and knowledge, vulnerability detection, and applied cyber threat-intelligence tasks.The suite includes CTI-Bench, SECURE, SecEval, MMLU-Cyber, PrimeVul, and AthenaBench.Source: §4 · p. 6
  • General-domain benchmarks are also used to quantify catastrophic forgetting resulting from domain-specific training.Cybersecurity evaluations use representative security tasks.Source: §4 · p. 6
  • Contamination controls audit seed records, avoid multiple-choice synthetic prompts, and apply an identical 13-gram filter across mid-training, SFT, and DPO datasets.The combined mid-training and SFT pipeline removes approximately 3.78% of candidate samples, but the controls cannot rule out paraphrased or semantically equivalent leakage.Source: §4 · pp. 6–7
  • Benchmark scores are averaged over five runs, with non-reasoning inference used when a separate reasoning mode is available.The Table 1 mean is the unweighted average across benchmark columns.Source: §4 · p. 7, Table 1 · p. 7

5 Results

MiST improves cybersecurity benchmark performance while largely preserving general capabilities and providing a stronger initialization for downstream specialization. Results compare the models with Qwen baselines, raw continual pre-training, and task-specific adaptation.

5.1 Cybersecurity Results.

  • MiST-8B-SFT scores 62.4 versus 48.6 for Qwen3-8B, while MiST-32B-SFT scores 62.9 versus 54.6 for Qwen3-32B.The 8B improvement is +13.8 percentage points (+28.4% relative); the 32B improvement is +8.3 percentage points (+15.2% relative).Source: §5.1 · p. 7
  • MiST-8B-DPO gains +13.1 percentage points (+27.0% relative) and MiST-32B-DPO gains +8.6 percentage points (+15.8% relative) over their corresponding Qwen baselines.The results attribute the main cybersecurity improvement to mid-training and SFT rather than DPO.Source: §5.1 · p. 7
  • Gains are broad across cybersecurity benchmarks, strongest on CTI-Bench and Athena, and smaller but generally positive on SECURE and PrimeVul.SFT checkpoints account for nearly all final cybersecurity performance; general-preference DPO largely retains capabilities acquired earlier.Source: §5.1 · pp. 7–8

5.2 General Benchmarks.

  • MiST largely retains or improves performance on general non-cybersecurity benchmarks; the authors hypothesize that general instruction data contributes to this result.The evaluated benchmarks cover reasoning, instruction following, and knowledge.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Mid-training vs. Raw CPT

  • MiST outperforms raw CPT on nearly all benchmarks despite using fewer training tokens.The raw-CPT corpus contains approximately 2.4B tokens, about twice the size of the mid-training corpus.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4 Downstream Task Adaptation

  • Downstream adaptation is evaluated in two settings: reinforcement learning with verifiable rewards and supervised fine-tuning.Source: §5.4 · p. 8

5.4.1 Task-Specific Reinforcement Learning

  • Across three verifiable cybersecurity tasks, RL from MiST-8B-DPO reaches higher validation accuracy throughout training while maintaining lower KL divergence than Qwen3-8B.The authors suggest mid-training moves MiST closer to the downstream task distribution before RL begins.Source: Figure 6 · p. 8, §5.4.1 · pp. 8–9

5.4.2 Task-Specific SFT

  • Task-specific SFT yields larger gains for MiST than for Qwen at both model scales under direct and chain-of-thought prompting.The authors report this pattern on the held-out PrimeVul paired test split.Source: §5.4.2 · p. 9

6 Conclusion

MiST combines a curated synthetic mid-training corpus with cybersecurity fine-tuning to produce 8B and 32B models. The paper reports stronger benchmark results than open cybersecurity models and stronger downstream adaptation than Qwen.

6 Conclusion

  • MiST comprises 8B and 32B models built with a data curation and generation recipe that constructs mid-training corpora from information-dense data.The approach frames corpus construction around data quality and structure rather than token volume alone.Source: §6 · p. 9
  • MiST outperforms existing open cybersecurity models, preserves general capabilities, and matches similarly sized proprietary models.The curated mid-training approach is also reported as more effective than continual pre-training on substantially larger raw domain corpora.Source: §6 · p. 9
  • MiST adapts more effectively than Qwen under task-specific SFT and reinforcement learning, with consistently larger PrimeVul SFT gains across scales and prompting modes.In reinforcement learning, MiST achieves higher validation accuracy with lower KL movement.Source: §6 · p. 9, Figure 7 · p. 9

Limitations

The evaluation primarily covers knowledge-intensive cybersecurity understanding, threat intelligence, vulnerability reasoning, and benchmark-style analysis. The paper also identifies limits in operational coverage, RL task breadth, synthetic-data analysis, and contamination control.

Limitations

  • Evaluation focuses on knowledge-intensive security understanding, cyber threat intelligence, vulnerability reasoning, and benchmark-style analysis, not comprehensive operational settings.The results indicate improved cybersecurity knowledge and structured analysis, not deployment readiness in live security operations.Source: Limitations · p. 9
  • RL experiments cover only a small set of structured cybersecurity tasks and omit broader recipes, tool use, and agentic reinforcement learning.The paper calls for testing whether mid-training benefits extend to more complex multi-step security tasks.Source: Limitations · pp. 9–10
  • MiST relies heavily on LLM-generated synthetic data, while the effects of generator choice, prompts, filtering thresholds, and source mixtures remain unquantified.The generator and verifier share a model family and may therefore share blind spots.Source: Limitations · p. 10
  • Contamination cannot be ruled out because exact- and near-match filtering may miss paraphrased or semantically equivalent benchmark content.The paper notes particular overlap risks around public CVEs, CWEs, ATT&CK techniques, and vulnerability descriptions.Source: Limitations · p. 10

Ethical Considerations

MiST uses public and third-party cybersecurity resources and treats synthetic outputs as governed derivatives. The paper frames release and deployment around provenance, sensitive-content screening, dual-use risks, and safeguards.

Ethical Considerations

  • Synthetic samples may preserve or amplify errors and sensitive details from their source material, so MiST treats them as governed derivatives.Public availability does not necessarily grant unrestricted permission to redistribute source text or derived examples.Source: Ethical Considerations · p. 10
  • MiST is intended for defensive research, but its capabilities may lower barriers to harmful use and preliminary safety evaluation does not establish robustness to adaptive attacks.The paper recommends human oversight, access controls, logging, monitoring, and domain-specific review, especially in live operations.Source: Ethical Considerations · p. 10

A Estimated Training Time

The appendix reports training costs and hyperparameters, then details the general and cybersecurity benchmarks, evaluation protocol, and prompting strategies used to assess MiST.

A Estimated Training Time

  • 7.1 hours (56.8 GPU-hours) for MiST-8B and 42.7 hours (341.3 GPU-hours) for MiST-32B; SFT accounts for the largest share of compute.Estimates are for the full pipeline on one node with 8×B200 GPUs.Source: A Estimated Training Time · p. 14, Table 3 · p. 14

B Training Hyperparameters

  • All stages use two epochs, AdamW, cosine learning-rate scheduling, bfloat16, and gradient clipping; DPO uses β = 0.1 and early stopping after 200 steps.The 32B model uses ZeRO-3 for memory reduction; other mid-training settings match the 8B model.Source: B Training Hyperparameters · p. 14, Table 4 · p. 15

C General Benchmarks Details

  • General benchmarks assess broad capabilities and potential specialization-related degradation, including instruction following, general knowledge, science reasoning, and multistep arithmetic.IFEval measures constraint adherence; MMLU, ARC-Challenge, and GSM8K probe knowledge, science reasoning, and math, respectively.Source: C General Benchmarks Details · pp. 14–15

D Cybersecurity Benchmark Details

  • The 13 cybersecurity tasks span CTI knowledge and mapping, critical-infrastructure advice, cybersecurity questions, paired vulnerability detection, and applied threat-intelligence reasoning.The suite includes CTI-Bench, SECURE, SecEval, MMLU Computer Security, PrimeVul, and AthenaBench.Source: D Cybersecurity Benchmark Details · p. 15, E Evaluation Protocol · p. 16

E Evaluation Protocol

  • Evaluations disable separate reasoning modes when available, run each benchmark five times with different seeds, and report the mean.The inference setup uses a 16,384-token context length and a maximum generation length of 4,096 tokens.Source: E Evaluation Protocol · p. 16
  • Scoring extracts benchmark-formatted answers, uses task-specific metrics, and averages benchmark scores without weighting; CTI-MCQ subject subscores are excluded from suite averages.The protocol uses exact match for GSM8K and CTI-RCM, the official loose IFEval metric, and accuracy for other benchmarks.Source: E Evaluation Protocol · p. 16

E.1 Prompting Strategies

  • Custom prompts specify response formats for MMLU, SecEval, and other benchmarks when official prompts are unavailable or multiple configurations exist.For ARC-Challenge, the prompt requests step-by-step reasoning followed by an answer in the specified format.Source: §E.1 · p. 16

F Ablation Studies

Ablations attribute cybersecurity gains to mid-training and complementary synthetic-data flows, while the appendix also documents training-data composition, decontamination, blog curation, and task-specific reinforcement learning.

F Ablation Studies

  • Mid-training raises mean cybersecurity score from 57.6 to 62.4 (+4.8 points) with SFT fixed; base initialization also scores higher than post-trained initialization after SFT and DPO.The checkpoint comparison is 62.4 vs. 60.6 after SFT and 61.7 vs. 60.4 after DPO.Source: F Ablation Studies · pp. 17–18, Table 8 · p. 18, Table 10 · p. 19

G Cybersecurity Resources and Acronyms; H Training Data Composition

  • The seed corpus and synthetic pipeline draw on resources spanning vulnerability records, abstract security concepts, and operational artifacts; the mid-training split is primarily synthetic, while SFT mixes cyber and general instruction data.The SFT mixture is intended to preserve specialization and general instruction-following ability.Source: G Cybersecurity Resources and Acronyms · p. 18, H Training Data Composition · p. 18

I Model Identifiers and Hugging Face URLs

J Decontamination

  • Decontamination combines seed-source filtering, constraints on synthetic-data structure, and 13-gram overlap filtering across mid-training, SFT, and DPO.The filter removes a sample when it matches a benchmark n-gram.Source: J Decontamination · pp. 18, 20, 21

F Ablation Studies

  • No single synthetic flow matches the full corpus, removing any flow lowers the mean by 1.4 to 2.8 points, and different flows peak on different benchmarks.The best single-flow variant, paraphrasing at 60.3, remains 2.1 points below the full corpus.Source: F Ablation Studies · pp. 17–18, Table 9 · p. 19

J Decontamination

  • N-gram filtering can miss heavily paraphrased or translated benchmark items; source-level filtering and excluding MCQ-style generation are intended to mitigate residual risks.The full pipeline removes approximately 3.78% of candidate mid-training and SFT samples.Source: J Decontamination · p. 21

K RSS Blogposts Curation

  • An LLM judge filters RSS posts for CTI relevance, reducing the collection from 7,916 to 2,995 posts.The retained subset targets concrete threat, attacker-behavior, and real-world-incident descriptions.Source: K RSS Blogposts Curation · p. 21

L Task-Specific Reinforcement Learning Details

  • GRPO is applied to MiST-8B and Qwen3-8B on three verifiable cybersecurity tasks, using deterministic rule-based rewards without a learned reward model.The tasks are CVE-to-CWE mapping, CVE-to-CVSS vector scoring, and vulnerable-code-to-CWE mapping.Source: L Task-Specific Reinforcement Learning… · p. 22

M Task-Specific SFT Details

The appendix describes PrimeVul task-specific fine-tuning and paired evaluation, reporting larger post-fine-tuning gains for MiST than for Qwen and showing examples of generated training data.

M Task-Specific SFT Details

  • PrimeVul SFT uses real-world C/C++ functions and paired vulnerable/patched examples to test whether models identify security-relevant code differences.Training uses PrimeVul training and validation splits; the held-out paired test split is evaluated with direct and chain-of-thought prompting.Source: M Task-Specific SFT Details · p. 22
  • Paired accuracy counts a pair as correct only when both the vulnerable function and its patched counterpart are classified correctly.This metric is intended to better assess code-level distinction than standard accuracy, which can be misleading under dataset priors or class overprediction.Source: M Task-Specific SFT Details · p. 22
  • After task-specific SFT, MiST-8B-DPO-PrimeVul improves from 10.3 to 16.3 on PRIMEVUL and from 15.2 to 19.2 on PRIMEVUL-COT; MiST-32B-DPO-PrimeVul improves from 8.6 to 16.4 and from 15.3 to 21.3, respectively.Qwen gains only marginally under the same procedure; results report PrimeVul paired accuracy.Source: M Task-Specific SFT Details · pp. 22–23, Table 14 · p. 23

N Data examples

  • The appendix presents representative generated persona-conditioned dialogues, standalone question–answer pairs, and educational content.The examples are associated with Figures 9 and 10.Source: N Data examples · p. 23

O Safety Evaluation

A preliminary safety evaluation compares MiST-8B checkpoints with Qwen3-8B on misuse-oriented cybersecurity prompts. MiST performs at least as well as the baseline in the reported evaluations.

O Safety Evaluation

  • The preliminary evaluation tests whether specialization affects safe handling of misuse-oriented requests, comparing MiST-8B-SFT and MiST-8B-DPO with Qwen3-8B.CyberSecEval MITRE prompts are mapped to the MITRE ATT&CK framework.Source: O Safety Evaluation · p. 23
  • Both MiST checkpoints improve safe-response rate over the Qwen3-8B baseline on the CyberSecEval MITRE benchmark, indicating specialization did not erode safe-handling behavior on this benchmark.Source: O Safety Evaluation · p. 23, Table 15 · p. 23
  • Qualitative red-teaming and an internal misuse-oriented benchmark found MiST comparable to or better than the baseline.Source: O Safety Evaluation · p. 23

Related papers

Loading 2609.18496v1…