PaperlayerView exact revision on arXiv

Source-linked AI summary

Ensembling LLMs for AI-Augmented Cybersecurity Software Requirements Generation

Santiago Perez-Acuna, Yod-Samuel Martín, Juan C. Yelmo

arXiv:2609.10316v1cs.SE

TL;DR

Cybersecurity requirements generation needs broad coverage, but individual LLM runs vary and can hallucinate. The paper applies post-generation information-retrieval fusion to 24 runs, finding that agreement-based and reliability-weighted rankings prioritize valid requirements more effectively than original outputs. The resulting lightweight layer improves coverage and review prioritization without retraining or changing upstream models.

  • Problem

    Individual LLM runs vary across models and stochastic executions, risking missed valid safeguards and plausible hallucinations in recall-sensitive cybersecurity requirements generation.

  • Method

    The study treats 24 black-box LLM runs as retrieval systems and applies Uniform voting fusion plus Naive-Bayes weighting over a frozen expert-labeled corpus.

  • Results

    Uniform fusion improves ranking over original outputs, while Naive-Bayes weighting further refines it; gains remain positive in at least 92% of out-of-bag resamples.

  • Takeaways & Limitations

    Post-generation fusion provides a lightweight review-prioritization layer that preserves diverse candidate coverage while ranking likely valid requirements ahead of hallucinations.

  • Takeaways & Limitations

    Naive-Bayes weights rely on practical conditional-independence and shared-rate assumptions, so the resulting rule is not necessarily optimal for the corpus.

Abstract

from arXiv · show

Translating high-level controls from security standards into concrete, system-specific requirements is central to cybersecurity requirements engineering. Large language models (LLMs) can accelerate this labor-intensive, recall-sensitive task, but any single run is unreliable: it misses valid safeguards while introducing plausible hallucinations, and outputs shift across runs and models. We reframe this variability as a resource: rather than selecting one output, we study post-generation ensembling, aggregating stochastic runs with information-retrieval data-fusion operators. We propose two strategies: Uniform fusion rewards mere cross-run agreement, whereas Naive-Bayes fusion weights each run by its estimated reliability. We evaluate both over 24 runs from 12 configurations across four model families, generated for ten ISO/IEC 27002:2022 controls and expert-judged against a gold standard of 72 valid requirements. Pooling every run's output recovers all 72 (whereas single configurations recover on average under half) but also 111 hallucinations. Fusion separates the wheat from the chaff, ranking valid requirements well ahead of hallucinations. In the areas under the precision-recall and ROC curves, Uniform fusion alone largely surpasses every original run and configuration by 0.142 and 0.118 over the best configuration. Naive-Bayes weighting adds a further 0.039 and 0.052, reaching 0.864 and 0.869 while attaining useful operating points earlier. Internal validation confirms the stability of these gains: they stay positive in at least 92% of out-of-bag bootstrap resamples and every structured-perturbation sample. Post-generation fusion thus turns apparent noise into a practical asset: a lightweight layer giving analysts broader coverage and a better prioritized review queue, using affordable, below-frontier models alone.

I. INTRODUCTION

LLM output variability creates both coverage opportunities and review risks in cybersecurity requirements engineering. This study treats that variability as complementary evidence, using post-generation fusion to improve prioritization without changing upstream generation.

  • LLM outputs vary across stochastic runs, models, prompts, and decoding settings, making single-run evaluation insufficient.
  • Recall matters because missed requirements can leave protection gaps, while plausible nonrelevant requirements are generally easier to reject during review.
  • The contribution addresses a gap in studies that commonly evaluate single runs despite recognized stochasticity, prompt sensitivity, and hallucinations.
  • The study reframes run and model diversity as complementary candidate evidence that can improve recall while containing invalid requirements.
  • Post-generation ensembling combines frozen LLM outputs as retrieval-style requirement sets using agreement-based voting and reliability-aware Naive-Bayes weighting.
  • The evaluation covers 12 configurations across four model families against an expert-derived gold standard of ISO/IEC 27002 instantiated requirements.

B. LLMS FOR DEFENSIVE CYBERSECURITY

Prior cybersecurity and requirements-engineering research has largely focused on immature, post-development, or closed-ended applications rather than open-ended security-requirements generation. This study adapts information-retrieval data fusion to aggregate black-box LLM outputs for that underdeveloped task.

  • Cybersecurity LLM research is dominated by post-development, code-centric artifacts, while requirements-stage activities are absent from one surveyed task taxonomy.
  • Existing security benchmarks often use closed-ended questions, whereas open-ended analyses requiring specialized security knowledge remain harder for LLMs.
  • Security-requirements research remains emerging and lacks standardized evaluation metrics and benchmarks, hindering reproducibility and consolidation.
  • Earlier work found that different runs retrieved different valid subsets and that hallucinations were tied to particular runs or models rather than systematic.
  • The approach treats each complete LLM response as a black-box retrieval output and applies voting-based data fusion after generation.

A. TASK DESCRIPTION AND SOURCE CORPUS

The study reuses a frozen, expert-judged corpus generated by instantiating ISO/IEC 27002:2022 controls for a realistic industrial system. It evaluates candidate prioritization as closed-world retrieval, emphasizing recall relative to review burden.

  • A. TASK DESCRIPTION AND SOURCE CORPUS: The generation pipeline combines applicability checks, placeholder mapping, free-text requirement generation, and JSON formatting across ten annotated control templates.
  • A. TASK DESCRIPTION AND SOURCE CORPUS: Each configuration fixes the model, prompt pipeline, and settings, while stochastic decoding makes repeated runs produce different candidate sets across four model families.
  • A. TASK DESCRIPTION AND SOURCE CORPUS: Candidates are pooled, decomposed into atomic requirements, canonicalized, and expert-adjudicated as valid or hallucinated in a frozen corpus.
  • B. JUDGED UNIVERSE AND CONTINGENCY-TABLE METRICS: The judged universe contains 183 candidates, including 72 gold-standard valid requirements, and evaluates only candidates produced by participating sources.
  • B. JUDGED UNIVERSE AND CONTINGENCY-TABLE METRICS: The evaluation uses a closed-world retrieval setup in which valid requirements are relevant items and hallucinations are nonrelevant items.
  • B. JUDGED UNIVERSE AND CONTINGENCY-TABLE METRICS: F2 and weighted Youden metrics encode a 4:1 preference for avoiding missed valid requirements over false positives.

C. ENSEMBLING AS VOTING-BASED DATA FUSION

Post-generation ensembling converts unranked candidate sets from black-box LLM runs into a single ranked review queue. The ranking rewards cross-run agreement and can additionally weight sources by estimated reliability.

  • Each run casts a binary vote for each candidate, and weighted votes produce a fused score over the candidate universe.
  • Sorting fused scores creates a ranked list, allowing analysts to review candidates by prefixes determined by budget or acceptance thresholds.
  • Uniform agreement implements the chorus effect, while reliability-aware weights give greater influence to runs that better separate valid requirements from hallucinations.

IV. IMPLEMENTATION

The implementation converts unranked LLM requirement outputs into a scored ranking over the judged universe using either uniform agreement or reliability-aware Naive-Bayes fusion. Uniform fusion rewards cross-run support, while Naive-Bayes weights model votes using estimated validity-related rates under practical assumptions.

  • Fusion setup: Fusion converts binary run outputs into a single scored ranking over the judged candidate universe U.Candidates are ranked by decreasing CombSUM-style support from run-level votes.
  • Uniform fusion: Uniform fusion assigns equal weight to every run and ranks candidates by the fraction of runs that generated them.Its normalized score depends only on cross-run support.
  • Naive-Bayes fusion: Naive-Bayes fusion estimates model-specific vote reliability and treats run votes as conditionally independent repeated measurements of candidate validity.The model uses true-positive and false-positive rates estimated from expert-labeled candidates.
  • Assumptions and interpretation: 26?
  • Weight estimation: Jeffreys regularization prevents zero or unit rate estimates that would otherwise create infinite log-odds weights.The correction adds half-counts to rate numerators and one to denominators.
  • Assumptions and interpretation: The Naive-Bayes score has a posterior-log-odds interpretation over the judged universe, with absent votes contributing evidence through the closed-universe formulation.Ranking by the score is equivalent to ranking by the model’s posterior validity probability.

V. EVALUATION PROTOCOL

The evaluation compares fusion rules with original LLM outputs over the judged universe, balancing valid-requirement coverage against hallucination burden. Metrics are evaluated across review depths so performance can be compared at different operating points.

  • Evaluation criteria: Evaluation measures valid-requirement coverage with recall and hallucination burden with precision and fall-out or specificity.The combined criteria F2 and Jw give greater relevance to recall.
  • Evaluation criteria: Metrics vary with review depth k, producing @k operating points rather than a single fixed evaluation cutoff.This allows fusion strategies to be compared across compliance-relevant review depths.

A. RANKED OUTPUTS AND @k OPERATING POINTS

The protocol treats ensembles as ranked retrieval systems and single runs as binary retrieval systems, then evaluates both across review-depth cutoffs and precision–recall and ROC curves. Two fixed review depths anchor interpretation, while curve maxima summarize recall-sensitive operating quality.

  • Ranked outputs: Ensembles produce ranked candidate lists from fusion scores, whereas single runs form a tied generated block above their non-generated remainder.The single-run representation encodes membership rather than an inherent LLM ordering.
  • @k operating points: At cutoff k, larger review depths generally increase recall while admitting more hallucinations and reducing precision and specificity.Each cutoff induces TP, FP, FN, and TN counts for operating-point metrics.
  • Fixed review depths: kM = 40 is the largest original LLM output size, while kR = 72 equals the number of valid requirements and marks the earliest possible full-recall rank.These fixed depths provide common review-budget reference points.
  • Operating-point summaries: The protocol reports maximum F2@k and Jw@k values together with the first rank at which each maximum occurs.The maximizing rank indicates how early the selected recall-sensitive operating point is reached.
  • Global curves: The precision–recall curve relates precision to recall, while the ROC curve relates recall to fall-out and is insensitive to class prevalence.AP summarizes early high-precision retrieval, and ROC AUC summarizes pairwise separation of valid and hallucinated candidates.

C. TIE HANDLING

The protocol handles tied scores by averaging over uniformly random within-block orderings, yielding deterministic expected operating-point metrics and dedicated expected AP and ROC AUC calculations. Configuration baselines use distinct micro- and macro-aggregation rules because no single pooled configuration ranking exists.

  • Tie-aware evaluation: Ties are handled by uniformly randomizing item order within each equal-score block and computing expected metric values.This avoids arbitrary document-ID effects and makes evaluation deterministic and reproducible.
  • Operating-point metrics: For a tie block with t items and v valid items, each position contributes expected valid mass v/t and hallucinated mass (t − v)/t.These fractional counts support exact expected precision, recall, and F-score values at cutoffs inside a block.
  • Global metrics: Expected AP and ROC AUC require dedicated tie-aware calculations because they are not affine in the contingency-table counts.Tied valid–hallucinated pairs receive half credit for expected ROC AUC.
  • Single-run baselines: Single runs are evaluated as two-block rankings, with generated candidates above non-generated candidates and ties handled by the same model.Their ROC AUC reduces to (TPRr + TNRr)/2.
  • Configuration baselines: Configuration ROC and operating-point summaries use micro-averaged counts, whereas configuration AP is the macro-average of single-run AP values.These are separate aggregates because no pooled configuration-level ranking exists for AP.
  • Configuration baselines: The configuration PR point and APc are deliberately separate aggregates rather than quantities derived from one another.This follows from combining micro-averaged rates with macro-averaged per-run AP.

E. BENCHMARKS: REFERENCE CURVES AND NORMALIZED UPLIFT

The benchmarks define reference curves for interpreting fusion rankings and use normalized uplift to measure progress from random ordering toward an idealized linear-fusion benchmark. Internal validation tests whether these comparisons remain stable under resampling and structured perturbations, while noting limits on external generalization.

  • Reference curves: Four reference curves bound fusion results by finite-dataset feasibility, random ranking, and additional benchmark perspectives.The feasible region reflects limits imposed by |V| valid requirements and |H| hallucinated candidates; all four curves accompany the fusion curves in Fig. 3.
  • Normalized uplift: Normalized uplift measures how much of the gap between random ordering and the numerical linear-fusion benchmark a ranking approach recovers.The approach may be a fusion strategy or a baseline such as the best single configuration.
  • Internal validation: The primary internal-validation analysis uses 200 requirement-level out-of-bag bootstrap replicates with refitting.It evaluates whether learned-weight advantages persist on unseen candidates from the same judged universe.
  • Scope and assumptions: The validation does not separately test transfer to unseen control templates, and percentile bands are descriptive rather than classical confidence intervals.Naive tests across overlapping replicates are anti-conservative; analytic intervals also rely on an independence idealization that ignores candidate clustering by control template.
  • Internal validation: The OOB procedure is conservative because reduced bootstrap training samples pessimistically bias performance estimates, making OOB values lower bounds on corresponding full-data performance.Across replicates, the procedure averages about 116 unique in-bag and 67 out-of-bag candidates.
  • Internal validation: Fixed-weight perturbation analysis drops one control slice and one run at a time to test whether results depend on particular templates or idiosyncratic runs.The perturbations combine leave-one-out analysis over runs with ex-ante control slices.

VI. RESULT ANALYSIS

Uniform fusion improves the review ordering over original LLM outputs, while Naive-Bayes weighting further improves ranking quality and reaches useful operating points earlier. Global metrics and workload-aligned analyses show gains across precision, recall, fall-out, and recall-sensitive utility measures.

  • Across review depths: Uniform fusion improves the ordering of candidate requirements over every original output across review-depth and global-ranking views.It improves precision at a given recall, recall at a given precision, fall-out, F2, and Jw at comparable review depths.
  • Across review depths: At kM = 40, Uniform fusion improves P@kM, R@kM, FPR@kM, F2@kM, and Jw@kM relative to the best configuration.The deeper representative operating point is kR = 72, the number of valid requirements in the gold standard.
  • Across review depths: Naive-Bayes fusion adds a consistent improvement over Uniform by shifting rankings toward higher precision, lower fall-out, and better recall-sensitive metrics.The comparison holds across the practically relevant range of review depths.
  • Global ranking quality: Uniform fusion recovers 86.2% of the AP gap and 76.9% of the ROC AUC gap, compared with 56.8% and 48.4% for the original configuration.These normalized uplifts measure progress from expected random ordering toward the numerical linear-fusion benchmark.
  • Global ranking quality: Naive-Bayes fusion reaches normalized uplifts of 94.1% for AP and 89.6% for ROC AUC.The gains over Uniform are moderate but consistent across PR and ROC views, improving early placement and pairwise separation.
  • Global ranking quality: Naive-Bayes reaches a higher maximum F2 of 0.831 at k = 91, whereas Uniform reaches 0.764 only at k = 183.For Jw, the corresponding values are 0.679 and 0.489 at comparable review depths.

VII. DISCUSSION: IMPACT, STABILITY, AND SCOPE

Post-generation ensembling improves coverage and prioritization for recall-sensitive security requirements engineering, while internal validation and scope analysis clarify where the gains apply. Uniform fusion already strengthens ordering, and Naive-Bayes weighting further improves review efficiency.

  • Impact: All 72 gold-standard requirements are recovered by the ensemble, whereas the best single configuration covers less than half on average.Pooling expands coverage, although additional candidates must still be reviewed for false positives.
  • Impact: Ensembling converts run variability into ranked evidence, placing supported valid requirements earlier within bounded review budgets.The fused ordering addresses the practical constraint of how many candidate requirements analysts can inspect.
  • Impact: Uniform fusion outperforms every original configuration, while Naive-Bayes adds reliability-weighted gains and reaches its higher maximum F2 at roughly half the review depth.The additional weighting refines the ordering beyond simple cross-run agreement.
  • Stability: Naive-Bayes’ earlier ascent matters because shared peak values can conceal ordering advantages under internal validation.Peak utility alone understates differences in how quickly useful candidates appear.
  • Scope: Uniform fusion cannot distinguish candidates supported by the same vote count, whereas per-model Naive-Bayes weights partially resolve single-vote ties.Candidates with identical vote patterns remain inseparable to vote-based rules.
  • Scope: The approach requires overlapping, canonicalizable candidate sets from several LLM runs and cannot recover requirements absent from every run.Its applicability therefore depends on both repeated generation and candidate comparability.
  • Process impact: Fused rankings can support SQUARE by enlarging standards-linked candidates during elicitation and adding evidence-of-validity during prioritization.At inspection, the ranking provides a queue whose depth can match the available review budget while final acceptance remains human.
  • Scope: The study’s models are not frontier models as of 2026, leaving the portability of the reported advantage across future model tiers as an open scope boundary.The authors identify cheaper, lower-tier, or open-weight ensembles as a practical opportunity rather than an established result.

B. INTERNAL VALIDATION AND CONSTRUCT STABILITY

Internal validation shows that fusion gains persist across resampling and structured perturbations, while the study’s conclusions remain bounded by corpus, metric, sampling, and temporal constraints.

  • Bootstrap stability: Uniform fusion improves on the best single configuration across all four metrics in every bootstrap replicate.The baseline is reselected within each replicate, making this a conservative comparison.
  • Bootstrap stability: Naive-Bayes fusion beats Uniform fusion with win rates of 99.5% for AP, 99.0% for ROC AUC, 100% for max Jw, and 92.5% for max F2.The max F2 p05 reaches zero because some replicates tie.
  • Effect magnitude: 0.158 is Naive-Bayes fusion’s mean out-of-bag AP advantage over the best single configuration, positive in every replicate.This is the total advantage across both fusion improvements.
  • Structured perturbations: Naive-Bayes fusion outperforms Uniform fusion on all four metrics in every one of 240 control-slice/run perturbation samples.The result indicates that global ordering and weighted recall–specificity balance are not driven by one slice or run.
  • Scope and validity: The evaluation covers one English-language case study with ten ISO/IEC 27002 controls, fixed prompts, decoding settings, model snapshots, and an expert-constructed gold standard.Absolute trade-offs and gain magnitudes may vary across systems, domains, organizational contexts, standards, and annotation decisions.
  • Scope and validity: The base chain received four stochastic runs while cooler and warmer variants received one, so run counts may confound temperature-driven differences.Equal-run replications or subsampling sensitivity analyses would strengthen causal claims.

VIII. CONCLUSION

The study frames post-generation aggregation of LLM cybersecurity requirements as an information-retrieval fusion problem. Across its evaluated configurations, fusion improves review prioritization, with reliability-aware weighting adding stable gains while remaining bounded to the studied corpus.

  • Conclusion: The study evaluates voting- and quality-weighted ranking rules across 12 configurations from four model families.Each LLM run is treated as a black-box retrieval system over a canonicalized requirement universe.
  • Conclusion: Uniform fusion increases recall without a matching hallucination rise, while Naive-Bayes fusion further improves AP and ROC AUC.Naive-Bayes weighting discounts support from less reliable sources and reaches comparable coverage at shallower review depths.
  • Conclusion: Fusion gains remain positive in at least 92% of out-of-bag resamples and every structured perturbation.This supports robustness over the evaluated corpus rather than external generality.
  • Practical implication: The proposed workflow leaves upstream models and prompts untouched and adds no retraining or finetuning.It combines multiple stochastic runs across heterogeneous models and prompts with voting and reliability-aware weighting.
  • Future work: Future work should test additional standards, system artifacts, and languages, add control-specific checks, and integrate fusion into analyst workflows.These directions address generality, plausible off-scope consensus, and dynamic review selection.

APPENDIX. REPRODUCIBLE RESEARCH

The study’s evaluation is reproducible from an archived frozen corpus, code, metrics, curves, and result tables. All reported fusion operates offline on recorded per-run candidate presence.

  • Archived artifacts: The Zenodo deposit archives the frozen candidate corpus, ensembling code, computed metrics, curves, and result tables.It is separate from the source study’s reproducible-research package.
  • Execution model: All ensembling is offline post-processing over the frozen corpus, with no new LLM calls or regenerated requirements.Strategies operate only on recorded per-run presence for each candidate.
  • Repository scope: The article reports Uniform and Naive-Bayes fusion with core metrics, while the broader toolkit contains additional strategies, metrics, and optimizers.The repository README maps internal names to the article’s labels and documents the remaining functionality.
Loading 2609.10316v1…