Research Methodologies for Cybersecurity in Enterprise Environments: A Narrative Review, Synthesis and Executable Guide

I. INTRODUCTION; A. The problem this paper addresses

• Different methods measure different outcomes, so studies may answer unasked questions, fail to transfer beyond benchmarks, or produce organizational conclusions that cannot be checked.The field spans technical, operational, human-factors, and governance research; critical-infrastructure cyber events can propagate into physical disruption,.Source: A. The problem this paper addresses · p. 1, §I · p. 1

B. Scope

• The review focuses on study design, conduct, evaluation, and reporting in enterprise and organizational settings, including cyber-physical enterprises as proxies where a general enterprise population is inaccessible.It does not impose or characterize a publication-date window, and it cautions that a selective narrative review cannot establish comprehensive coverage for a date range.Source: B. Scope · p. 2

C. Contributions

• The paper contributes an eleven-family taxonomy, executable protocols and visual maps, three study recipes, and a worked analysis of inconsistent intrusion-detection rankings.It also provides a reference corpus verified against source registries; the protocols specify steps, instruments, evaluation criteria, validity threats, failure modes, and reporting checklists.Source: C. Contributions · p. 2

D. How to read this paper

• The protocols are intended to be followable: each has ordered, gated steps with required outputs, and its figures and checklists support study design and review.Figures use blue for steps, amber for decisions, green for artifacts, red italic for validity threats, and dashed arrows for iterations; readers can also consult Section III for method selection and Section XVI or Appendix A for review.Source: D. How to read this paper · p. 2

II. HOW THIS REVIEW WAS CONDUCTED; A. Review type, and why this one

• The narrative synthesis suits a question spanning incommensurable research traditions because no common outcome measure or risk-of-bias instrument applies across the included study types.The review follows documented-search and reporting guidance,,,,,, while disclaiming completeness, exhaustive partitioning by its eleven families, and validated-instrument status for its confidence labels.Source: A. Review type, and why this one · pp. 2–3, §II · p. 2

B. Sources and retrieval; C. Inclusion criteria

• The review searched OpenAlex, Semantic Scholar, and arXiv with one methodology facet per query, and included works meeting security, enterprise relevance, methodological explicitness, detail, and stated-evaluation criteria.The paper acknowledges that it does not report the complete query record, dates, or screening counts and says the full search record should be released as supplementary material.Source: B. Sources and retrieval · p. 3, C. Inclusion criteria · p. 3

D. Verification procedure

• 129 of 151 works were verified against DOI records and 22 against arXiv records; the checks caught two identifier failures, and all 41 works added in the second search round checked out.References were checked against source metadata, and seed-synthesis numeric claims against publisher abstracts; the paper uses issue dates for five references with a one-year online-first discrepancy.Source: D. Verification procedure · p. 3
• The verified corpus spans 1977–2026, has a median publication year of 2020, and includes 58 works from 2023 onward; these counts describe the review’s selection, not the field’s composition.Table I records the corpus composition, whose counts serve as provenance for frequency characterizations in the family sections.Source: Table I · p. 4, D. Verification procedure · p. 4

E. Synthesis

• The synthesis did not pool effect estimates because outcomes were incommensurable, and it assigns Strong, Moderate, or Limited confidence using structured editorial judgment rather than GRADE scores.Strong denotes consistent findings across multiple independent studies; Moderate reflects contextual dependence, expert input, or single-site evidence; Limited reflects one study, formative evaluation, or population mismatch.Source: E. Synthesis · p. 4

F. Limitations of this review

• The review is selective rather than exhaustive and has six limitations, including database and publication selection, no formal risk-of-bias instrument, incomplete full texts, and single-coder extraction.Its family frequencies are not counted or reliability-tested estimates; claims based on abstracts or structured records are capped at Moderate. Retrieval, inclusion criteria, verification, and known under-coverage are reported to make selection inspectable,.Source: F. Limitations of this review · p. 4

G. Audit of the antecedent synthesis

• All 49 antecedent references checked out, and four load-bearing numeric or methodological claims matched publishers’ abstracts,,,.These included 98% accuracy for convolutional and recurrent models versus 99.9% for random forest; an ensemble approaching 100% on four metrics under stated conditions; a four-indicator framework from 32 papers; and a four-week experience-sampling design.Source: G. Audit of the antecedent synthesis · p. 4
• The antecedent’s 400 identified, 200 screened, 150 excluded, and 50 included figures were unsupported, implying no full-text exclusions; the review does not restate them as its own retrieval.The paper notes that clean references do not validate a process narrative; the antecedent’s limitations section also conceded that stated totals exceeded the records whose data were presented.Source: G. Audit of the antecedent synthesis · pp. 4–5

A. Eleven families; B. Selection procedure

• Researchers should classify the intended claim, use Table II’s licensed-claim guidance, check data access, consider technical and organisational triangulation, and pre-specify what would count as failure.If no method family survives the access check, the claim must be renegotiated.Source: B. Selection procedure · p. 5, A. Eleven families · p. 5, Table II · p. 6

A. Purpose and when to use it

• Secondary research addresses the state of knowledge; systematic reviews appraise focused questions, mapping studies categorise broader fields, and SoKs reorganise fields around conceptual contributions –[27].The paper identifies secondary research as a landscape-building method across several cybersecurity topics, –.Source: A. Purpose and when to use it · p. 5

B. Protocol

• The review protocol specifies the question, registers search and extraction plans, searches multiple databases and preprint servers, snowballs, screens, extracts, synthesises, and reports against PRISMA 2020.It calls for recorded search details, two-stage screening with sample double-screening, exclusion reasons, piloting the extraction form, and an auditable synthesis.Source: B. Protocol · pp. 5–6

C. Instruments

• PRISMA 2020, Kitchenham, Petersen, and ROSES are reporting standards; PRISMA-P supports protocol preparation, while PRISMA-ScR suits scoping reviews,, –.The paper treats standard selection as part of research design, not merely a submission-stage formatting choice.Source: C. Instruments · p. 6

D. What the evidence supports; E. Failure modes •

• Secondary research is strong for mapping methodological landscapes but only moderately supports comparative-effectiveness conclusions; review pitfalls include unreconciled flows, vote counting, single-database retrieval, and incommensurable outcomes.Network-intrusion-detection reviews repeatedly find dataset, metric, and reproducibility variation that obstructs comparison,,.Source: D. What the evidence supports · pp. 6–7, E. Failure modes • · p. 7

F. Checklist

• The checklist calls for a pre-search protocol, verbatim queries and dates, multiple databases plus preprints, snowballing, dual screening with agreement statistics, and recorded full-text exclusions.Source: F. Checklist · p. 7
• Checklist items also require a reconciling PRISMA flow, a piloted extraction form, and a clear distinction between review limitations and limitations of included studies.Source: F. Checklist · p. 7

A. Purpose and when to use it

• Design science suits contributions that create an artifact to address a class of enterprise-security problems and ask whether it works.Examples include maturity models, threat-modeling languages, risk frameworks, testbeds, and training platforms –.Source: A. Purpose and when to use it · p. 7

B. Protocol

• The protocol moves from problem identification and measurable objectives through artifact design, demonstration, evaluation, and communication, distinguishing formative improvement from summative testing.Qualitative objectives can weaken evaluation because they leave no clear way for the artifact to fail; the formative loop back to design shows feasibility, not effectiveness.Source: B. Protocol · p. 7, Figure 3 · p. 7

C. Evaluation designs, in increasing strength

• Design-science evaluation ranges from illustrative scenarios and expert review through real-organisation cases, shared-task comparisons, and randomized controlled experiments, with controlled experiments strongest but rare.The passage ranks these designs in increasing strength; expert, case, comparative, and controlled designs each provide different evaluation evidence.Source: C. Evaluation designs, in increasing… · p. 7

D. What the evidence supports

• Design science is moderately supported for enterprise-security frameworks, models, and platforms, but operational-effectiveness claims remain limited because evaluations are mostly formative, expert-based, or single-case.The review says operational-effectiveness claims need evaluation rung 4 or 5, while the corpus mostly uses rungs 2 and 3; one artifact study reported formative evaluation with ten professionals across three financial institutions and no summative evaluation.Source: D. What the evidence supports · pp. 7–8, Figure 4 · p. 8

E. Failure modes

• Common design-science failures include setting objectives after building, demonstrating without evaluating, underspecifying artifacts, relying on collaborator experts, and omitting practitioners’ existing practice as a baseline.Source: E. Failure modes · p. 8

F. Checklist

• The checklist calls for a stated problem, pre-set measurable objectives, a reproducible artifact, named evaluation type, validity threats, and artifact release.It specifies formative or summative evaluation and points to Section XVI-C for artifact release.Source: F. Checklist · p. 8

A. Purpose and when to use it

• Qualitative inquiry explains practitioners’ actions and work, and disagreements between technical measures and practitioner accounts can be findings.Use it when phenomena are organisational, variables are unknown, or quantitative results are inexplicable. Studies exposed gaps between academic and practitioner priorities, communication failures, and performance problems across organisational levels; sampling analysts and managers together located problems in their relationship, while another study documented what SOC metrics omit.Source: A. Purpose and when to use it · p. 8

B. Reflexive thematic analysis: the protocol

• Reflexive thematic analysis uses six phases, but its assumptions are often violated when themes are treated as emerging or inter-coder agreement is sought.The phases are familiarisation, systematic coding, generating, developing and reviewing, refining and naming themes, and writing up; Figure 5 adds a Step 0 choice before coding because it determines what counts as rigour.Source: B. Reflexive thematic analysis: the… · p. 8, Figure 5 · p. 9

C. Choosing among the qualitative designs •

• Interview studies require purposive sampling, open questions with planned probes, and an explicitly operationalised stopping rule rather than an unsupported claim of saturation.One study interviewed 54 penetration-testing stakeholders to argue for standardised scoping, delivery, and reporting; combining a programming task with interviews showed insecure password storage often reflected task interpretation, not lack of knowledge.Source: C. Choosing among the qualitative… · pp. 8–9
• Use cognitive task/work analysis when the decision itself is the study object; Delphi for structured expert consensus unavailable in datasets; and human-factors mixed methods when management and employee perspectives diverge.Source: C. Choosing among the qualitative… · p. 9

D. Rigour and reliability

• Researchers must distinguish reflexive analysis, where interpretation is the instrument, from codebook or content analysis, where inter-coder agreement is appropriate.Reflexive rigour rests on reflexivity, an audit trail, and analytic depth; agreement statistics should not be imposed on that design.Source: D. Rigour and reliability · p. 9
• Cohen’s κ suits two coders and nominal categories but can be depressed by category prevalence; Krippendorff’s α suits more coders, ordinal categories, or missing data.Landis and Koch’s interpretive bands remain common but are unreliable for skewed distributions. Report the statistic, coder count, double-coded proportion, and disagreement resolution.Source: D. Rigour and reliability · p. 9
• Use COREQ for interview and focus-group studies or SRQR for qualitative designs generally, and plan against the chosen standard from the outset.COREQ has 32 items covering the research team, design, and analysis; both standards are easier to satisfy prospectively because some items concern pre-collection decisions.Source: D. Rigour and reliability · p. 9

E. What the evidence supports

• Organisational coordination, cognitive workload, rule conflict and temporal behavioural variation can affect security outcomes but remain beyond technical measurement.Confidence is moderate; self-report bias, small purposive samples, sector-specific sites and limited transferability remain limitations.Source: E. What the evidence supports · p. 9

F. Checklist •; G. Checklist

• Checklist reporting should document design, sampling, recruitment, instruments, coding and any agreement statistic, while distinguishing participants’ contributions.For surveys, also report the full measurement model and the population the estimate generalises to; for Delphi, specify expertise criteria, panel size and attrition, the stopping rule and consensus statistic.Source: F. Checklist • · pp. 9–10, G. Checklist · p. 11

A. Purpose and when to use it; C. The generalisation problem, and what to do about it

• Surveys estimate population distributions and covariation in latent constructs, but convenience-sample findings may transfer differently by measure and demographics.State the population an estimate represents rather than generalising from a sample such as “350 crowdworkers” to “users”; recurring theoretical frames include protection-motivation and institutional-governance accounts.Source: A. Purpose and when to use it · p. 10, C. The generalisation problem, and… · p. 10

B. Protocol

• Survey protocols define population and frame, validate measures, power sample size, pilot items, track response, assess measurement before structural models, and report web-survey procedures.The figure orders measurement assessment before path coefficients and makes sample-size planning a gate; the pilot is the inexpensive point to correct misread items. Report invitation, response and completion rates, non-response checks, the full measurement model, and CHERRIES details including view, participation and completion rates.Source: B. Protocol · p. 10, Figure 6 · p. 10

D. Longitudinal and within-person designs

• Cross-sectional surveys capture between-person patterns, not within-person change; Cram et al.’s four-week experience-sampling study found cybersecurity behaviour varies within a person over time.For within-person theory, a one-time survey is the wrong instrument regardless of sample size; experience sampling uses frequent brief prompts and multilevel models separating within- from between-person variance.Source: D. Longitudinal and within-person… · p. 10

E. Structured expert elicitation: the Delphi design

• When questions require expert knowledge, Delphi uses successive anonymous rounds of aggregated responses until the panel converges or convergence stably fails.Campbell’s model used 20 information-security practitioners over three rounds to prioritise problem areas and countermeasures; report expertise criteria, panel size and attrition, stopping rule and consensus statistic, with anonymity limiting senior-participant influence.Source: E. Structured expert elicitation: the… · p. 11
• Delphi supports claims about expert consensus, not whether countermeasures work.Reporting consensus as an effectiveness result overstates what the design establishes.Source: E. Structured expert elicitation: the… · p. 11

F. What the evidence supports

• Survey designs have moderate support for latent organisational constructs when measurement models are reported, while within-person findings have limited support from one four-week study.The methodological case for within-person designs is independent of that study’s findings.Source: F. What the evidence supports · p. 11

A. Purpose and when to use it

• Threat modeling is useful for asking which attack paths exist, what they cost an attacker, and where controls would be most effective.It connects technical weaknesses to enterprise risk decisions.Source: A. Purpose and when to use it · p. 11

B. STRIDE: the protocol

• STRIDE decomposes a system into elements, maps applicable threat categories to each, assesses impact and likelihood, then plans countermeasures and records residual risk.The element-to-category mapping makes the method systematic; component-level analysis can miss interaction-only threats, and enumeration should be paired with mitigation analysis.Source: B. STRIDE: the protocol · p. 11, Figure 7 · p. 11

C. Risk-centric and graph-based extensions

• STRIDE enumerates but does not compose attacks; PASTA aligns countermeasures with business impact, while integrated models, attack-graph meta-languages, and CySeMoL represent systems, sequences, or attack probabilities.The integrated software- and attack-centric approach was worked through on a railway communication network; attack-graph meta-languages enable simulation of attack sequences and time-to-compromise, and CySeMoL was validated against professional judgement and case studies.Source: C. Risk-centric and graph-based… · p. 12

D. Reducing dependence on expert construction

• Expert-built models scale poorly and inherit expert assumptions; Malcoda automates risk estimation with security-product data, while dynamic multilevel modeling lacks reported datasets and validation conditions.In a virtual enterprise network, Malcoda completed assessments quickly, produced probabilities that tracked intrusion-detection alerts, and had complexity no greater than comparable models; security assurance metrics supply a needed measurement layer.Source: D. Reducing dependence on expert… · p. 12

E. Decision-theoretic and network-analytic extensions

• Expected-resilience modeling jointly allocates defensive and recovery investment, but researchers should state the sources of breach-probability, residual-loss, and risk-attitude inputs and test sensitivity to them.The optimum depends on assumptions rather than measurements.Source: E. Decision-theoretic and… · p. 12
• ATT&CK semantic-network analysis uses betweenness centrality to prioritize techniques recurring across pathways, but taxonomy structure and reporting bias can distort centrality; incident-frequency edge weights make the claim empirical.Credential access and valid-account abuse are usual central findings; nearly two years of operational privilege-graph monitoring remains an uncommon benchmark for what modern risk models approximate.Source: E. Decision-theoretic and… · p. 12

F. What the evidence supports

• Evidence moderately supports threat modeling and probabilistic or graph-based analysis for structuring enterprise risk, but generalization is limited by case-specific assumptions, expert input, scarce real-incident validation, and changing threats.Source: F. What the evidence supports · p. 12

G. Failure modes

• Failure modes include treating CVSS base scores as risk estimates, validating with the same experts who supplied model parameters, undocumented attack-graph pruning, and probability outputs without sensitivity analysis.Source: G. Failure modes · p. 12

H. Checklist

• The checklist requires showing system decomposition and trust boundaries, naming the method and its scope limit, separating parameter sources from validators, testing sensitivity, validating beyond author judgement, and recording residual risk.Source: H. Checklist · p. 12

IX. FAMILY F: SIMULATION AND MODEL-BASED ANALYSIS — A. Purpose and when to use it

• Simulation is suited to cascading-consequence analysis, sensitivity tests, and large-scale defensive comparisons when live experimentation would disrupt critical processes or require dangerous capabilities.It is especially useful for “what changes if” questions, not for establishing what is true.Source: A. Purpose and when to use it · p. 13

IX. FAMILY F: SIMULATION AND MODEL-BASED ANALYSIS — B. The main formalisms •

• Formalism examples include stochastic Petri nets optimizing grid detection intervals and redundancy, system dynamics tested on grid and chemical models, SURE cyber-physical games in transport, and dependency models checked through expert interviews.The system-dynamics tests used IEEE 14-bus and 300-bus grids and the Tennessee Eastman chemical process; dependency analysis examined power-grid models and a municipal grid.Source: B. The main formalisms • · p. 13

IX. FAMILY F: SIMULATION AND MODEL-BASED ANALYSIS — C. Protocol

• The seven-step protocol specifies the question and model, labels parameter sources, separates implementation verification from model validation, justifies run length and replications, tests assumed-parameter sensitivity, and seeks external validation.It calls for confidence intervals across replications; if a conclusion reverses within an assumed parameter’s plausible range, that reversal is itself a finding. Figure 8 highlights gates and warns that unlabeled parameter tables undermine sensitivity analysis.Source: C. Protocol · p. 13, Figure 8 · p. 13

IX. FAMILY F: SIMULATION AND MODEL-BASED ANALYSIS — D. What the evidence supports

• Confidence is moderate for simulation’s methodological usefulness but limited for predictions beyond modelled infrastructure and scenarios, because model fidelity and human responses under time pressure constrain generalisation.The studies cover grids, transport, chemical processes, and a hospital as proxies rather than instances of a general enterprise population.Source: D. What the evidence supports · p. 13

IX. FAMILY F: SIMULATION AND MODEL-BASED ANALYSIS — E. Checklist

• The simulation checklist calls for a reimplementable model specification, parameter-source classifications, distinct verification and validation, justified replications, confidence intervals, sensitivity analysis, and external validation.It also requires an explicit statement of the population the model represents.Source: E. Checklist · p. 13
• The checklist also requires an explicit statement of the population represented by the model.This reporting requirement addresses the limited basis for generalising beyond the modelled infrastructure and scenarios.Source: E. Checklist · p. 13, D. What the evidence supports · p. 13

X. FAMILY G: MACHINE-LEARNING EXPERIMENTATION — A. Purpose and when to use it

• Machine-learning detection experiments assess whether a model discriminates event classes under stated conditions at a usable operating point.Source: A. Purpose and when to use it · p. 13

B. The baseline protocol

• A detection protocol defines operational costs, justifies audited data, splits before preprocessing, resamples within training, matches splits to deployment, tunes on validation, reports metrics and costs, uses baselines, and repeats seeds.Audits of CICIDS2017 found that correcting defects in traffic generation, attack execution, and labelling changes measured performance and can affect detector comparisons.Source: B. The baseline protocol · p. 14

C. Metrics, and why accuracy is the wrong one

• Accuracy can reward an always-benign classifier under class imbalance; detection rate remains low unless false alarms are exceptionally rare, and precision–recall curves better show ranking under heavy imbalance.Operational reporting should include the false-alarm rate at the deployed base rate, not recall alone.Source: C. Metrics, and why accuracy is the… · p. 14

D. Temporal bias, spatial bias and drift

• Future-performance claims require temporally consistent splits; TESSERACT formalised temporal and spatial bias constraints and introduced time-aware comparison over a deployment period [85] [86].Random k-fold splitting answers whether classes can be separated, not whether a model will work next month.Source: D. Temporal bias, spatial bias and drift · pp. 14–15
• Drift studies need time-stamped longitudinal data: LAMDA enables multi-year Android-malware evaluation, while conformal methods can identify deployed samples to reject [89].A single-snapshot benchmark cannot exhibit drift, regardless of how it is split; the dataset must be chosen for time coverage first.Source: D. Temporal bias, spatial bias and drift · pp. 14–15

E. Pitfalls with names

• Arp et al. catalogue and quantify recurring machine-learning security pitfalls, while Rossow et al. give malware-specific practices for dataset construction, goodware handling, and execution transparency [90].The pitfalls include sampling and label problems, data snooping, poor baselines and measures, base-rate neglect, lab-only evaluation, and inappropriate threat models.Source: E. Pitfalls with names · p. 15
• Adaptive-adversary evaluations need established adversarial-ML methodology, and explainability claims require criteria-based assessment rather than judging an explanation by plausibility.Warnecke et al.’s criteria include descriptive accuracy, sparsity, stability, efficiency, and robustness.Source: E. Pitfalls with names · p. 15

F. Comparing randomised systems: the statistics step 9 requires

• Randomised-system comparisons require repeated trials, a non-parametric test such as Mann–Whitney U, a standardised effect size alongside the p-value, and multiple-comparison correction.With enough repetitions, any difference may become significant; effect size indicates whether it matters.Source: F. Comparing randomised systems: the… · p. 15
• Fuzzing results changed with seed corpus, timeout, and crash de-duplication, and many published comparisons used too few trials; stochastic security experiments need multiple trials, suitable tests, and effect sizes.The reported re-evaluation shows how protocol choices can change which tool wins and inflate bug counts.Source: F. Comparing randomised systems: the… · p. 15

G. What the evidence supports

• Comparative detection experiments with transparent preprocessing, splits, and metrics are appropriate, but algorithm dominance is unresolved and benchmark-to-enterprise transfer is limited –.The corpus evaluates almost entirely on benchmark, simulated, or laboratory data rather than continuously collected enterprise populations.Source: G. What the evidence supports · p. 15

H. Checklist

• Detection studies should state the operational task, data provenance, preprocessing and resampling boundaries, split rationale, and temporal consistency when claiming future performance.The checklist also calls for reporting hyperparameter search space and budget, full confusion matrices, realistic-base-rate false alarms, cost, incumbent baselines, and multi-seed tests with effect sizes.Source: H. Checklist · p. 16
• TRIPOD’s 22-item prediction-model checklist maps closely to detection reporting, including sample selection, predictors, missing data, model specification, performance, and validation.It is not a security-specific instrument, so its vocabulary must be translated.Source: H. Checklist · p. 16

A. Why this needs its own methodology

• Unknown training corpora, free-text outputs, and agentic behavior create distinct validity threats; benchmark contamination must be checked first and mitigated with held-out or perturbed tasks [104], [105].Report model version and date, and treat public-benchmark results as an upper bound; passive context contamination in log analysis can evade conventional accuracy measures.Source: A. Why this needs its own methodology · p. 16, B. Contamination and construct validity · p. 16

C. Language models as judges

• Judge reliability does not establish validity: calibrate against human labels with chance-corrected agreement, randomize presentation order, and test position and verbosity effects [106], [107].Agreement between judge runs alone is not an evaluation.Source: C. Language models as judges · p. 16

D. Agentic security systems

• Agent benchmarks must specify environments and path-dependent success; examples include capability ladders, comparisons with cybersecurity professionals, and evaluating generated detection rules [108]–[111], [27].Confidence is moderate that contamination control, judge calibration, and environment specification are necessary, but limited for current operational-capability claims; Figure 10 identifies contamination as disqualifying and highlights judge calibration and cost reporting.Source: D. Agentic security systems · p. 16, F. What the evidence supports · p. 16, Figure 10 · p. 17

E. Protocol additions

• Extend the protocol with model version and access date, sampling temperature, attempts per task, pass-at-k, contamination checks, and judge calibration; report human-label agreement where a judge is used.The protocol also requires precise environment descriptions and cost per task, since success at arbitrary expense may not solve an operational problem.Source: E. Protocol additions · p. 16, G. Checklist · p. 17

G. Checklist

• Randomize judge presentation order, test position and verbosity effects, specify environments to testbed standards, and report cost per task.These checklist additions complement the broader language-model protocol.Source: G. Checklist · p. 17

A. Purpose and when to use it

• The family asks what is observable, because detectors cannot find unrecorded behaviour and telemetry helps explain apparent laboratory–production performance gaps.Source: A. Purpose and when to use it · p. 17

B. Instruments and findings

• Logging standards, vendor exercises, and detector benchmarks are measurement instruments whose construct validity and protocol choices shape what their results establish [112], [113], [114], [26].Source: B. Instruments and findings · p. 17
• Honeypot findings depend on bait configuration, while intelligence formats vary in concepts and can be combined by use case.The honeynet study analysed interaction level, origin, and attack pattern; comparison of six intelligence formats identified 18 core concepts.Source: B. Instruments and findings · p. 17

C. Protocol

• Define behaviour, establish independent ground truth, measure coverage before efficacy, report exact configuration, separate detection from alerting and response, and use STROBE.Coverage and efficacy have different denominators; deriving ground truth from the measured system makes subsequent numbers circular.Source: C. Protocol · pp. 17–18, Figure 11 · p. 18

D. Measuring consequences: the event-study design

• Event studies estimate firms’ normal returns before breach announcements and test short-window market reactions, measuring expected market cost rather than actual cost.The design requires publicly traded firms, dated announcements, and a narrow event window; its sample excludes unannounced breaches, and disclosure regimes alter comparability across periods.Source: D. Measuring consequences: the… · p. 18

E. What the evidence supports

• Moderate confidence supports telemetry coverage as a first-order determinant of detection outcomes that remains under-measured relative to model performance.Source: E. What the evidence supports · p. 18

F. Checklist

• Define observables as behaviours, establish independent ground truth, and measure coverage before efficacy, each with its denominator.Source: F. Checklist · p. 18
• Report agent, rule, log-source, and retention settings; distinguish logged, detected, alerted, and actioned; use STROBE; and state honeypot device set, exposure period, and network placement.Source: F. Checklist · p. 18

E. Protocol

• Use testbeds when executing attacks, production access is impossible, or results must be reproducible, and specify the environment as code with explicit fidelity and containment claims.The six-step protocol also calls for a prewritten taxonomy-mapped emulation plan, independent instrumentation, and joint publication of configuration, plan, and data; Figure 12 highlights fidelity and containment as gates.Source: A. Purpose and when to use it · p. 18, E. Protocol · p. 19, Figure 12 · p. 19

B. The design space

• Testbed research advances reproducibility [122], scale through federation [123], and scenario generation and cyber-range lifecycle design [124].The reference map organizes cyber ranges and testbeds by scenario, function, tool, and architecture.Source: B. The design space · p. 18

C. Adversary emulation as a measurement instrument

• A prewritten, taxonomy-mapped emulation plan makes red-team behavior repeatable and comparable [113], while penetration-test reporting and scoping practices affect interpretability.A firewall audit illustrates actionable reporting through scanning, testing, severity classification, and a risk report; ambiguity in engagement scoping and delivery undermines comparisons.Source: C. Adversary emulation as a… · pp. 18–19

D. Studying real adversaries in controlled infrastructure

• Controlled infrastructure can expose real offenders’ target selection and actions, but raises ethical constraints around facilitation, consent, victim data, containment, and approvals.Ricaldi et al. propose an Access-as-a-Service platform with manipulable market signals and a honeypot back end, presenting the experimental design for community critique before execution.Source: D. Studying real adversaries in… · p. 19

F. What the evidence supports

• Confidence is moderate that ranges and testbeds support controlled security experimentation and training, but limited for production transfer when fidelity claims are not falsifiable.The review emphasizes that researchers should state which properties of real environments are and are not reproduced.Source: F. What the evidence supports · p. 19

G. Checklist

• The checklist requires falsifiable fidelity claims, documented containment with egress prevented, independent instrumentation, and publication of configuration, emulation plan, and data together.It also requires environment-as-code and a taxonomy-mapped plan written before execution.Source: G. Checklist · p. 19

B. Field experiments

• Field experiments can support causal claims about behavioral interventions only when randomized, with enterprise phishing studies requiring designs that prevent contamination between conditions.A field study compared embedded with non-embedded post-simulation feedback and reported effects on subsequent behavior; recommended practices include a pre-registered primary outcome, intention-to-treat analysis, and ethics approval covering employee deception.Source: A. Purpose and when to use it · p. 19, B. Field experiments · p. 19
• Report enterprise trials against CONSORT, including allocation, blinding, participant flow, assigned and analysed units, and attrition.The required flow diagram makes post-randomization exclusions and departures visible; software-engineering experimentation supplies a complementary validity-threat vocabulary.Source: B. Field experiments · pp. 19–20

C. Training and range-based education

• A randomized exercise redesign improved learning experience and shortened training time with equivalent knowledge outcomes; workplace phishing training improved detection and avoidance and tested a mindfulness mechanism.Adaptive range training was evaluated on performance, knowledge, retention and incident-response capability, while CyberCIEGE reported improved engagement and concept application in simulation.Source: C. Training and range-based education · p. 20

D. Awareness programme evaluation

• Training completion measures policy compliance, not awareness; a 32-paper review adapts impact, sustainability, accessibility and monitoring indicators for systematic evaluation.Small and medium-sized enterprises are an evidence-poor context for awareness needs, resources, expertise and incentives.Source: D. Awareness programme evaluation · p. 20

E. Human-subjects methodology in security

• Usable-security reviews examine how risk is represented to participants because that design choice affects whether behavior reflects real stakes; generalisation cautions also apply.Source: E. Human-subjects methodology in… · p. 20

F. What the evidence supports

• Evidence gives moderate confidence that ranges, adaptive progression and interactive exercises improve engagement and short-term performance, but limited confidence in long-term effectiveness and scalability.Sustained retention and organisational behavior change are not consistently evaluated in the corpus.Source: F. What the evidence supports · p. 20

G. Checklist

• A behavioral study checklist calls for stating randomisation and addressing contamination, describing allocation concealment, prespecifying outcomes, using intention-to-treat analysis and reconciling participant flow.It also requires ethics approval covering deception.Source: G. Checklist · p. 20
• The checklist favors behavioral outcomes, incumbent-practice comparators, stated follow-up intervals and effect sizes with intervals rather than p-values.Source: G. Checklist · p. 20

XV. OPERATIONS, INCIDENT RESPONSE AND MATURITY

• Incident-response maturity should be assessed as a capability system, pairing a maturity instrument with an independent outcome measure; confidence is moderate for structured process but limited for cross-organisational comparison.The literature lacks validated measures of incident-response effectiveness, while organisational resources and commitment vary widely.Source: §XV · pp. 20–21

A. Process models as research objects: the forensic case

• Forensic process models should be consolidated and judged for completeness and correspondence between steps and evidential requirements; explicit tool models enable systematic error analysis.Source: A. Process models as research objects:… · p. 21

B. Prioritising organisational change under disagreement

• Prioritizing zero-trust change, a 29-professional fuzzy-Delphi panel and CRITIC weighting identified four dimensions and 33 criteria, with rankings that can reverse as conditions change.Because multi-criteria weights capture a contingent preference structure, studies should test ranking sensitivity; combining expert elicitation with correlation-based weighting avoids asking the panel to determine both criteria and importance.Source: B. Prioritising organisational change… · p. 21

A. Validity, stated as four questions

• Validity requires defining the measured construct, addressing alternative causes, stating the target population, and ensuring the analysis supports its inference.Threats include benchmark F1 not representing enterprise intrusion detection, leakage or confounds, unspecified generalisation, and uncorrected multiple comparisons, single-run differences, or missing effect-size intervals.Source: A. Validity, stated as four questions · pp. 21–22

B. Reporting standards, by family

• Five families lacked a located reporting standard, while usable standards often transfer across fields with vocabulary changes rather than changes of substance.The five were threat modeling, simulation, language-model evaluation, testbeds, and design science; empty rows reflect retrieval, not a systematic search demonstrating absence, so family checklists are a weaker substitute.Source: Table III · p. 22, B. Reporting standards, by family · p. 22

C. Reproducibility and artifact sharing

• Reproducible studies should release code, pinned dependencies, data splits or regeneration scripts, environment details, and seeds, while distinguishing open-science commitments that may not all be feasible.Artifact evaluation checks whether an artifact works, not whether it is safe to run: analysis of 509 artifacts found 41.60% of prevalent findings posed potential security concerns under practical usage [147]. Authors should pin dependencies, document network behaviour, and remove credentials, keys, and permissive defaults.Source: C. Reproducibility and artifact sharing · pp. 22–23

D. Disclosure as part of the method

• Disclosure should be designed around the vulnerability-notification pipeline, with an identifiable issuer, a timeline beginning at report, and disclosure timing reported in the paper.Analysis of 3,798 reviewed GitHub security advisories and 4,033 disclosed open-source bug-bounty reports found that missing or late CVE assignment can delay notification of dependent projects [148].Source: D. Disclosure as part of the method · p. 23

E. Ethics and legal constraints

• Enterprise research should address phishing deception, production-telemetry handling, and live-weakness disclosure, then state the ethical principle and trade-off behind consequential decisions, [150].The Menlo framework supplies relevant principles, while approval alone does not explain ethical reasoning; researchers should name imposed costs, benefits, and the principle used to weigh them [150].Source: E. Ethics and legal constraints · p. 23

F. From result to claim

• The review recommends matching each claim’s strength to traceable evidence, using a claim-to-evidence table and weakening claims when their supporting artifacts are missing or ambiguous.Each claim should identify its supporting artifact and that artifact’s verification status; checked evidence is necessary for strong comparative language.Source: F. From result to claim · p. 23

A. The contradiction

• Four careful intrusion-detection comparisons disagree: ensembles, feed-forward networks, recurrent systems, and random-forest results rank differently,,,.One study found ensembles generally strong, while naive Bayes trained faster and had advantages for new attack types; another favored deep feed-forward networks across four datasets. A recurrent system was reported superior to four alternatives, while a later comparison reported 98% accuracy for convolutional and recurrent models versus 99.9% for random forest.Source: A. The contradiction · p. 23

B. The resolution; C. What a researcher should do about it

• Joint variation in dataset, features, preprocessing, network context, class balance and tuning can explain ranking differences, making benchmark accuracy an unstable estimate of enterprise performance.A hybrid ensemble reported accuracy, precision, recall and F1 approaching 100%, yet still had computational overhead, latency, overfitting risk and no explicit unseen-attack evaluation; publication bias is also plausible but unquantified. The review recommends multiple datasets, temporal splits, realistic false-alarm rates, simple baselines, cost reporting and unseen-attack tests.Source: B. The resolution · p. 24, C. What a researcher should do about it · p. 24

XVIII. THREE WORKED PROTOCOLS

• Three worked protocols compose existing methodology families rather than adding a fourth; Figure 16 makes their steps, decision gates, outputs and characteristic early failures visible.The recipes fix what can be observed or claimed, force a decision that determines what results license, and specify required outputs.Source: §XVIII · p. 24, Figure 16 · p. 25

A. Recipe 1: Evaluating a detection approach for a real deployment

• For deployment detection, establish telemetry coverage, reproduce the environment in a coded testbed, create labelled data, use temporal testing, and report performance, cost and unseen-attack degradation.Families G, I and J are combined; the held-out period should contain behaviours absent from training.Source: A. Recipe 1: Evaluating a detection… · p. 24

B. Recipe 2: Studying a security-operations practice

• To study security-operations practice, combine a rapid review, varied practitioner interviews across at least three organisations, reflexive analysis and—if indicated—design science evaluated by non-interviewees.Report the formative-summative distinction honestly.Source: B. Recipe 2: Studying a… · p. 24

C. Recipe 3: Quantifying risk in an enterprise architecture

• To quantify enterprise risk, combine STRIDE threat enumeration, attack graphs and parameter provenance with sensitivity analysis and validation from independent evidence or experts.Report any conclusion that inverts within plausible parameter ranges; prefer deployed-product inputs to elicitation where available.Source: C. Recipe 3: Quantifying risk in an… · p. 24

XIX. GAPS AND A RESEARCH AGENDA; A. Gaps in the field

• Priority research gaps include longitudinal enterprise telemetry across organisations, harmonised comparison protocols, validated operational outcomes, interaction-level threats, uncertainty reporting, under-represented contexts and agentic-system evaluation.The review notes limited direct work with micro and small enterprises, including a ransomware-mitigation model developed and validated for them.Source: A. Gaps in the field · pp. 24–25

B. Gaps in this review

• The review under-covers operational security metrics, insider-threat study design, adversary recruitment and observation at scale, deception measurement, and forensic process validation.These facets are named as absences from the review, not evidence that the topics lack importance.Source: B. Gaps in this review · p. 25

XX. CONCLUSION

• Reported detection performance is high, but rankings vary across studies; the review attributes this inconsistency to joint evaluation-design differences, not an isolated algorithmic factor.Risk models produce usable probabilities in virtual enterprise networks but await broader replication, while organisational research shows that communication, decision load, human factors, intelligence integration and temporal variation shape security outcomes,,.Source: §XX · p. 26

• Table IV gathers the checklist items most often missing from published work in the corpus and is intended for writers and reviewers.Source: Appendix A · p. 26