Source-linked AI summary
Decision-Making and Biases in Cybersecurity Capability Development: Evidence from a Simulation Game Experiment
M. S. Jalali
TL;DR
Cybersecurity investment decisions are difficult because incidents are uncertain and capability development takes time. The paper uses a simulation game to compare experienced and inexperienced decision-makers and examine learning through repeated decisions. The groups showed no performance differences, while experienced players showed positive learning effects for proactive decision-making.
Problem
Cybersecurity decisions involve uncertain cyber incidents and delays in developing capabilities, complicating managers’ resource allocation.
Source: §2 · p. 5, §2.2 · p. 8Method
The authors use a simulation-game experiment to examine decision-making by experienced and inexperienced players.
Source: §6 · p. 28Results
Experienced and inexperienced players showed no performance differences, while experienced players showed positive learning effects on proactive decision-making.
Source: §6 · p. 28Takeaways & Limitations
The findings support training for experienced managers to better understand cybersecurity complexities and develop proactive decision-making.
Source: §6 · p. 28The study did not examine how or in what priority prevention, detection, and response capabilities should be developed.
Source: §5.3 · p. 25
Abstract
from arXiv · showhide
We developed a simulation game to study the effectiveness of decision-makers in overcoming two complexities in building cybersecurity capabilities: potential delays in capability development; and uncertainties in predicting cyber incidents. Analyzing 1,479 simulation runs, we compared the performances of a group of experienced professionals with those of an inexperienced control group. Experienced subjects did not understand the mechanisms of delays any better than inexperienced subjects; however, experienced subjects were better able to learn the need for proactive decision-making through an iterative process. Both groups exhibited similar errors when dealing with the uncertainty of cyber incidents. Our findings highlight the importance of training for decision-makers with a focus on systems thinking skills, and lay the groundwork for future research on uncovering mental biases about the complexities of cybersecurity.
1. Introduction
The introduction frames cybersecurity capability development as a managerial challenge amid growing cyber risks, then asks whether experience and iterative simulation learning support proactive decisions. It presents a simulation game and systems approach to examine capability-development delays and cyber-risk uncertainty.
1.1. Research motivation
- Cyberattacks can bring substantial financial and reputational harm, while organizations may underestimate risks or rely on generic solutions.The introduction cites a 19% confidence rate among CISOs that their companies can effectively address an incident.Source: §1.1 · pp. 1–2
1.2. Research objective and approach
- The study asks whether experienced managers are more proactive and whether iterative learning in a simulation can teach proactive decision-making.The game has participants allocate investment to cybersecurity capabilities and monitor effects over five simulated years.Source: §1.2 · p. 2
1.3. Research contributions
- The study examines experience and iterative learning under cyber-risk uncertainty and capability-development delays, using a simulation game and system dynamics modeling.Its stated contributions also include applying a systems approach to cybersecurity.Source: §1.3 · p. 3
- The paper targets limited attention to individual proactive and reactive investment decisions under cyber-risk uncertainty and delayed visibility of capability benefits.It focuses on biases about delays in capability development and applies systems thinking to business and management aspects of cybersecurity.Source: §1.3 · p. 4
2. Theoretical background
The theoretical background treats cybersecurity investment as a resource-allocation problem shaped by uncertain incidents and delayed capability benefits. It argues that event-oriented decisions can obscure feedback and delays, complicating effective investment.
2. Theoretical background
- Cybersecurity capability configurations are linked to organizational performance, but achieving an effective allocation is difficult and resources diverted to security reduce cash flow.The trade-off is particularly constrained for small and medium-sized enterprises with limited additional funding.Source: §2 · p. 5
2.1. Uncertainty of cyber incidents
- Cyber incidents are difficult to assess because hypothetical losses, incident likelihood, and investment costs and benefits are hard to measure.Limited historical data, poor predictive ability, and uncertainty about the types and range of cyber risks compound the challenge.Source: §2.1 · p. 6
- Managers may underestimate incident frequency and the time capabilities need to become active when cyberattacks go undetected.The absence of detected attacks can provide little motivation to invest in cybersecurity.Source: §2.1 · p. 7
2.2. Complex systems and delays in building cybersecurity capabilities
- Event-oriented decision frameworks can obscure connections among system components and delays between decisions and their effects.Such delays can create instability and oscillation, while efforts to close a perceived time gap may come after corrective action has restored equilibrium.Source: §2.2 · p. 7
- Cybersecurity capability development can take years, and reactive investment after an attack may leave systems vulnerable before they recover.The background notes that managers’ intuition about these delays is inadequate despite their apparent familiarity with them.Source: §2.2 · p. 8
3. Methods
The methods section introduces a system-dynamics simulation game for studying cybersecurity capability development. For simplicity, it groups capabilities into three categories and represents information systems in four states.
3.1.1. Cybersecurity capabilities at the core
- The game condenses the five NIST cybersecurity framework categories into prevention, detection, and response/recovery capabilities.Identify and protect are grouped as prevention; respond and recover are combined as response/recovery.Source: §3.1.1 · pp. 8–9
3.1.2. Simulation model in the game
- The game uses a system-dynamics model with computer-based information systems, cybersecurity capabilities, and cyber incidents as its main entities.The model is presented as a simplified structure for the simulation game.Source: §3.1.2 · p. 9, Figure 1 · p. 9
- The model classifies information systems as not at risk, at risk, affected, or affected and detected.The authors use these four general categories to keep the game simple.Source: §3.1.2 · p. 9
Model structure and formulation
The model links resource-allocation decisions to capability growth and transitions among system-risk states. Players seek accumulated profit over five years while balancing investment costs, incident protection, and delayed capability effects.
Model structure and formulation
- Capability-development speed depends on how players allocate available resources across capability categories.The game uses three categories: prevention, detection, and response capabilities.Source: Model structure and formulation · p. 10
- Prevention reduces movement from not-at-risk to at-risk systems, while detection and response/recovery support detection and recovery of affected systems.The model assigns six months to risk promotion and detection of at-risk systems, and two months to detection of affected systems and patching.Source: Model structure and formulation · p. 10
- Over 60-month runs, players aim to maximize accumulated profits by allocating resources between profit-making and cybersecurity capabilities.The model feeds observed profits back into players’ decision parameters, allowing decisions to change over time.Source: Model structure and formulation · pp. 11–12
- Prevention alone cannot fully eliminate incident risk or enable detection and recovery; early investment across all three capability categories is associated with early profit reductions but high long-term profits.The model treats cybersecurity investment as an expense that may protect against costly attacks, and uses profit as its sole performance measure.Source: Model structure and formulation · pp. 11–12
- The authors use a simplified model, arguing that detail does not necessarily make a complex-system simulation more realistic and that simplicity helps explain player behavior.They also state that the simpler model helps managers from different industries relate it to their organizations.Source: Model structure and formulation · p. 12
Cyber incident patterns
The simulation models five external cyber incidents across 60 months and lets players adjust capability investments while monitoring profits. Its interface supports iterative decisions, and the incident-pattern section links attacks to profit changes.
Cyber incident patterns
- Each run includes five incidents with varying impacts, modeled as external and not categorized by attack type; the authors acknowledge insider-attack risk.They argue the simplifications remain relevant because successful attacks can cause data loss or operational disruptions regardless of their nature.Source: Cyber incident patterns · pp. 12–13
- With no cybersecurity investment, the blue-versus-red profit gap in Figure 2(b) is directly attributed to the attacks shown in panel (a).Source: Cyber incident patterns · p. 13, Figure 2 · p. 13
3.1.3. How the game works
- Players allocate resources across prevention, detection, and response, advance the simulation in 12-month intervals, and can revise their strategy over five trials.Each capability allocation can range from 0% to 5% of the IT budget; profit graphs and accumulated profits update as the simulation advances.Source: §3.1.3 · pp. 13–14
Two levels of the game
The game contrasts fixed, predictable attacks with random attacks whose total impact is controlled across players. Experienced professionals and inexperienced students played both levels under a common, repeatable procedure.
Two levels of the game
- Level one uses five attacks at fixed times with fixed impacts; level two uses five attacks at random times and impacts drawn from a uniform distribution.The total impact of the five random attacks is matched across players and to the total fixed-attack impact in level one.Source: Two levels of the game · pp. 14–15
3.2. Experimental setup
- The experiment compared experienced and inexperienced players in both deterministic and non-deterministic settings.Source: §3.2 · p. 15
3.3.1 Subjects
- The experienced group comprised 38 cybersecurity professionals averaging 15 years of IT and cybersecurity experience; the control group comprised 29 students with no prior experience.The students were master's candidates in a general IT course, and the experiment took place at the semester's start.Source: §3.3.1 · p. 15
3.3.2 Experiment methods
- Participants played separately without speaking, received background information and a demonstration, and could practice before the game.Source: §3.3.2 · pp. 15–16
- Players spent ten minutes on each level and could repeat runs freely, but were not told the levels differed until results were presented.Source: §3.3.2 · p. 16
4. Results
Across both game levels, experienced players did not outperform inexperienced players overall, but experienced players improved with repeated runs. Proactive investment cost more early and paid off after attacks began.
4. Results
- The analysis measured investment decisions and profits across 60-month runs, excluding 14% of experiment-group runs and 10% of control-group runs that ended early.Accumulated profit at the end of each full run measured individual performance.Source: §4 · p. 16, Table 1 · p. 16
4.1. Proactive vs reactive
- Proactive players earned less early but performed much better than reactive players once cyberattacks began.The early profit difference reflects the trade-off between investing in capability development and short-term returns.Source: §4.1 · p. 16, Figure 4 · p. 17, Figure 5 · p. 17
4.2. Comparison of the group results
- Performance varied more in random-attack level two—almost twice as much as in level one—while level-one results clustered near the M$2,500 maximum.Because attacks are random in level two, its maximum accumulated profit can be higher or lower than level one's.Source: §4.2 · pp. 17–18, Figure 6 · p. 18
- Experienced players showed no significant performance advantage over inexperienced players in either level, including the random-attack level.The comparisons found no difference at the 5% significance level.Source: §4.2 · p. 18
4.3. Effect of players’ iterative learning on their performance
- Only experienced players showed significantly positive correlations between run count and best performance in both levels.The analysis used players who completed at least five rounds in a level.Source: §4.3 · pp. 19, 21, Table 2 · p. 19
- Among experienced players, those with the highest performance slope in level one performed among the worst in level two, while inexperienced players’ rankings changed more arbitrarily.The experienced group’s top six in level one had an average 56% decrease in ranking in level two, compared with 22% for the inexperienced group’s top six.Source: §4.3 · pp. 21–22
5. Discussion
The discussion finds that experience alone did not improve proactive cybersecurity decisions, while iterative learning helped decision-makers learn to act proactively. It highlights feedback delays and uncertain incidents as challenges, and recommends training that addresses cognitive search and systems thinking.
5.1. Contribution of research
- In dynamic settings, decision-makers respond poorly to feedback delays, often using feedback rather than feedforward strategies, which reduces decision effectiveness.The discussion suggests these problems may be more pronounced in real-world cybersecurity environments, with more delays and uncertainties than the experiment.Source: §5.1 · p. 23
- Experienced managers did not perform better than inexperienced individuals in making proactive decisions about building cybersecurity capabilities.The discussion also notes that experienced players who reached a possible optimum in the first level did not achieve near optimality in the second.Source: §5.1 · p. 23
- Unpredictable incident timing and impacts make optimal decisions hardly feasible; experienced players’ search for alternative optima contrasted with less experienced players’ greater flexibility in exploring alternatives.The discussion describes iterative learning through simulation as a way to address these decision-making shortcomings.Source: §5.1 · pp. 23–24
5.2. Implications for practice
- The discussion calls for training that addresses cognition-driven optimization and feedback delays, which it says are often overlooked in cybersecurity decision-making.It notes that current training does not address these needs.Source: §5.2 · p. 25
- Cybersecurity training for executives can use simulation games to challenge cognitive searches for optimal decisions and show the effects of feedback delays.Players can try strategies repeatedly without real-world consequences, and the game approach was reported to improve proactive decision-making through iterative learning.Source: §5.2 · p. 25
5.3. Limitations and future research directions
- The study aimed to draw attention to managerial understanding of cybersecurity capability-development complexities, not to improve that understanding, and it did not measure individuals’ risk tolerance.The authors identify risk tolerance as a possible focus for future research on managerial decision-making biases.Source: §5.3 · pp. 25–26
- The authors report that organizations may consider prevention and detection while ignoring response capabilities, and suggest studying the long-run consequences of that approach.They also include a quoted recommendation for companies to prepare, plan, and test for cyber incidents.Source: §5.3 · pp. 25–26
- Future studies could examine decision heuristics and test different game settings, including attack vectors, time horizons, player roles, initial conditions, and cybersecurity training.The authors also propose fitting the model to historical cyber-event data and using larger samples or non-security business experts.Source: §5.3 · pp. 26–27
6. Conclusions
The experiment found no overall performance difference by experience, while experienced players showed positive learning effects on proactive decision-making. The authors call for training that addresses delays in capability development and uncertainty about cyber events.
6. Conclusions
- Proactive capability decisions are difficult because investment competes with profitable activities, capability benefits are delayed, and cyber events are uncertain.The perceived payoff is affected by the delay before capabilities reach full functionality.Source: §6 · p. 28
- Experienced and inexperienced players showed no overall performance difference, and experienced players’ level-one success did not predict their level-two success.Inexperienced players adapted more dynamically across repeated runs and changing environments.Source: §6 · p. 28
- Experienced players showed significantly positive learning effects on proactive decision-making, but experience alone did not help with cybersecurity decisions.The authors conclude that experienced managers need training to understand cybersecurity’s complexities.Source: §6 · p. 28
- The authors emphasize training decision-makers to challenge entrenched mindsets and encourage proactive cybersecurity capability development.They describe management flight simulators as effective training tools for improving decision-making.Source: §6 · p. 28
- The game’s general capability-development mechanisms may apply beyond cybersecurity, while organizational decision-making about cybersecurity capabilities remains underexamined.The study focuses on how managers make proactive cybersecurity capability investment decisions.Source: §6 · pp. 28–29