Source-linked AI summary
Poster: Towards Selecting Threat Appropriate Industrial Intrusion Detection Systems
Stefan Lenz, Johannes Weidmann, Martin Henze
TL;DR
Dataset-wide IDS metrics can obscure scenario-specific performance, even though detectors may suit different attacks. The paper proposes matching shared CTI about current threats to an attack-specific performance database to select detectors. Its initial evaluation finds that the best-performing detector varies by attack scenario, while deployment must address when to switch detectors without losing coverage of other threats.
Problem
Dataset-level metrics average performance across diverse behaviors and attacks, obscuring how IDSs perform in specific threat situations.
Source: §1 · p. 1Method
The paper proposes selecting IDSs by cross-referencing shared CTI about prevalent TTPs with a database of detector performance for those techniques.
Source: §2 · pp. 1–2Results
Detection performance varies by attack scenario; on the complete dataset GeCo scores 0.62, while DoS eTaR is 0.87 for InterArrivalTime and 0.55 for GeCo.
Source: §3 · p. 3Takeaways & Limitations
TTP-based evaluation can decouple performance from datasets to specific scenarios, highlighting the potential of threat-specific detector selection.
Source: §3 · p. 3, §5 · p. 3Switching to a detector specialized for a single TTP may hinder detection of other TTPs and potentially reduce overall detection rates.
Source: §4 · p. 3
Abstract
from arXiv · showhide
As the threat landscape against industrial control systems is dynamic, effective security requires detection strategies capable of timely reactions to these evolving threat situations. To address this problem, we propose the idea of a counter-threat intelligence sharing based mechanism to select appropriate detectors for current circumstances. To highlight the potential of this mechanism, we conduct attack-level performance evaluations of various intrusion detection systems. Results show the variance of intrusion detection performance depending on the attack scenario, emphasizing the benefits of such a mechanism for industrial control system security
1 Motivation
ICS intrusion detection is often evaluated with dataset-wide metrics, but detector performance can vary by threat scenario. The paper proposes using shared threat intelligence to select detectors suited to the current situation.
1 Motivation
- Dataset-level recall and F1 averages can obscure scenario-specific performance in datasets containing diverse behaviors and attacks.Detectors using different features may suit different attacks; timing-based detection may identify DoS flooding better than process-state detection.Source: §1 · p. 1
- Ensembles combine detectors with different strengths, but still optimize dataset-level performance rather than the currently observed threat landscape.Shared CTI could help operators select a detector suited to a threat reported as prevalent by other operators.Source: §1 · p. 1
- The paper presents a CTI-sharing-based mechanism for dynamic IDS selection and reports initial results on its potential.The paper outlines the selection mechanism, presents initial results, and discusses challenges to realizing it.Source: §1 · p. 1
2 Threat Appropriate IDS Selection
The proposed mechanism uses shared CTI about prevalent attacker techniques, a performance database indexed by those techniques, and administrator selection to match IDSs to current threats. It may include feedback on whether detection of reported techniques increased.
2 Threat Appropriate IDS Selection
- The mechanism has three parts: CTI analysis and sharing, a performance database for attacker techniques, and IDS selection by matching CTI reports to that database.Organizations can share prevalent tactics, techniques, and procedures (TTPs) in standardized formats such as MITRE ATT&CK.Source: §2 · p. 1
- Operators share detected TTPs and can use reports from others to select a detection strategy suited to threats that may also target their systems.The first step is intended to increase situational awareness and inform adaptation of defenses.Source: §2 · pp. 1–2
- The performance database is built by mapping benchmark-dataset attack descriptions to TTPs and cross-referencing evaluations of multiple detectors on those TTPs.Maintaining and growing the database is described as supporting increasingly informed choices.Source: §2 · p. 2
- After CTI reports are matched to the database, an administrator chooses and deploys an appropriate detector; the mechanism may add feedback on detection of reported TTPs.The paper notes that an end-to-end version does not yet exist and evaluates TTP-level performance to show potential.Source: §2 · p. 2
3 Proof of Concept
Detector performance varied across attack scenarios and TTPs, unlike the best-performing detector on the complete dataset. The evaluation maps WDT attacks to MITRE ATT&CK TTPs and compares four IDS approaches; the authors conclude that TTP-based evaluation can separate scenario performance from dataset-level performance.
3 Proof of Concept
- Mapping WDT attack descriptions to TTPs is not one-to-one: most descriptions fit multiple TTPs, and the evaluation assigns six TTPs to three scenarios.The selected WDT attacks cover both process-oriented and network-component TTPs.Source: §3 · p. 2
- The evaluation compares four IDS approaches spanning communication monitoring and process-state monitoring, using IPAL and WDT attacks mapped to MITRE ATT&CK ICS TTPs.The approaches are InterArrivalTime, Kitsune, OneClassSVM, and GeCo.Source: §3 · p. 2
- Communication-based detectors alert persistently in DoS scenarios, while process-aware detectors alert consistently for process-state attacks such as Adversary in the Middle and Manipulation of View.The figure labels attacks with TTPs and distinguishes true alerts (black) from false alerts (red).Source: Figure 2 · p. 2, §3 · p. 3
- GeCo scores 0.62 on the complete dataset, while DoS eTaR is 0.87 for InterArrivalTime versus 0.55 for GeCo; Kitsune scores 0.09 for Loss of View and 0.64 for DoS.These results show that the detector with the strongest dataset-level score need not lead on an individual TTP.Source: §3 · p. 3, Table 1 · p. 3
- The results show performance varies by attack scenario, and TTP-based evaluation decouples detector performance from datasets to specific scenarios.The authors present this variance as evidence of the potential for threat-specific detector selection.Source: §3 · p. 3
4 Future Challenges
The proposed selection mechanism faces participation and data-quality challenges in CTI sharing, plus unresolved decisions about when to switch detectors. Suggested mitigations include trusted third parties, automated TTP mapping, and weighted ensembles.
CTI Sharing Incentives and Privacy Concerns
- CTI sharing may be deterred by privacy concerns or insufficient incentives, requiring participant trust and assurances of no negative impact.For the small, sometimes government-controlled ICS sector, government agencies could serve as trusted third parties.Source: §4 · p. 3
Consistent TTP Labels
- Inconsistent TTP labels could distort the IDS performance database and lead to incorrect detector choices.Automated mapping tools, including approaches based on large-language models, may minimize these inconsistencies.Source: §4 · p. 3
Timely IDS Selection and Switching
- Switching too soon may overlook still-relevant threats, while parallel or single-TTP detectors can create alert uncertainty or reduce overall detection rates.A weighted ensemble could prioritize detectors for current TTPs while retaining others to maintain TTP coverage.Source: §4 · p. 3
5 Conclusion
The paper proposes selecting IDSs by matching the current threat landscape to attack-scenario performance data. Initial experiments show that detector performance varies across attacks, highlighting the potential of situation-specific selection.
5 Conclusion
- The mechanism cross-references the current threat landscape with an attack-scenario-specific performance database to select appropriate detectors.Source: §5 · p. 3
- Initial experiments show that intrusion detectors perform differently across attack scenarios, indicating the potential of selecting a well-suited detector for the current threat landscape.Source: §5 · p. 3
- The mechanism aims to improve ICS attack detection and simplify and demystify IDS capabilities for practical application.Source: §5 · p. 3