Source-linked AI summary
MITRE-SAGE: A Multi-Agent Cybersecurity Question-Answering Model
Ali Habibzadeh, Farid Feyzi, Reza Ebrahimi Atani
TL;DR
Cybersecurity QA systems must handle noisy, heterogeneous information while preserving semantic and structural relationships. MITRE-SAGE addresses this with a hierarchical multi-agent RAG framework and MITRE-QA benchmark, achieving the best overall performance on five of eight tasks and up to 58% higher multiple-choice accuracy than the strongest baseline.
Problem
Traditional RAG is sensitive to retrieval noise and flat representations that lose relational structure, limiting reliable multi-hop cybersecurity question answering.
Method
MITRE-SAGE uses an orchestrator, specialized retrieval agents, and summarization agents to integrate knowledge-graph, text, and web-based evidence for cybersecurity QA.
Results
MITRE-SAGE achieves the best overall performance on five of eight MITRE-QA tasks, with improvements up to 58% in multiple-choice accuracy and 35.5% in text-generation answer correctness.
Takeaways & Limitations
The results support MITRE-SAGE as a scalable and interpretable framework for reliable cybersecurity question answering across varying reasoning complexity.
Takeaways & Limitations
Future work must expand knowledge sources and improve integration of unstructured text with structured graph data for broader coverage and deeper reasoning.
Abstract
from arXiv · showhide
Effective cybersecurity operations require timely and accurate analysis of large-scale heterogeneous security information; however, analysts increasingly struggle with information overload, alert fatigue, and time-constrained decision-making. Although large language models (LLMs) have demonstrated promising capabilities for question answering (QA), their effectiveness in cybersecurity remains limited by insufficient domain knowledge, a tendency to hallucinate, and difficulties in capturing both semantic and structural relationships. This work proposes MITRE-SAGE, a multi-agent retrieval-augmented generation framework that integrates semantic and structural cybersecurity knowledge to improve the reliability and interpretability of LLM-based QA systems. By decomposing complex tasks into query interpretation, evidence retrieval, and answer synthesis, MITRE-SAGE effectively supports cybersecurity tasks such as vulnerability assessment, threat profiling, and relationship extraction. Furthermore, we propose MITRE-QA, a comprehensive benchmark comprising 3,000 question-answer pairs for evaluating LLMs across diverse cybersecurity knowledge tasks, and use it to systematically evaluate MITRE-SAGE against representative baseline methods. Extensive experiments demonstrate that MITRE-SAGE consistently outperforms standalone LLMs and conventional RAG approaches. Notably, a lightweight configuration comprising Qwen2.5-7B sub-agents and a Qwen2.5-14B orchestrator achieves superior performance on five of the eight benchmark tasks, indicating the effectiveness of the proposed multi-agent framework. The results highlight the potential of MITRE-SAGE as a scalable and interpretable approach for reliable cybersecurity QA, while MITRE-QA provides a standardized benchmark for future research.
1 Introduction
The introduction frames cybersecurity QA as a response to overwhelming, heterogeneous security data and limits of conventional SOC analytics and RAG pipelines. It presents MITRE-SAGE’s collaborative multi-agent framework and MITRE-QA benchmark as solutions for trusted grounding, semantic and structural reasoning, and rigorous evaluation.
- Motivation: Cybersecurity operations face massive, heterogeneous security data that overwhelms SOC analysts making high-stakes decisions under severe time constraints.Excessive false positives further cause alert fatigue, cognitive workload, and analyst burnout.
- Problem: Traditional rule-based SIEM correlation and machine-learning classifiers inadequately capture semantic complexity, contextual depth, and evolving attack patterns in security telemetry.These systems also lack the cognitive reasoning needed for actionable forensic insights.
- Problem: RAG improves factual accuracy and verifiability through domain-specific grounding, but traditional pipelines remain sensitive to retrieval noise and other failure points.Even a few irrelevant documents can mislead the generator.
- Proposed approach: MITRE-SAGE integrates knowledge-graph, text, and web sources through a hierarchical architecture of an Orchestrator Agent, three retrieval agents, and corresponding Summarization Agents.The Orchestrator coordinates retrieval and synthesizes the final response.
- Benchmark and evaluation: MITRE-QA evaluates semantic understanding and structural reasoning over entities and relationships from MITRE and NVD knowledge bases.The benchmark is designed to provide more comprehensive and rigorous cybersecurity model evaluation.
2 Related Works
Related work spans cybersecurity question answering, fine-tuning, retrieval-augmented generation, and agentic systems. Existing approaches offer domain adaptation, knowledge grounding, or multi-step coordination, but face limitations involving evolving threats, hallucination, data structure, and operational complexity.
- Question Answering: Cybersecurity QA models support AI-driven assistance in digital defense, extending question answering’s broader role in information retrieval and expert decision-making.QA systems are described as computational systems that process natural-language queries and retrieve precise information across sectors including medicine, finance, education, and law.
- Fine-Tuning in Cybersecurity: Fine-tuning updates model parameters with specialized cybersecurity datasets, but operational use is hindered by evolving threats, catastrophic forgetting, hallucination, and limited expert-curated data.The passage characterizes these models as often impractical for operational cybersecurity environments because they cannot readily adapt to rapidly changing threats.
- RAG in Cybersecurity: Cybersecurity RAG frameworks ground LLM responses in up-to-date external repositories and help mitigate hallucinations, yet remain sensitive to noisy documents and limited by flat representations and conventional chunking.These limitations hinder multi-hop semantic relationship modeling and can fragment critical information.
- LLM-based Agentic Systems: LLM-based agentic systems coordinate specialized agents through structured interaction and role division to manage multi-step workflows and integrate external tools for complex domain-specific operations.The paper adopts this approach with a multi-agent framework centered on graphs, text, and the web.
3 MITRE-QA Benchmark
MITRE-QA is a 3,000-question cybersecurity benchmark spanning eight tasks built from authoritative knowledge sources and existing out-of-distribution benchmarks. Its tasks assess single-hop and multi-hop reasoning across conceptual, entity-centric, relational, threat-intelligence, and detection-rule mapping capabilities.
- Benchmark Scope: MITRE-QA comprises 3,000 question-answer pairs constructed from reliable cybersecurity sources, including MITRE and the NVD, to evaluate cybersecurity knowledge understanding and reasoning.The benchmark is designed specifically for language-model question answering over cybersecurity knowledge.
- Benchmark Scope: The benchmark contains eight tasks: six newly designed from MITRE ATT&CK, CAPEC, CWE, and NVD CVE datasets, plus two adapted out-of-distribution tasks.The two adapted tasks are not directly represented in the knowledge sources used by MITRE-SAGE.
- Question Construction: Questions were generated with Python templates for five of six tasks, while Entity Identification questions were generated by GPT-4.1 from supplied documents.Template instantiation used entities, attributes, and relationships extracted from the underlying cybersecurity datasets.
- Reasoning Difficulty: The tasks are categorized as single-hop or multi-hop according to whether answers require one knowledge-source traversal or integration across multiple retrieval interactions.Single-hop tasks use one retrieval agent, whereas multi-hop tasks require at least two orchestrator-subordinate-agent interactions.
- Task Categories: Single-hop tasks cover conceptual understanding, entity attribute retrieval, structured knowledge retrieval, and entity identification, while multi-hop tasks cover relation-aware retrieval, threat profiling, relation-aware identification, and Sigma-to-ATT&CK mapping.These tasks assess definition answering, entity metadata extraction, graph traversal, semantic matching, relational threat inference, and mapping detection rules to adversarial techniques.
4 Method
MITRE-SAGE uses a central Orchestrator Agent with three specialized retrieval-agent groups, each paired with a Summarization Agent for knowledge-source-specific evidence processing. The framework combines query decomposition, graph and document retrieval, up-to-date information access, and evidence refinement before answer synthesis.
- Architecture: MITRE-SAGE comprises an Orchestrator Agent and three specialized retrieval-based agent groups, each containing a Retrieval Agent and Summarization Agent.The Orchestrator analyzes queries, selects retrieval sources, coordinates agents, and synthesizes collected evidence.
- Query orchestration: The Orchestrator uses few-shot chain-of-thought prompting to decompose requests into interpretable sub-queries and delegate them to appropriate agents.It evaluates returned evidence and can determine whether additional agent contributions are needed.
- Structured retrieval: The graph retrieval agent generates Cypher queries and executes them over a cybersecurity knowledge graph built from MITRE and NVD entities and relationships.Soft prompt tuning updates learnable virtual tokens while keeping the original language-model parameters frozen for text-to-Cypher generation.
- Textual retrieval: The document retrieval agent uses sparse retrieval for entity identifiers and attributes, and hybrid retrieval combining sparse and dense rankings through Reciprocal Rank Fusion.Documents are ranked by fused scores, with the highest-ranked results returned as the final retrieval set.
- Evidence refinement: A complementary retrieval agent obtains up-to-date topic information and clarifies technical terms when other agents provide insufficient evidence.Each retrieval group’s Summarization Agent removes irrelevant or redundant content, extracts query-relevant evidence, and forwards a concise context to the Orchestrator.
5 Experiments
MITRE-SAGE is evaluated against GPT-4.1 and a conventional hybrid RAG baseline on the MITRE-QA benchmark using task-specific accuracy and RAGAS-based generation metrics. It achieves the best overall performance on five of eight tasks, with improvements of up to 58% in multiple-choice accuracy and 35.5% in answer correctness over the strongest baseline.
- Baselines: The evaluation compares MITRE-SAGE with GPT-4.1 and a hybrid RAG system using Qwen2.5-32B with dense and sparse retrieval.GPT-4.1 serves as a general-purpose LLM baseline, while the hybrid RAG baseline combines dense and sparse retrieval mechanisms.
- Evaluation Metrics: Multiple-choice tasks use accuracy, while text-generation tasks use RAGAS answer relevance, answer similarity, and answer correctness.GPT-4.1-mini performs LLM-based evaluation, and text-embedding-3-large provides embeddings.
- Implementation: Experiments use Qwen2.5-14B as the orchestrator and Qwen2.5-7B for subordinate agents, deployed with 8-bit quantization on a single 80 GB NVIDIA A100 GPU.Responses are limited to 1,024 tokens, and orchestrator and retrieval agents use few-shot prompting.
- Overall Results: MITRE-SAGE achieves the best overall performance on five of eight MITRE-QA tasks and remains competitive on the other three.The benchmark covers four single-hop tasks and four multi-hop tasks.
- Overall Results: 58% and 35.5% are the maximum improvements over the strongest baseline in multiple-choice accuracy and text-generation answer correctness, respectively.These gains are reported across the MITRE-QA benchmark comparisons.
- Baseline Analysis: GPT-4.1 performs best on conceptual understanding and sigma-rules-to-attack-techniques mapping, whereas hybrid RAG performs best on Entity Identification.GPT-4.1 and hybrid RAG both decline substantially on multi-hop text generation compared with single-hop tasks.
6 Conclusion
MITRE-SAGE is introduced as a hierarchical multi-agent RAG framework that integrates structural and semantic cybersecurity knowledge to address limitations in LLM-based cybersecurity QA. Future work includes expanding knowledge sources and developing tightly integrated graph reasoning for heterogeneous cybersecurity data.
- Contributions: MITRE-SAGE integrates structural and semantic cybersecurity representations within a hierarchical multi-agent RAG framework.The framework is designed to address limitations of existing LLM-based cybersecurity question-answering approaches.
- Future work: Future work will expand the text vector store with cyber threat intelligence reports and other cybersecurity corpora.Entities and relationships extracted from these resources could enrich the cybersecurity knowledge graph, improving knowledge coverage and reasoning capabilities.
- Future work: Unified models combining unstructured text and structured graph data through advanced graph reasoning and tightly coupled GraphRAG remain promising.The goal is to enable deeper insights across heterogeneous knowledge ecosystems.