Source-linked AI summary
It is Not Yet Another Tool: Creating and Deploying an Agentic AI Companion in a Security Operations Center
Kritan Banstola, Faayed Al Faisal, Duy Dao, Ryan Irving, Daniel Lende, Xinming Ou
TL;DR
SOC ticket triage is repetitive, while documentation of how agentic AI assistants operate in practice remains limited. This paper uses embedded fieldwork to design and deploy an AI companion in a university SOC, finding that verifiable, adaptable outputs were embraced and saved analysts time.
Problem
SOC ticket processing is monotonous and laborious, while how vendor-provided agentic AI assistants operate has little documentation beyond marketing materials.
Method
Researchers conducted 14 months of participant-observation fieldwork in a university SOC and created a locally deployed ReAct-style LLM companion that investigated alerts using the SOC workflow.
Results
Analysts overwhelmingly adopted easy-to-verify Companion outputs in ticket closing reports, and prolific users saved an estimated 30−50% of ticket-processing time.
Takeaways & Limitations
Embedding the Companion in SOC practice and adapting it to analysts’ needs supported adoption while streamlining the existing workflow rather than disrupting it.
Takeaways & Limitations
The study was conducted in a single university SOC, whose alert mix, organizational structure, analyst experiences, and evidentiary practices may differ from other security environments.
Abstract
from arXiv · showhide
Security Operations Centers (SOCs) process large amounts of tickets, most of which are low-interest events not worthy of further investigation. The repetitive nature of this task and similarity of the vast amounts of tickets make it a prime candidate for generative AI-based automation. We created and deployed an agentic AI companion utilizing large language models through fieldwork within a SOC for over one year. The design of the SOC AI companion was driven by researchers' participation and interactions within the SOC's daily work. SOC analysts were invited to use it during the last four months of the fieldwork. We analyzed the analysts' usage of the companion and found that in more than 90% of the cases the companion's outputs were reused by analysts in the ticket's closing report. Our results showed that when designed "in the trenches" with the intended users, a SOC AI companion can go beyond being yet another tool, but rather a system that co-evolves with its human users as it traverses through the various types of workloads. Analysts naturally started to shape the AI companion's behaviors to fit their particular needs. Our data show that the more human analysts shape the AI companion's behaviors, the more they become comfortable trusting the output from the AI system, resulting in improved productivity.
1 Introduction
SOC alert triage is repetitive and laborious, yet trustworthy automation must preserve analyst verification and adapt to local workflows. This paper presents a field-designed AI companion whose outputs were widely reused and reduced ticket-processing time.
- Motivation: Most SOC alerts are benign, but analysts must still assemble context across tools to determine whether each indicates an intrusion.This repetitive work contributes to fatigue and attrition, motivating generative-AI assistance.
- Design challenge: Analysts must verify an AI companion’s reasoning because they remain responsible for ticket decisions, and excessive verification would defeat automation.The paper identifies trust and verification effort as central adoption constraints.
- Design challenge: A fixed, one-size-fits-all companion may fail to fit each SOC’s data sources, risk environment, organizational structure, and workflow differences.The proposed alternative is a companion that co-evolves with analysts and their changing work environment.
- Approach: The study used 14 months of participant-observation fieldwork to design and deploy a ReAct-style LLM agent that investigates alerts and drafts evidence-supported reports.The companion followed analysts’ workflow by querying SOC sources, enriching indicators, and presenting evidence in the manual-analysis format.
- Findings: Designing around existing workflows and making evidence easy to verify increased adoption, while analysts shaped behaviors through personalized system prompts.The paper argues that a companion can offload work without forcing analysts to reorganize their workflow.
2 Related Work
Prior research has examined LLM assistance across SOC and incident-response tasks, documenting usefulness alongside reliability and verification concerns. This work extends that literature by studying the full design, deployment, and use of an agentic assistant inside an operational SOC.
- Prior LLM research: Recent studies examine LLM use in SOC and incident-response settings, including sensemaking, telemetry interpretation, technical communication, and incident summarization.The cited work includes 3,090 queries from 45 SOC analysts and 50 real-world incidents involving 18 analysts.
- Prior LLM research: Existing findings report that autonomous summaries can omit critical details or introduce inaccuracies, whereas collaborative assistance can reduce effort and improve readability and consistency.Other studies also identify concerns about reliability, verification, security, data leakage, and organizational guidance.
- Scope of prior work: Earlier cybersecurity research covers tasks such as penetration testing, phishing detection, vulnerability analysis, alert classification, and SOC triage.This paper distinguishes itself by evaluating an agentic assistant through observation in a working SOC.
- Security-operations foundations: The study builds on security-operations research emphasizing false-positive burden, analyst burnout, tacit knowledge, organizational practice, and context-sensitive tool design.These strands motivate evaluating security tools in relation to the work practices where they are used.
3 Methodology
The researchers conducted a longitudinal embedded study combining participant observation, iterative companion design, and deployment during live SOC ticket work. They analyzed field notes, system logs, and ticket records while preserving analysts’ decision authority.
- Study design: The 14-month study combined participant observation, iterative AI-companion development, and analysis of analysts’ live use during ticket work.These three efforts ran concurrently within the SOC.
- Participant observation: Two PhD researchers joined the SOC as tier 1 analysts, working regular shifts to learn practices through participation rather than documentation or interviews alone.Their embedded role helped them become trusted members of the team.
- Data collection: Field notes recorded triage practices, delays, tool switching, investigation uncertainty, analyst feedback, and potential automation opportunities.Collaborative analysis converted these observations into subsequent design decisions.
- Research focus: Alert triage emerged as the main intervention target because it was frequent, repetitive, and strongly shaped by local organizational practice.Analysts repeatedly gathered evidence from multiple systems while retaining responsibility for judgment informed by local context.
- Deployment: Researchers first used the companion in their own triage, then voluntarily introduced it to experienced analysts during the final four months.Analysts could rely on its output, continue manually, or decide whether to use it on each ticket.
- Study boundary: The observational study prioritized authentic SOC interactions because controlled experiments were not feasible.The resulting evidence reflects analysts’ real ticket-solving behavior rather than randomized comparisons.
- Analysis: Researchers analyzed field notes, Companion logs, and ticketing-system records using complementary qualitative and quantitative approaches.Logs included prompts, model responses, tool activity, evidence, prompt versions, and timestamps.
4 SOC AI Companion Design Evolution
The Companion was designed to reduce repetitive evidence gathering while keeping analysts responsible for judgment, using an adaptable agent, familiar tools, and inspectable reports. Its architecture supported persistent investigations, streamed reasoning, and SOC-specific evidence collection.
- 4.1 Design principles: The design targeted context assembly rather than automated ticket decisions, because analysts spent much of their time moving information across disconnected systems.The Companion was intended to reduce repetitive evidence-gathering and reporting labor while preserving analyst inspection and control.
- 4.1 Design principles: The Companion needed to remain adaptable rather than behave as a fixed, one-size-fits-all tool.This requirement follows from analysts’ different investigation routes and reporting preferences.
- 4.2 Tool integration: The agent assembled information from OSINT and threat-intelligence services, the SIEM, DHCP lookup, and device-identification systems.These sources corresponded to systems analysts already used manually during triage.
- 4.2 Tool integration: The system was designed so failed runs wasted an investigation without changing dependent systems, while sensitive internal information remained inside the organization’s network.This constrained the Companion to evidence gathering rather than direct modification of SOC infrastructure.
- 4.3 Analyst interaction: A web interface let analysts open investigations, ask follow-ups, interrupt or branch runs, and inspect reasoning steps and raw tool responses.The interface also recorded transcripts, tool calls, prompt edits, settings, and run timing.
- 4.4 Agent architecture: A ReAct-style agent ran investigations on the server, iterated between reasoning and tool calls, and produced reports after retrieving the ticket.Server-side execution allowed runs to survive page refreshes or a closed laptop and terminated over-budget runs as failed.
- 4.5 Final design: The output layer produced structured reports following SOC conventions, combining assessment, recommendations, and supporting screenshots.The evidence presentation was intended to match manual analysis and remain easy for analysts to verify.
5 Data Analysis and Findings
Across 108 tickets, analysts usually reused the Companion’s draft in closing reports, but adoption depended strongly on verifiable evidence and agreement with analyst judgment. The deployment also revealed clear scope limits and failure modes, including non-random voluntary use, changing system configurations, missing tool results, invalid calls, and unsupported outputs.
- Limitations: The corpus was not a random alert sample, analysts were not independent accuracy evaluators, and the Companion changed prompts and models during deployment.The authors therefore frame the study around how analysts used the Companion in practice rather than as an unbiased accuracy evaluation.
- Draft reuse: Median draft reuse was 96.7%, but reuse was polarized: 59 of 108 tickets reached at least 95%, while 28 fell below 75%.Analysts tended to accept drafts nearly wholesale or rewrite them substantially rather than edit them incrementally.
- Verifiability and adoption: After screenshots were attached, median reuse rose from 50.0% across three earlier tickets to 96.7% across 105 later tickets.The authors treat this comparison as descriptive rather than a controlled baseline because only three tickets preceded the change.
- Verifiability and adoption: 90% of post-change tickets retained at least some captured screenshots, indicating that visual evidence was important to analysts’ acceptance of the Companion’s output.The initial text-only presentation of tool results led analysts to verify queries manually, whereas rendered screenshots made the evidence easier to inspect.
- Adoption across output layers: 91% of tickets reused the Companion’s draft text in whole or in part, while 69% retained its recommendation and 67% matched its verdict.Content adoption therefore exceeded adoption of the Companion’s judgment and recommended action.
- Verdict and recommendation adoption: When Companion and analyst verdicts conflicted outright, median reuse fell to 67.9% versus 97.4% when verdicts matched.Analysts often removed recommendations or changed verdicts, especially for uncertain and malicious findings, while retaining much of the surrounding draft.
6 Discussion
Embedding the AI Companion in actual SOC practice helped align its outputs with analysts’ workflows and made results easier to verify. Analysts remained able to review, modify, or reject its recommendations.
- Embedding the Companion in the SOC helped its actions and outputs mirror manual analysis, streamlining rather than disrupting existing workflows.The authors connect this fit to firsthand awareness of the SOC’s specific practices and constraints.
- Analysts viewed generated reports cautiously until screenshots enabled visual verification of the Companion’s external-tool lookups.They modified verdicts and recommendations when they disagreed and identified some mistakes or hallucinations.
- Analysts retained judgment by reading reports, changing recommendations when necessary, and identifying cases where the Companion made mistakes or hallucinated.
7 Limitations
The study’s evidence is bounded by a single university SOC, voluntary and uneven usage, an evolving system, approximate time comparisons, and researchers’ dual roles. These conditions support detailed naturalistic observation but limit controlled and broadly generalizable comparisons.
- The study was conducted in a single university SOC, whose alert mix, organization, analyst experience, and evidentiary practices may differ from other security environments.The authors present the embedded analysis as detailed evidence for grounded comparisons with other SOCs.
- Voluntary use produced a non-random ticket corpus because analysts selected tickets, varied in representation, and used the Companion under differing workload conditions.
- The Companion’s prompts, model, visual evidence layer, and integrations evolved during deployment, preventing a controlled comparison of a fixed artifact.
- Time analysis was approximate because run-to-close durations included interruptions and manual baselines relied on self-reported estimates rather than matched controls.
- Researchers’ dual roles as embedded analysts and system builders created possible interpretive bias, mitigated through logs, ticket records, reuse measures, and analyst feedback.
8 Conclusion
The study found that an AI Companion designed through embedded fieldwork was adopted as decision support when it adapted to analysts’ needs and presented verifiable results. Its use saved time without overriding analyst judgment.
- A 14-month university-SOC fieldwork study examined the design, deployment, and usage of an AI SOC Companion.
- The Companion was embraced when researchers designed it in the trenches, adapted it to analysts’ work needs, and presented analysis results in verifiable form.
- Analysts overwhelmingly reused the Companion’s output in ticket-closing reports, including main findings when they rejected proposed recommendations.
- Using the Companion saved time in closing tickets while serving as an aid that did not override analysts’ judgment.
Ethical Considerations
The study used voluntary, consented participation and protected analysts’ identities and sensitive operational data. The Companion operated as read-only decision support, while analysts retained responsibility for final security decisions.
- The research protocol received university IRB approval, and SOC analysts participated voluntarily after receiving information and providing verbal informed consent.
- Analysts were anonymized in reporting, while names, usernames, schedules, identifying details, and raw Companion logs remained restricted.
- Raw tickets, interaction logs, screenshots, and sensitive operational data were withheld publicly; paper examples were de-identified and generalized.
- Locally hosted language models kept sensitive SOC data within the organization’s environment, with access restricted to the research team.
- The Companion used predefined, schema-validated, read-only operations and could not modify systems, close tickets, disable accounts, or perform remediation.
Open Science
The paper makes its analysis codebook available but cannot share its most important deployment data or tailored source code because they contain security-sensitive and protected information.
- Open Science: The analysis codebook is available in Table 9 of Appendix B, with code frequencies reported in Table 2.These materials are available within the paper itself.
- Open Science: The deployment data include Companion transcripts, tool responses, screenshots, ticket records, and researchers’ field notes.These data underpin the research findings but contain live security telemetry and detailed incident descriptions.
- Open Science: The deployment data cannot be shared because of the SOC’s nondisclosure agreement and institutional review board protections.Some field notes are restricted to the research team and relevant supervising authorities.
- Open Science: The Companion’s source code and system prompts are also withheld because they reveal the fieldwork SOC’s practices and triage procedure.The tool layer targets the SOC’s systems, while prompts encode consulted sources, sufficient evidence, and abandonment conditions.
A Sample Ticket
The sample ticket presents a redacted SOC alert involving network communication that matched a CloudFlare tunnel domain pattern in TLS SNI.
- A Sample Ticket: The analysis records an internal source IP communicating with an external destination over destination port 443/TCP.The source port is listed as 34770/TCP.
- A Sample Ticket: The network traffic matched the signature “Default CloudFlare Tunnel Domain Pattern in TLS SNI.”
- A Sample Ticket: The sample is a real alert from the SOC, shown in redacted form.
B Codebook
The codebook defines how ticket usage, analyst interventions, and Companion behavior were coded, including exclusivity rules for selected usage categories.
- B Codebook: Usage codes compare the Companion’s draft with the analyst’s filed closing report.
- B Codebook: Intervention codes cover analyst messages sent after the opening request that initiated the run, across any thread of the ticket.
- B Codebook: Table 9 specifies the rule applied for each code listed in Table 2.The frequencies cover 108 coded tickets.
- B Codebook: U1–U3 and U6–U8 are mutually exclusive code groups, while other codes may co-occur freely.Tickets without Companion screenshots receive no code from U6–U8.