It is Not Yet Another Tool: Creating and Deploying an Agentic AI Companion in a Security Operations Center

1 Introduction

• SOC analysts must investigate mostly benign alerts, a monotonous task that involves assembling context and leaves them responsible for verifying AI-provided reasoning.The introduction argues that a companion should adapt to each SOC’s data sources, risk environment, and workflows rather than act as a rigid, one-size-fits-all tool.Source: §1 · pp. 1–2
• Over 14 months, two researcher-analysts used participant observation to build a ReAct-style agent that investigated alerts through SOC data sources and drafted reports for analyst review.Other analysts voluntarily used it on real alerts during the final four months of the fieldwork.Source: §1 · p. 2
• When outputs were easy to verify, analysts overwhelmingly reused them in closing reports; the authors identify verifiability, not accuracy alone, as key to adoption.The companion’s evidence was adapted to match the format of manual analysis.Source: §1 · p. 2
• Analysts shaped the companion with personalized prompts encoding their investigation routines and preferred report layouts, and this adaptability increased adoption..Source: §1 · p. 2
• 30−50% of ticket-processing time was estimated to be saved for the most prolific Companion users.Analysts also reported that closing tickets with the companion felt more productive than handling them manually.Source: §1 · p. 2

2 Related Work

• SOC studies find analysts use LLMs for sensemaking and communication while retaining decision authority, whereas autonomous summaries risk omissions and inaccuracies; collaboration can reduce analyst effort [4].Studies of Reddit discussions and interviews with 28 cybersecurity professionals across 26 organizations also report substantial LLM use alongside concerns about reliability, verification, security or data leakage, and organizational guidance.Source: §2 · p. 2
• This study follows the full cycle of designing, implementing, deploying, and observing an LLM-based agentic assistant in an operational SOC, using participant observation for first-hand data on analyst interactions.Researchers worked inside the SOC as analysts, enabling observation of everyday work and how analysts used the AI companion.Source: §2 · pp. 2–3
• Prior AI and LLM research spans penetration testing, phishing detection, vulnerability analysis, alert classification, and SOC triage [18] [19], while this work evaluates an assistant in a working SOC.It observes how analysts use and interact with the agentic assistant.Source: §2 · pp. 2–3
• Earlier security-operations research documents alert-validation burden, analyst burnout, and human, organizational, and tooling challenges, while field studies emphasize tacit knowledge, context, and practice.These works motivate evaluating AI assistance within everyday SOC practice and analysts’ activities and constraints, with practitioners involved in tool development and refinement.Source: §2 · p. 3

3 Methodology

• The study took place in a university SOC where student analysts primarily performed tier 1 triage, and the university IRB approved the research with voluntary verbal consent.Analysts combined formal procedures with local organizational knowledge; most were students working 10–20 hours per week. Participation was voluntary.Source: §3 · p. 3

3.1 Participant observation

• Two PhD researchers joined the SOC as tier 1 analysts, using participant observation and field notes to learn routine practices and inform design decisions.Working regular shifts helped them become part of the team, supporting the trust needed to study SOC work. Notes covered delays, tool switching, uncertainty, and analyst feedback.Source: §3.1 · p. 3

3.2 AI companion design and deployment

• Because triage was frequent, repetitive, and locally shaped, the Companion supported evidence gathering and report drafting while leaving final decisions to analysts.After iterative use by the embedded researchers, experienced analysts were offered voluntary use during the study’s final four months; they could choose whether to use its output or continue investigating manually. They were briefed on its capabilities, limitations, and data sources, and told to treat it as decision support.Source: §3.2 · pp. 3–4

3.3 Data collection and analysis

• The observational study was not experimentally controlled, and analysis drew on field notes, Companion logs, and ticket records using qualitative and quantitative approaches.Controlled experiments were infeasible; the data captured analysts’ interactions with the Companion during real SOC tasks.Source: §3.3 · p. 4

4.1 Design principles

• The Companion targets repetitive context assembly, not triage decisions, and presents evidence in analysts’ familiar reporting form so they can verify its work.Researchers’ participation in the alert queue revealed that analysts spent much of a shift moving information among disconnected systems, while judgment was a smaller share of elapsed time. Analysts also differed in preferred detail, report structure, and investigative habits, motivating an adaptable design.Source: §4.1 · p. 4

4.2 Workflow automation

• A ReAct-style agent automates dependent lookups across threat intelligence, SIEM, DHCP, and device-identification systems using a fixed set of read-only tools.The locally hosted agent uses schema-validated tool arguments; failed runs waste an investigation rather than changing SOC systems, and sensitive information stays inside the organization’s network.Source: §4.2 · p. 4

4.3 Empowering analysts to adapt the AI

• Analysts can adapt the Companion’s workflow and report layout by switching among named system-prompt versions, without researcher involvement or redeployment.Prompts specify tool order, when to abandon an inquiry, and report layout; analysts retain a shared default prompt if they have not created their own.Source: §4.3 · p. 4

4.4 Deployment and refinement

• Deployment with experienced analysts led to screenshot capture of tool results and a revised shared default prompt that changed report layout without changing reasoning or tool use.The Companion was deployed on an NVIDIA DGX Spark inside the SOC; Section 5 reports the impact of these refinements.Source: §4.4 · p. 5

4.5 Final design

• The final design combines a server-side ReAct agent, tools for manual triage systems, editable prompts, and structured reports with assessments, recommendations, and screenshot evidence.Through a web interface, analysts can inspect streamed and retained reasoning, tool calls, and responses; add context, request investigation or follow-ups, branch to explore alternatives, and change settings before rerunning. Runs persist across page refreshes, time out explicitly, and allow per-investigation model selection; GLM-4.7-Flash (30B) was used for most of deployment.Source: §4.5 · pp. 5–6, Table 1 · p. 5

5.1 Corpus and coding

• The analysis paired Companion transcripts with analysts’ final ticket reports for 108 tickets from six experienced analysts, with coding agreement of 99.23% and Cohen’s κ = 0.965.Use was voluntary and nonrandom, analysts were influenced by the Companion rather than independent accuracy evaluators, and the system changed during deployment.Source: §5.1 · pp. 6–7

5.2 How much of the Companion’s work reached the ticket closing report

• 59 tickets (55%) had at least 95% draft reuse, while 28 (26%) had less than 75%; overall median reuse was 96.7% and mean reuse was 85.5%.The distribution was concentrated near whole-draft reuse or rejection, rather than frequent piecemeal editing.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3 Verifiability is key to adoption

• After screenshots were introduced, median draft reuse rose from 50.0% across 3 tickets to 96.7% across 105, a descriptive rather than controlled comparison.After the change, 90% of tickets kept at least some captured screenshots; analysts valued evidence they could verify and attach to reports.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Three layers of adoption: content, verdict, and recommendation

• Analysts reused the Companion’s content in 98 tickets (91%), agreed with its verdict in 72 (67%), and kept its recommendation in 74 (69%).On 34 tickets, analysts removed the Companion’s proposed action and recommended no action; they did not add or escalate an action it had not proposed.Source: §5.4 · p. 9, Figure 4 · p. 9
• Analysts struck recommendations on 24 of 40 Companion-uncertain findings and 10 of 30 malicious findings; conflicting verdicts coincided with lower reuse than matching verdicts.The Companion often hedged where analysts used contextual judgment, while outright disagreements reduced reuse; transcripts also document missing attribution, prompt scaffolding, invalid calls, and a fabricated indicator caught by an analyst.Source: §5.4 · pp. 9–10

5.5 How analysts steered the Companion

• Only 17 tickets had analyst interventions; intervened drafts were written off on 24% of tickets versus 7% without intervention, while 14 tickets were rerun in new threads.Rerun tickets had median reuse of 91.2% versus 97.0% on the rest, but were not written off more often (7% versus 10%).Source: §5.5 · pp. 10–11, Table 8 · p. 11
• Analyst-written prompts had median reuse of 98.2% across 55 tickets, versus 69.8% across 38 tickets under the original default prompt.Utilization rose from 1.4% in weeks 1–4 to 47.9% across the final four weeks, peaking at 65.4% in week 15; it increased after screenshots and again after a revised default prompt, though the latter changed report layout only.Source: §5.5 · p. 11, Figure 7 · p. 11, Figure 6 · p. 11

5.6 Time

• For all 108 tickets, researchers recorded elapsed time from when an analyst loaded a ticket into the Companion until the analyst closed it, including interruptions.The supplied passage describes the timing measure but gives no time results in this part.Source: §5.6 · p. 11
• For the three most prolific Companion users, estimated time savings appear to be 30%-50%; A2 and A3 measured 9.9 and 7.6 minutes versus manual estimates of 10–20 and 15 minutes, respectively.A5’s observed median of 20.6 minutes was within the 20–30-minute estimate for complex tickets, while their follow-up estimate indicated savings of at least 30%.Source: §5.6 · p. 12

5.7 Analyst feedback

• Analyst A2 valued the Companion for shifting time from boring procedural tasks toward more in-depth ticket analysis.A2 described the benefit as making time to analyze better, not merely completing tickets faster.Source: §5.7 · p. 12
• Analyst A3 said greater trust would require seeing the pathway by which the Companion reached its conclusion.This feedback identified verifiability as a condition for trusting the system more.Source: §5.7 · p. 12
• Analysts A2, A3, and A5 asked when the Companion would return during outages, supporting a pattern of expecting it for routine triage.The authors cautioned that these comments are not a quantitative measure of adoption.Source: §5.7 · p. 12
• Analyst A6’s difficulty finding a critique was consistent with the design stabilizing after earlier analyst feedback had shaped the Companion.A6 joined near the end of deployment, after that feedback had already influenced the system.Source: §5.7 · pp. 12–13

6 Discussion

• The Companion streamlined rather than disrupted existing work because its outputs mirrored manual analysis and its design reflected practices and constraints observed in the SOC.The authors question how readily a companion developed through fieldwork in one SOC could be adapted to another with minimal help from its original authors.Source: §6 · p. 13
• Analysts initially scrutinized AI reports, verified external lookups through screenshots, changed verdicts or recommendations they disagreed with, and caught mistakes or hallucinations.The Companion was not officially sanctioned, so analysts used it at their own risk; how trust might change after formal management approval remains an open question.Source: §6 · p. 13

Limitations

• A single university SOC limits how directly findings transfer to enterprise, government, or managed-security settings, though detailed fieldwork and records can support grounded comparisons.The analysis combines ethnographic work with Companion logs and ticketing-system records.Source: §7 · p. 13
• Voluntary use made the ticket corpus nonrandom and unevenly represented analysts, limiting visibility into cross-ticket and cross-analyst variation while capturing use in real work.Analysts chose which tickets to bring, and a small number accounted for most Companion-assisted tickets.Source: §7 · p. 13
• The Companion’s prompt, model, visual evidence layer, and integrations changed during deployment, so the study did not compare a fixed system under controlled conditions.The system evolved as analysts used it and it was refined.Source: §7 · p. 13
• Time estimates are approximate because run-to-close time includes interruptions and manual baselines rely on analysts’ self-reported estimates rather than matched observations.Those estimates may be considerably inaccurate.Source: §7 · p. 13
• Researchers’ dual role as embedded analysts and system builders could bias interpretation, so analysis also drew on logs, ticket records, report-reuse measures, and analyst feedback.The embedded role provided access to the work practices that shaped the Companion.Source: §7 · p. 13

8 Conclusion

• Analysts overwhelmingly adopted the Companion’s analysis in ticket closing reports when results were easy to verify, and retained its main findings even when rejecting recommendations.The study reports adoption of outputs in closing reports, including cases where analysts did not follow the Companion’s recommendations.Source: §8 · p. 13
• When the Companion adapted to analysts’ work needs and habits, adoption increased considerably.Source: §8 · p. 13
• Using the Companion saved time in closing tickets and improved productivity while leaving analyst judgment intact.The conclusion draws on both the researchers’ measurement and analysts’ self-estimates of time saved.Source: §8 · p. 13

• Analysts participated voluntarily after informed consent, and the IRB-approved protocol framed the Companion as experimental decision support rather than an authority.Source: Ethical Considerations · p. 15
• Analysts were anonymized in reporting, identifying details were removed from quotations and examples, and only the research team could access Companion logs.Source: Ethical Considerations · p. 15
• Sensitive operational data were not released; de-identified examples were used, while locally hosted models and restricted access kept SOC records within the organization’s environment.Research artifacts minimized sensitive information and retained only data needed for the reported analyses.Source: Ethical Considerations · p. 15
• The Companion was limited to predefined, schema-validated, read-only operations, while analysts reviewed evidence and made every final ticket decision.It could not modify systems, close tickets, disable accounts, or take other remediation actions.Source: Ethical Considerations · p. 15

Open Science

• The analysis codebook and code frequencies are available to readers in the paper (Tables 9 and 2, respectively).The codebook appears in Appendix B.Source: Open Science · p. 15
• Companion deployment data cannot be shared because they contain security-sensitive organizational information and some field notes are protected by the IRB.The data include transcripts, tool responses, screenshots, ticket records, and researchers’ field notes; the non-disclosure agreement also bars sharing.Source: Open Science · p. 15
• The Companion’s source code and prompts cannot be shared because they reveal SOC practices, triage procedures, detection coverage, and potential security blind spots.Removing sensitive details would leave a system unlike the one built and evaluated.Source: Open Science · p. 15

A Sample Ticket

• Figure 9 shows a redacted real alert from the SOC.Source: Figure 9 · p. 19

B Codebook

• Usage codes compare the Companion’s draft with the analyst’s filed closing report; intervention codes cover later analyst messages, and other codes describe ticket handling and Companion behavior.Usage codes are U, intervention codes are I, and other codes are O.Source: B Codebook · p. 19
• U1–U3 and U6–U8 are each mutually exclusive, while all other codes may co-occur freely.Tickets without Companion-captured screenshots receive no U6–U8 code.Source: Table 9 · p. 20
• Intervention examples reproduce analyst messages verbatim, including their original typographic errors.Source: Table 9 · p. 20