PaperlayerView exact revision on arXiv

Source-linked AI summary

From Hype to ROI: An Economic Model for LLM Adoption in Information Systems and Security Operations

Pellegrino Casoria, Andrea Araldo, Rosario Patanè, Simon Pietro Romano

arXiv:2610.33352v1cs.CRcs.AIcs.CE

TL;DR

Organizations struggle to turn LLM pilots into production value, and adoption payoffs depend on operational context. This paper models LLM adoption economics under uncertainty and applies the framework to SOCs. Most analyzed configurations are not reliably profitable; returns depend more on model autonomy than price, provided accuracy is sufficient.

  • Problem

    Organizations struggle to translate LLM pilots into production value, despite broad interest in adoption.

    Source: §1 · p. 1

  • Method

    The paper frames LLM adoption as an investment under uncertainty and models SOC ROI distributions and potential losses to support go/no-go decisions.

    Source: §7 · p. 14

  • Results

    Most configurations are not reliably profitable; high-accuracy models pay off only when capital costs and uncertainty are low, and greater autonomy matters more than model price.

    Source: §6 · p. 13, §7 · p. 14

  • Takeaways & Limitations

    Organizations can enter their own operating figures into the released code to assess adoption profitability, capital-cost limits, and loss risk.

    Source: §7 · p. 14

    Security accuracy inputs rely on heterogeneous general cyber benchmarks that cover disjoint models and subdomains and can report different scores for the same model.

    Source: §6.1 · p. 14

Abstract

from arXiv · show

Large Language Models (LLMs) promise significant efficiency gains across information-intensive operations by automating repetitive language-based tasks. However, adoption decisions remain largely opportunistic, as organizations struggle to move from pilots to sustainable industrialization. Without systematic economic frameworks, practitioners cannot assess whether LLM investments create value in a given context. This paper develops a general economic model to support informed LLM adoption decisions in operational information systems. Facing high uncertainty, we represent parameters as random variables centered on authoritative benchmarks, and evaluate the model by Monte Carlo simulation across pessimistic-to-optimistic assumptions on LLM efficiency. The ROI distributions show the probability of profitability and the Value at Risk for each configuration, together with a sensitivity analysis of its main drivers. As a first instantiation, we apply the model to Security Operations Centers (SOCs), mapping workflows onto established standards to identify tasks suited to LLM augmentation. The model balances productivity gains, driven by alert volume, analyst effort, and LLM efficiency, against operational costs, including inference, infrastructure, and error remediation, covering both cloud API and self-hosted deployments. Results show that adoption economics are context-dependent, and that automation is an exploitable lever, gated by accuracy, while the cost of errors and the model's price are secondary. Across the analyzed models and prices, most configurations are unprofitable, with profitability limited to high-accuracy, low-uncertainty deployments. The model gives organizations a quantitative basis for evidence-based adoption decisions, by locating the boundary between profit and loss, and the loss risked beyond it.

1. Introduction

LLM pilots often fail to reach measurable production value, while practitioners lack tools to estimate ROI for a specific operational context. The paper develops a benchmark-grounded economic model and simulation-based decision support, first applied to SOCs.

1. Introduction

  • Organizations struggle to convert widespread AI pilots into production value, and estimating ROI in advance is a frequently reported obstacle to AI adoption.Reports cited in the introduction include 95% of enterprise generative AI implementations failing and only 10–15% of organizations reporting measurable business impact.Source: §1 · p. 1
  • SOC capacity pressures make it a high-stakes first setting for the model, as attack volumes and workforce shortages strain security teams.The paper cites 4,875 incidents recorded by ENISA from July 2024 to June 2025 and a global cybersecurity workforce gap of 4.8 million.Source: §1 · pp. 1–2

1.1. Contributions of this work

  • The model weighs productivity gains against validation, error-remediation, inference, and infrastructure costs, while accounting for uncertainty in LLM effectiveness.It derives closed-form conditions for the maximum capital expenditure a deployment can bear while still creating value.Source: §1.1 · p. 2
  • Benchmark-grounded parameters and deployment archetypes spanning cost, accuracy, and deployment characteristics provide a reusable basis for evaluation.The parameterization includes salaries, task effort, work-item volumes, and LLM cost and accuracy.Source: §1.1 · p. 2
  • Monte Carlo simulation estimates each configuration’s profitability probability and value at risk, while sensitivity analysis identifies influential parameters.The implementation is released as open source for analysis under organizations’ own operating conditions.Source: §1.1 · p. 2
  • Most analyzed configurations are unprofitable; profitability is limited to high-accuracy API deployments with low capital costs and low uncertainty.Under low uncertainty, full autonomy raises the capital expenditure supported by the most accurate models from about 10% to 80–90% of baseline operating costs, while eliminating rework raises it only to 20%.Source: §1.1 · p. 2

2. Background and Related Work

Prior work examines technology adoption, LLM capabilities, and SOC operations, but does not provide a quantitative tool for assessing LLM adoption economics before deployment. The paper grounds an organizational cost-benefit model in SOC tasks, roles, and operational metrics.

2. Background and Related Work

  • Prior research characterizes LLM capabilities and deployment but leaves adoption’s economic implications largely unexplored, with no quantitative framework for evaluating economic value.The review connects economic modeling foundations with empirical work on SOC economics and productivity.Source: §2 · p. 2

2.1. Methodological and Economic Foundations

  • The model extends task- and model-level AI economics to organizational adoption by combining accuracy, verification and rework costs, and inference costs under uncertainty.It uses distributional reasoning and simulation to represent plausible ROI outcomes and interactions among uncertain parameters.Source: §2.1 · pp. 2–3

2.2. SOC Economics and Productivity

  • Existing SOC and LLM research does not model adoption costs, ROI, or the analyst-automation tradeoff, so the paper proposes a pre-adoption quantitative cost-benefit model.The model explicitly represents accuracy, token and infrastructure costs, and error remediation.Source: §2.2 · p. 3

2.3. A Reference SOC Model

  • The reference SOC model uses NIST SP 800-61 revisions 3 and 2 to map incident response structure and analyst-level actions.The paper notes that SOC and incident-response functions are combined in most organizations.Source: §2.3 · p. 3

2.3.1. Tasks

  • The simulations count recurring Detection and Analysis work, while excluding site-specific containment, eradication, and recovery execution.The selection of response measures remains in scope as the closing step of analysis; critical-severity cases are redirected to a dedicated response team.Source: §2.3.1 · p. 4

2.3.2. Professional Roles

  • The role model combines ITIL’s functional and hierarchical escalation distinctions with the practice-based L1–L3 analyst levels.Functional escalation maps task complexity across levels, while hierarchical escalation sets each level’s seniority.Source: §2.3.2 · p. 4

2.3.3. Performance Metrics

  • Because no single standard governs SOC performance metrics, the paper uses SOC-CMM’s monitoring and incident-management categories to derive core model variables.The selected time metrics include Mean Time to Pick-up, Mean Time to Analyse, and Mean Time to Respond.Source: §2.3.3 · p. 4

3. Methodology: The Cost Model

The cost model compares baseline operations with LLM-assisted operations, then evaluates savings and investment risk under uncertain inputs. It provides thresholds for assessing when adoption can meet a target return.

3.1. Cost without LLM

  • The baseline personnel cost combines event volume and severity, the share needing attention, task effort, role allocation, and employee costs.Events not requiring attention are handled by traditional frameworks without substantial human intervention.Source: §3 · p. 4, §3.1 · p. 5

3.2. Cost when using LLMs

  • LLM-assisted effort accounts for time saved, answer accuracy, and manual rework, with model assignments allowed to vary by task, criticality, and role.The speed ratio includes prompting and checking; incorrect answers are assumed to be detected and redone, preserving human-baseline output quality.Source: §3.2 · p. 5

3.2.1. Technology cost (𝐶𝑚 tech )

  • Cloud API and self-hosted deployments separate inference from GPU infrastructure costs; linear transfer of reference efficiency and omitted serving optimizations make GPU cost estimates upper bounds.Self-hosting assumes rented GPUs, while cloud APIs incur token charges and no owned infrastructure cost.Source: §3.2.1 · pp. 5–6

3.3. Operational Savings, Capital Expenditure Ratio, and Return on Investment (ROI)

  • Operational savings are baseline costs minus LLM costs, and Proposition 1 sets a capital-expenditure threshold for achieving a target return.Capital costs cover adoption work such as workflow changes and training, are difficult to estimate, and are amortized over three years; escaped errors reduce expected savings.Source: §3.3 · p. 6

3.4. Uncertainty Characterization

  • Probabilistic sensitivity analysis uses 104 independent Monte Carlo draws, Beta and Gamma distributions, and a common uncertainty coefficient of variation across parameters.At higher uncertainty levels, overlapping output distributions make the analysis uninformative.Source: §3.4 · pp. 6–7

3.5. Profitability and Value at Risk

  • Profitability probability and Value at Risk quantify return confidence and downside exposure, while a known capital-expenditure ratio yields a deterministic adoption threshold.The analysis does not assume that greater capital expenditure improves LLM performance because no quantified relationship is available.Source: §3.5 · p. 7

4. Parameter Calibration

The calibration combines benchmark-based inputs with practitioner judgment and treats uncertain parameters probabilistically. It instantiates the model for SOC workflows, analyst costs, LLM archetypes, and deployment costs.

4.1. Model Instantiation (, , , )

  • The SOC model covers triage, analysis, and documentation across L1–L3 roles, excludes critical alerts, and compares three API with three self-hosted archetypes.GPU cost is assigned to operating or capital expenditure depending on whether GPUs are rented or owned.Source: §4.1 · p. 8

4.2. Personnel Cost (𝑐𝑟)

  • Analyst costs use BLS wage percentiles mapped to SOC levels, with a 1.43× fully loaded multiplier for employer benefits and other compensation costs.L1 maps to P10–P25, L2 to P25–P50, and L3 to P50–P75; the mean hourly wage for each band supplies the model’s role cost.Source: §4.2 · p. 8, Table 3 · p. 8

4.3. Alert Volume and Severity (𝑁, 𝑝𝑎, ℎ𝑎, Δ𝑇)

  • The baseline is 1,000 alerts per day over a 24-hour window; critical incidents are excluded, and modeled alerts are true positives reaching analysts after upstream pre-triage.The severity mix and true-positive rates draw on operational experience and practitioner review.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Analyst Effort and Allocation (𝜏𝑘,𝑎, 𝛼𝑎,𝑟)

  • Because task-level effort lacks a standardized benchmark, practitioner-reviewed estimates are uncertain; each alert’s effort is allocated 20% to triage, 50% to analysis, and 30% to documentation.Effort is allocated across roles by severity, with more complex high-severity work assigned to more senior analysts.Source: §4.4 · pp. 8–9, Table 5 · p. 9

4.5. LLM Efficiency and Rework (𝜂, 𝛽)

  • The model uses a common uncertain speed ratio η=0.75, representing a 25% time reduction, and a common rework penalty β=η for a late-detected error requiring a full manual redo.The efficiency evidence is an imperfect proxy: benchmarks vary and may be outdated, so the model uses illustrative accuracy-tiered archetypes rather than ranking individual models.Source: §4.5 · pp. 9–10

4.7. Infrastructure and Inference Demand (𝑐gpu, 𝑉gpu, 𝜙𝑚, 𝑡𝑜𝑘in∕out

  • Self-hosted infrastructure is sized using H100 GPUs at $8 per hour, with throughput anchored to a conservative 3,000 tokens per second for an 8B dense reference model.Input and output throughput are treated equally for an upper-bound GPU cost estimate; inference demand varies by severity and uses a 4:1 input-to-output token ratio.Source: §4.7 · p. 10, Table 8 · p. 10

4.8. Economic Parameters (𝜃, 𝐶𝑖𝑛𝑣, 𝜆)

  • The evaluation sweeps the known capital expenditure ratio θ, sets non-operational cost and baseline miss rate to zero, and treats economic parameters as deterministic.Under these assumptions, θ expresses capital expenditure relative to the personnel cost affected by the LLM.Source: §4.8 · p. 10

5. Results

The SOC results compare profitability and downside risk across deployment archetypes and operating assumptions. Profitability is most responsive to automation, while uncertainty, accuracy, and self-hosted GPU costs shape where adoption is viable.

5.1. Profitability under uncertainty

  • At CV=5%, high-confidence capex limits extend from below 10% for high-accuracy APIs (β=η) to about 80–90% for leading models under full automation (η=0).With β=0, the confident range extends to θ<20% at CV=5%; at CV≥25%, the conservative case has no model with guaranteed high-confidence profitability.Source: Figure 2 · p. 11, §5.1 · p. 11
  • API profitability is flat as alert volume changes, while self-hosted profitability rises in steps as additional GPU replicas are provisioned and their fixed costs are amortized.At the 1,000-alert baseline, SH-High becomes competitive only under full autonomy; at high volume, its larger GPU count can leave it less profitable than SH-Mid.Source: Figure 3 · p. 11, §5.1 · pp. 11–12
  • API-High’s added costs are negligible and mainly reflect rework, leaving total cost below baseline, while SH-High’s GPU infrastructure cost absorbs efficiency savings and pushes total cost above baseline.Figure 4 compares LLM cost with the no-LLM baseline and shows 10–90 percentile whiskers.Source: Figure 4 · p. 11, §5.1 · p. 12

5.2. Downside risk

  • The worst-decile safe region widens from conservative rework to full automation and shrinks as θ or CV grows; in the conservative case, losses exceed half the committed capex across almost the whole grid.Under full automation, the worst decile remains profitable up to high θ over most of the grid, making the tolerable loss a capex boundary for risk-averse SOCs.Source: Figure 5 · p. 12, §5.2 · p. 12

5.3. Parameter sensitivity

  • With technology costs excluded, profitability requires accuracy above a threshold that rises with analyst effort; speed dominates high- and mid-accuracy models, while rework becomes more important at low accuracy.Alert volume does not affect API returns and moves only self-hosted models, most visibly SH-High.Source: Figure 6 · p. 12, §5.3 · pp. 12–13, Figure 7 · p. 13
  • At Cmiss=$10.22 million, API-High yields 1/λmax≈7 years at mean values under full automation, β=0, θ=10%, and CV=5%.The paper defines λmax as the breach-rate adoption can sustain and recommends full automation only where the breach rate stays below that bound.Source: §5.3 · p. 13

6. Discussion

The discussion identifies automation, model accuracy, and operating conditions as central to SOC adoption economics. Its estimates remain bounded by benchmark and aggregate-input limitations, motivating validation and refinement.

6. Discussion

  • Returns are more sensitive to automation than to rework penalties, and autonomy supports higher capital expenditure only when accuracy is sufficient for trust.The autonomy regime is the only one reported to remain profitable in worst-decile outcomes at high capital expenditure.Source: §6 · p. 13
  • Most configurations are unprofitable; only high-accuracy API archetypes achieve high-confidence profitability, and only with low capital expenditure and uncertainty.The autonomy regime can support higher capital expenditure, but only when model accuracy is sufficient for outputs to be trusted without full human re-verification.Source: §6 · p. 13
  • Model price has only a marginal effect, while API token costs are a small share of total cost and price is nearly uncorrelated with accuracy (Spearman ρ=0.04).Source: §6 · p. 13
  • Self-hosting rarely helps at realistic alert volumes because fixed GPU costs outweigh efficiency savings; API costs and savings scale together as alert volume changes.Privacy requirements or models that reduce deployment requirements without sacrificing performance could shift the balance toward self-hosting.Source: §6 · p. 13
  • Decision-makers can use organizational operating data to examine profitability across capital-expenditure ratios and uncertainty, while VaR10% represents potential loss in the worst decile.Adoption is defensible when capex supported at the chosen confidence level covers estimated one-off costs and worst-decile loss remains tolerable.Source: §6 · p. 14

6.1. Limitations and future works

  • The analysis uses public benchmarks and author experience; validation with live-SOC data is needed to test fit and how consistently required metrics are tracked.Source: §6.1 · p. 14
  • Aggregate values limit task-, role-, and severity-specific modeling, while the current model subsumes dedicated AI roles and tasks into existing ones.The authors identify these distinctions as possible refinements.Source: §6.1 · p. 14
  • Self-hosted costs are a conservative upper bound because the estimate omits serving efficiencies such as batching and KV-cache reuse.Those efficiencies may be large enough to change the adoption decision; real deployment studies could improve comparison with APIs.Source: §6.1 · p. 14

7. Conclusion

The conclusion presents an economic framework for assessing LLM adoption, with SOCs as its first application. Most simulated configurations are not reliably profitable, but organizations can use the released code to evaluate their own costs and risks.

7. Conclusion

  • The framework treats adoption as an investment under deep uncertainty and provides ROI distributions and potential loss estimates for SOCs.Source: §7 · p. 14
  • Most configurations are not reliably profitable; profitability is limited to high-accuracy models with low capital costs and uncertainty.Returns depend less on model price than on how far outputs can be acted on without human review; self-hosting rarely helps at typical SOC alert volumes.Source: §7 · p. 14
  • Organizations can use the released code to enter their own figures and assess profitability, affordable capital cost, and risk of loss.Source: §7 · p. 14

Funding

  • The authors state that the study received no specific grant from funding agencies.Source: Funding · p. 14

Data availability

  • The simulation code and benchmark-derived parameters are stated to become publicly available upon publication.Source: Data availability · p. 14

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

  • The authors report using Claude for language editing, structural revision, clarity improvements, and coding support, and say they reviewed and edited the content and code.Source: Declaration of generative AI and… · p. 15

A.1. Return-on-investment threshold

The proposition gives a bound on the capital expenditure ratio compatible with a target return, obtained by rewriting ROI as a function of that ratio.

A.1. Return-on-investment threshold

  • The proposition bounds the capital expenditure ratio θ compatible with a target return y.Source: §A.1 · p. 18
  • The bound follows by rewriting ROI as a function of θ.Source: §A.1 · p. 18
  • The displayed condition expresses ROI greater than y as a corresponding upper bound on θ.Source: §A.1 · p. 18

A.2. Distribution parametrization

The appendix explains how the model parameterizes uncertainty: for Beta and Gamma distributions, native parameters are recovered from the mean and coefficient of variation.

A.2. Distribution parametrization

  • For uncertain parameters, the mean μ and coefficient of variation CV are used to recover native parameters for Beta and Gamma distributions.Source: §A.2 · p. 18
  • For a Beta-distributed X, the passage identifies α and β as positive parameters and μ as a mean in (0, 1).It also gives expressions for the mean and variance in terms of α and β.Source: §A.2 · p. 18
  • For a Gamma-distributed parameter, κ and ς are identified as positive shape and scale parameters, respectively.The passage says these parameters can be expressed using the mean.Source: §A.2 · p. 18

B. Cybersecurity Benchmark-Derived Model Parameters

The section compiles model deployment, pricing, size, and cybersecurity benchmark accuracy data from six benchmarks. Table 9 covers 71 distinct models across 73 rows, including two models evaluated in both API and self-hosted deployments.

B. Cybersecurity Benchmark-Derived Model Parameters

Loading 2610.33352v1…