PaperlayerView exact revision on arXiv

Source-linked AI summary

Incident Memory: Training-Free Operational Memory through Sequential Pattern Mining and Velocity-Stratified Retrieval

Adarsh Agrawal, Rahul Suresh Babu

arXiv:2609.01616v1cs.IR

TL;DR

Incident response needs memory that preserves the order, freshness, and provenance of operational knowledge. Incident Memory addresses this with velocity-stratified retrieval, fingerprint-conditioned PrefixSpan mining, and provenance-aware metric definitions. Across controlled and UCI ITSM evaluations, it reports high ordered precision, broad held-out coverage, freshness protection, and stronger ordered playbooks than the direct Claude Haiku baseline.

  • Problem

    Operational knowledge is often stored without preserving the order of successful actions, the freshness of facts, or the provenance of metric definitions.

  • Method

    Incident Memory is a training-free system combining velocity-specific fact decay, fingerprint-conditioned PrefixSpan playbook mining, and provenance-linked SQL conflict detection.

  • Results

    99.2% ordered playbook precision was achieved on controlled traces, while the UCI ITSM evaluation covered 84.3% of held-out incidents across 39 mined playbooks.

  • Takeaways & Limitations

    When repeated incidents yield low-entropy action traces, exact sequential memory provides stronger playbooks than open-ended generation.

  • Takeaways & Limitations

    The system is strongest for recurring incidents with consistent action vocabularies; rare fingerprints, changing procedures, and causal MTTR claims remain outside the demonstrated scope.

Abstract

from arXiv · show

Incident response is a memory problem: teams accumulate tickets, traces, postmortems, and wiki pages, but the knowledge needed for the next incident is rarely stored with its order, freshness, and provenance intact. We present Incident Memory, a deterministic system that accumulates operational knowledge without model training. It combines (i) velocity-stratified retrieval, which ages structural, behavioral, contextual, and ephemeral facts at different rates; (ii) fingerprint-conditioned PrefixSpan mining, which extracts ordered playbooks from successful investigations; and (iii) provenance-aware metric definitions, which detect conflicting definitions through executable checks. On the UCI ITSM event log, containing 141,712 events across 24,918 incidents, Incident Memory extracts 23,110 ordered traces, mines 39 playbooks, and covers 84.3% of 6,934 held-out incidents. On controlled benchmarks with known ground truth, it achieves 99.2% ordered playbook precision (controlled), an architectural staleness guarantee versus 36% stale returns for a flat baseline, and conflict-detection F1 of 0.876. A direct Claude Haiku baseline on 19 fingerprint groups reaches 0.661 ordered precision, compared with 0.985 for PrefixSpan. The central result is not that language models are weak at incident response; it is that repeated incident histories are low-entropy once conditioned on fingerprint and previous action. In that regime, exact memory is a stronger primitive than open-ended generation.

1 Introduction

Incident response requires memory that preserves action order, freshness, and metric-definition provenance, not merely textual access. Incident Memory uses deterministic sequential mining and layered memory because repeated incident traces become low-entropy when conditioned on fingerprint and previous action.

  • Motivation: The paper identifies missing temporal validity, action order, and provenance as limitations of existing wikis, retrieval systems, and language-mediated incident workflows.These systems may preserve or retrieve text without retaining the operational structure needed for the next incident.
  • Motivation: Incident histories have H = 1.92 bits, or approximately 3.8 effective continuations, after conditioning on fingerprint and previous action.The previous action contributes 1.10 bits of information, compared with 0.76 bits from fingerprint identity.
  • Approach: Incident Memory combines velocity-specific fact decay, fingerprint-conditioned PrefixSpan playbook mining, and provenance-linked SQL conflict detection without updating neural weights.The system updates support counts, timestamps, and provenance records while using a frozen embedding backbone.
  • Evaluation: The evaluation combines controlled traces, real UCI ITSM logs, and a direct Claude Haiku baseline to test precision, coverage, freshness, conflicts, and order preservation.The contribution is an evaluation design spanning multiple operational-memory properties rather than a single benchmark.
  • Evaluation: 99.2% ordered playbook precision was achieved on controlled traces, while 23,110 real traces supported 84.3% held-out coverage across 39 mined playbooks.The evaluation separates controlled oracle-label benchmarks from transfer to uncontrolled UCI ITSM event records.

2 Related Work

Related work spans AIOps agents, freshness-aware retrieval, sequential process mining, and structured agent memory. Incident Memory is positioned as complementary: it organizes recorded operational traces into empirically supported, fingerprint-specific playbooks.

  • AIOps and incident agents: AIOps research frames incident handling as detection, diagnosis, mitigation, and learning, with language models increasingly used for triage and root cause analysis.Production incidents also depend on deployment context, capacity changes, routing, hotfixes, and operational policy.
  • AIOps and incident agents: Incident Memory complements agents by supplying grounded context while mining playbooks from prior traces with empirical support.The playbook is not generated anew at query time.
  • Retrieval and freshness: Retrieval research treats grounding, robustness, and freshness as persistent challenges, while Incident Memory adds velocity-dependent aging for operational facts.The related-work discussion contrasts this setting with recency sorting or a single decay parameter.
  • Sequential mining and process structure: PrefixSpan and process-mining methods recover ordered structure from traces, but Incident Memory conditions mining on incident fingerprints because failure modes require different paths.This distinguishes fingerprint-specific playbooks from one workflow model describing all possible execution paths.
  • Memory architectures for agents: Structured memories for long-horizon agents address context-window limits, whereas Incident Memory organizes domain-specific operational knowledge after relevant traces have been recorded.The paper presents these objectives as complementary rather than competing memory designs.

3 Incident Memory

Incident Memory is a three-layer operational-memory architecture: it retrieves fresh facts, mines ordered fingerprint-specific procedures, and checks metric semantics through provenance. Its pipeline preserves current context while replaying supported fast paths and warning about conflicting definitions or detours.

  • MetricMind: MetricMind assigns structural, behavioral, contextual, and ephemeral knowledge half-lives of 5 years, 2 years, 90 days, and 7 days.Velocity classes are assigned by ingestion metadata rules rather than a trained classifier.
  • MetricMind: Temporal dominance returns the freshest nonexpired unit per velocity class within each topic group, preventing superseded facts from dominating retrieval.Older facts remain auditable while current facts control retrieval.
  • Incident Archaeologist: Incident Archaeologist represents traces as fingerprinted ordered action sequences, groups resolved traces by fingerprint, and mines high-confidence subsequences with PrefixSpan.It stores the longest sequence that passes support and confidence filters; lookup is hash-based after indexing.
  • Incident Archaeologist: The example playbook orders four actions—check_upstream_deps, validate_schema, rollback_deploy, and verify_metrics—while the anti-skill flags restart_service in 73% of slow traces.The anti-skill is associated with an average of 12 minutes wasted in that example.
  • Living Glossary: The Living Glossary stores metric definitions with executable-query provenance, ranks conflicts by provenance, citation count, and recency, and archives superseded definitions.At query time, the system combines fresh context, a fingerprint-specific playbook and anti-skills, and unresolved metric conflicts in one response.
  • Architecture: The architecture combines temporal freshness, ordered procedures, and provenance records rather than treating incident knowledge as static text.Its three layers accumulate support counts, timestamps, and provenance records instead of gradient updates.

4 Why Exact Sequential Memory Fits

Incident histories are structured enough for exact sequential mining when conditioning on fingerprint and prior action sharply reduces next-action uncertainty. The resulting system can replay high-support ordered paths instead of generating procedures anew.

  • Why Exact Sequential Memory Fits: 1.10 bits of additional information comes from the previous action, exceeding the fingerprint's 0.76 bits.The mutual-information decomposition gives I(A; F)+I(A_t; A_t−1 | F) = 0.76+1.10 = 1.86.
  • Why Exact Sequential Memory Fits: Sequential context supplies the larger share of the information used to reduce uncertainty.The traces include 9 unique actions, 107 fingerprint groups, and 23,110 traces.
  • Why Exact Sequential Memory Fits: 49% lower next-action entropy leaves fewer than four effective continuations per context.On the UCI ITSM traces, conditioning yields H(A_t | F, A_t−1) = 1.25 bits, a 49.0% reduction from H(A) = 2.44 bits.
  • Why Exact Sequential Memory Fits: For supported fingerprints, replaying the high-support ordered path and flagging known detours avoids inventing a procedure for every incident.This explains why unordered baselines can recover plausible action sets yet fail on ordered precision.

5 Experimental Design

The evaluation combines controlled traces with known ground truth and real UCI ITSM logs to assess both measurable performance and external validity. The design also compares Incident Memory against multiple retrieval, runbook, frequency, and generation baselines.

  • Controlled and Real-Data Evaluation: 500 controlled investigation traces span 60 fingerprint types, 20 investigation actions, and 8 root-cause templates.The companion knowledge base contains 1,000 factual units embedded with Amazon Titan Text Embeddings V2 using 1,024 dimensions.
  • Controlled and Real-Data Evaluation: 141,712 events across 24,918 incidents form the real-data UCI ITSM validation set.The adapter maps state changes, reassignments, knowledge-base use, and resolution events into a finite investigation vocabulary.
  • Controlled and Real-Data Evaluation: 23,110 traces remain after removing traces shorter than two actions, with 16,176 training traces and 6,934 held-out traces.The retained traces have mean length 5.32.
  • Baselines and Settings: The baselines include no guidance, static runbooks, frequency ordering, Claude Haiku generation, decay variants, and a flat wiki-style freshness baseline.Unless stated otherwise, PrefixSpan uses minimum support σ = 3 and confidence threshold γ = 0.6.

6 Results

Results show that Incident Memory preserves ordered operational procedures, transfers to real incident histories, and improves retrieval freshness and conflict detection through complementary components.

  • 6.1 Ordered Playbooks Outperform Generated and Static Playbooks: 99.2% ordered precision: PrefixSpan reaches 1.000 ordered precision in the controlled comparison, while the unordered frequency baseline reaches 0.453.The same comparison reports 0.986 unordered precision for the frequency baseline, exposing the central order gap.
  • 6.1 Ordered Playbooks Outperform Generated and Static Playbooks: 0.985 ordered precision: PrefixSpan exceeds Claude Haiku's 0.661 on 19 fingerprint groups in the direct comparison.The paired Wilcoxon test reports p = 0.000196; PrefixSpan used all training traces per group, while Claude Haiku received 10 exemplars.
  • 6.2 Real ITSM Logs Contain Reusable Ordered Structure: 84.3% held-out coverage: the same PrefixSpan pipeline mines 39 playbooks from UCI ITSM histories and covers held-out traces as ordered subsequences.The mined playbooks average 3.8 steps, range from 3 to 7, and are unique to individual fingerprint groups.
  • 6.2 Real ITSM Logs Contain Reusable Ordered Structure: 15.7% of held-out incidents remain uncovered because rare fingerprints lack σ = 3 support or multi-root-cause traces combine steps from multiple playbooks.The system withholds guidance in these cases rather than generating unsupported recommendations, defining a ceiling for deterministic sequential replay.
  • 6.3 Freshness and Provenance: 0.532 average P@5: velocity-stratified retrieval exceeds 0.457 for no decay and 0.394 for uniform decay.Temporal dominance filtering returns 0% stale facts by construction, whereas the flat wiki-style baseline returns stale facts in 36% of tested cases; the Living Glossary reaches F1 0.876 on injected conflicts.
  • 6.4 Ablations: Each component contributes non-redundant information, while the system remains stable across 2 to 4× parameter variation.On UCI, varying σ from 2 to 5 yields coverage between 78% and 87%; the Shapley-style analysis evaluates component contributions across ablations.

7 Discussion

Incident Memory is designed for recurring incidents with available traces, using exact sequential memory and supporting LLM assistants rather than replacing them. It bootstraps quickly, while its LLM comparison remains limited to a fixed prompt protocol.

  • Incident Memory assumes recurring failure modes and available prior traces, narrowing its scope relative to a general RCA agent.It does not infer novel causal stories from arbitrary telemetry.
  • A deterministic sequence miner is therefore treated as the appropriate inductive bias for low-entropy incident traces.The discussion frames exact replay as preferable to unconstrained generation in this regime.
  • Incident Memory can serve LLM agents by retrieving fresh context, fingerprint-specific playbooks, anti-skills, and provenance for metric definitions.The memory layer is intended to reduce the burden on generation rather than compete with it.
  • The LLM comparison uses a single prompt protocol, so stronger tool-using agents with longer contexts may close part of the gap.The paper characterizes the comparison as a lower bound on the advantage.
  • At 10 incidents per week, 50% playbook coverage is reached in the first week.This rapid bootstrapping is contrasted with months-long wiki curation.

8 Conclusion

Incident Memory separates freshness, ordered procedure, and definition provenance to turn historical incident traces into auditable recommendations. Across controlled, real-log, and direct LLM evaluations, exact sequential memory performs better when repeated incidents yield low-entropy action traces.

  • Separating freshness, ordered procedure, and definition provenance converts historical incident traces into auditable recommendations.
  • Exact sequential memory gives stronger playbooks than open-ended generation when repeated incidents leave low-entropy action traces.The evidence spans controlled experiments, UCI ITSM logs, and a direct LLM baseline.

Limitations

The system is strongest for recurring incidents with consistent action vocabularies, while rare, changing, or causally complex settings remain constrained. Production deployment still requires evaluation beyond observational and controlled benchmarks.

  • Incident Memory is strongest when incidents recur and are recorded with consistent action vocabularies.Rare fingerprints, mixed root causes, and changing procedures remain difficult.
  • The UCI ITSM evaluation validates ordered sequence coverage, not causal reduction in resolution time.Its knowledge-use field is observational and confounded by incident difficulty.
  • The freshness layer depends on correct velocity classification, and current drift detection is limited to decreasing support counts in new traces.Misclassifying a short-lived workaround as structural could preserve it too long.
  • The LLM comparison evaluates playbook generation under a fixed prompt protocol rather than every possible agent design.More capable tool-using agents may close part of the gap.

Ethical Considerations

The evaluation uses public, anonymized, and controlled artifacts, separating real-data validation from synthetic traces used for oracle labels. Ethical deployment requires access controls, explicit documentation of assumptions and bias, and human ownership of incidents.

  • Artifacts and evaluation: The study uses public or controlled artifacts, including an anonymized UCI ITSM event log and synthetic controlled traces.Controlled traces are reserved for oracle labels, while real-data validation is reported separately.
  • Artifacts and evaluation: Controlled benchmarks provide oracle labels for action order, stale facts, metric conflicts, and component ablations.The UCI ITSM log supports aggregate sequence mining rather than reidentification.
  • Evaluation definitions: Exact coverage requires a mined playbook to appear as an ordered subsequence of a held-out trace.Partial coverage records substantial ordered overlap when traces diverge.
  • Coverage and cold start: High-frequency fingerprint groups achieve near-complete coverage, while rare fingerprints account for most of the 15.7% uncovered incidents.The system withholds guidance rather than generating unsupported playbooks for the long tail.
  • Coverage and cold start: Low-frequency fingerprints should receive fallback guidance until sufficient trace support accumulates.
  • Freshness and retrieval: Velocity classes assign different decay profiles because structural facts should persist longer than ephemeral operational context.A single half-life cannot serve both knowledge types.
  • Freshness and retrieval: Velocity-stratified retrieval maintains precision by filtering validity and applying temporal dominance, unlike static document memory.Structural facts retain relevance while ephemeral facts are downweighted as they age.
  • Coverage and cold start: 10 traces per fingerprint group yield approximately 50% coverage, while coverage plateaus near 75% as trace volume increases.The controlled benchmark measures accumulation of useful memory as trace volume grows; the cold-start result depends on support thresholds and fingerprint distribution.

E Sensitivity, Ablation, and Conflict Diagnostics

The system remains stable across 2–4× parameter variation, while ablations identify fingerprint matching and sequential mining as its largest contributors. Conflict detection uses executable SQL comparison, and retrieval applies velocity-specific temporal filtering rather than relying on embedding geometry alone.

  • Sensitivity and ablation: 2–4× parameter variation leaves the system stable, but its components are not interchangeable.Table 13 covers sensitivity; Table 14 reports Shapley-value contributions across 24 ablation permutations.
  • Conflict diagnostics: A 1.0% default conflict threshold favors high recall because missed SQL-semantic conflicts are considered more damaging than reviewable false positives.The glossary can be tuned toward either precision or recall depending on operational needs.
  • Sensitivity and ablation: Fingerprint matching and sequential mining provide the largest marginal contributions, followed by provenance SQL.Velocity classification contributes over time rather than as an immediate precision jump.
  • Retrieval and embedding checks: Embedding geometry is only a qualitative retrieval check: Figure 11 reports four clusters corresponding to structural, behavioral, contextual, and ephemeral velocity classes.The paper states that its primary claims rest on sequential mining, temporal filtering, and provenance comparison.
  • Retrieval and embedding checks: Velocity-specific decay returns only the freshest unit per velocity class, preventing superseded facts from appearing in retrieval results.This temporal dominance step produces the architectural staleness guarantee.
  • Conflict diagnostics: Executable SQL comparison detects metric conflicts that text-only matching can miss, including definitions that differ over cancelled orders.The comparison distinguishes nearly identical prose that computes different quantities.
Loading 2609.01616v1…