TingIS: Real-time Risk Event Discovery from Noisy Customer Incidents at Enterprise Scale

Jun Wang; Ziyin Zhang; Rui Wang; Hang Yu; Peng Di; Rui Wang

Source-linked AI summary

TingIS: Real-time Risk Event Discovery from Noisy Customer Incidents at Enterprise Scale

Jun Wang, Ziyin Zhang, Rui Wang, Hang Yu, Peng Di, Rui Wang

arXiv:2604.21889v3Apr 23, 2026cs.CLcs.AIcs.LG

TL;DR

Customer incidents are noisy, high-volume, and semantically diverse, making actionable risk discovery difficult despite their value for detecting missed risks. TingIS combines multi-stage event linking with efficient indexing, LLM reasoning, routing, and denoising; in production, it processes over 300,000 incidents daily, identifies 95% of high-priority incidents, and achieves 3.5-minute P90 alert latency.

Problem
Customer incidents provide signals for risks missed by monitoring, but noise, high throughput, and semantic diversity make actionable intelligence extraction challenging.
Method
TingIS combines multi-stage event linking using LSH, historical event association, and LLM reasoning with semantic distillation, cascaded routing, event state management, and multi-dimensional denoising.
Results
95% discovery rate for high-priority incidents and a P90 alert latency of 3.5 minutes were achieved while processing over 300,000 incidents daily and 2,000 per minute.
Takeaways & Limitations
Benchmark evaluations show TingIS significantly outperforms baselines in routing accuracy, clustering quality, and signal-to-noise ratio.
Takeaways & Limitations
High-priority incidents are defined as incidents requiring immediate attention from SRE teams.

Abstract

from arXiv · show

Real-time detection and mitigation of technical anomalies are critical for large-scale cloud-native services, where even minutes of downtime can result in massive financial losses and diminished user trust. While customer incidents serve as a vital signal for discovering risks missed by monitoring, extracting actionable intelligence from this data remains challenging due to extreme noise, high throughput, and semantic complexity of diverse business lines. In this paper, we present TingIS, an end-to-end system designed for enterprise-grade incident discovery. At the core of TingIS is a multi-stage event linking engine that synergizes efficient indexing techniques with Large Language Models (LLMs) to make informed decisions on event merging, enabling the stable extraction of actionable incidents from just a handful of diverse user descriptions. This engine is complemented by a cascaded routing mechanism for precise business attribution and a multi-dimensional noise reduction pipeline that integrates domain knowledge, statistical patterns, and behavioral filtering. Deployed in a production environment handling a peak throughput of over 2,000 messages per minute and 300,000 messages per day, TingIS achieves a P90 alert latency of 3.5 minutes and a 95\% discovery rate for high-priority incidents. Benchmarks constructed from real-world data demonstrate that TingIS significantly outperforms baseline methods in routing accuracy, clustering quality, and Signal-to-Noise Ratio.

1 Introduction

TingIS addresses the difficulty of extracting actionable risk events from noisy customer incidents at enterprise scale. It combines multi-stage event linking with supporting routing and denoising modules, achieving strong production performance and benchmark results.

Customer incidents complement metrics, logs, and traces by exposing monitoring blind spots and user-perceived impact.
2,000 messages per minute makes extracting systemic failures from as few as 3 noisy incidents a severe Signal-to-Noise Ratio challenge.
Five modules span data observation, semantic processing, and long-term memory in the TingIS architecture.
TingIS uses multi-stage event linking that combines efficient indexing, historical event association, and LLM reasoning to synthesize fragmented incidents into structured risk events.
95% of high-priority risk incidents were identified with a P90 alert latency of 3.5 minutes during one month of online deployment.
Benchmarks from real-world production data show significant improvements over system-level and specialized module-level baselines in routing accuracy, clustering quality, and signal-to-noise ratio.

2 System Architecture

TingIS maps noisy customer incidents to existing risk events, newly initialized events, or suppression through a resource-aware, multi-stage architecture. Its modules combine semantic normalization, business routing, event linking, state management, and layered denoising.

A customer incident is noisy, colloquial, subjective feedback, while a risk event is a persistent, stateful representation identified by business domain and topic.
TingIS maps each incoming incident to an existing risk event, a newly initialized event, or the null set.
Rule-based filtering, LSH, similarity thresholds, and persistent event states reduce the cost of applying LLM reasoning to streaming data.
Semantic Distillation (M1): Semantic distillation uses an LLM to convert colloquial incident text into compact subject-plus-problem representations while excluding emotion, filler, PII, and irrelevant details.
Cascaded Routing (M2): Cascaded routing first uses entity-priority keyword matching, then vector retrieval and reranking for incidents without keyword hits.
Event Linking: Event linking progressively combines domain partitioning, LSH clustering, representative LLM checks, historical retrieval, time decay, and final merge-versus-create adjudication.
State Management: Layered state management separates mutable alerting state, immutable audit evidence, and snapshot timelines for dynamic baselines.
Multi-dimensional Denoising: Three denoising layers suppress known false positives, filter deviations from dynamic baselines, and pause alerts while allowing explosive surges to penetrate silencing windows.

3 Experiments

TingIS is evaluated through complementary online production validation and reproducible offline benchmarks, covering system behavior, event-linking quality, routing, and efficiency. Across these evaluations, it reduces noise, improves event identity quality, and maintains high-throughput operation with low alert latency.

Evaluation framework: A one-month production validation measures recall and latency for high-priority risk events confirmed by SRE teams.High-priority events require immediate attention from SRE teams.
Evaluation datasets: Production snapshots yield an alarm replay set, SRE-annotated benchmark events, an event identity set, and a cold-start routing set.The benchmark events are annotated from 50 thousand incidents, while the routing set uses a 20%/80% split.
System-level performance: 95% high-priority incident discovery and a P90 alert latency of 3.5 minutes were achieved during the one-month production run.These figures summarize online system performance in production.
System-level performance: 94.3% noise reduction lowered alerts from 512 to 29 without reducing detection rate, while the event-to-alert ratio reached 1.23.The ratio was closest to the ideal value of 1.0 and was associated with alert silencing and penetration strategies.
Event linking quality: TingIS achieved B3-F1 of 0.826 with 5.8% fragmentation and 21.5% mismerge, reducing mismerge from DBSCAN’s 64.3%.The reported balance favors lower fragmentation and lower mismerge because unrelated failures can corrupt root-cause analysis, whereas fragmentation creates duplicate workflows.
Ablation analysis: Removing business partitioning caused a 15.6% B3-F1 drop, while LLM summary and two-stage LLM application improved B3-F1 and reduced mismerge.The LLM summary contributed 7.0% in B3-F1 and reduced mismerge by 66.5%; the two-stage M3 application contributed approximately 5% B3-F1 and 60% mismerge reduction.
Intelligent distribution: The cascaded routing architecture achieved Acc@1 of 0.669 versus 0.460 for parallel fusion, while TingIS Full maintained 88.1% coverage.Removing the reranker raised raw Acc@1 to 0.705 but produced 100% coverage; with a degraded database, the reranker improved accuracy while controlling coverage.
Efficiency: Parallelized LLM calls, vector searches, and batched database operations support stable throughput of 2,000 queries per minute.LLM-based reasoning accounts for 8.53 seconds, or 69.7% of total latency, while database and retrieval components are faster.

4 Conclusion

TingIS combines hybrid intelligence, routing, state management, and denoising to extract actionable risk events from noisy enterprise incidents. Production deployment and real-world benchmarks show strong operational scale and performance.

TingIS combines LLMs with efficient indexing and historical event association to address noisy, heterogeneous customer-incident data.
Cascaded routing, event state management, and multi-dimensional denoising support stable extraction of actionable risk events from colloquial incidents.
TingIS processes over 300,000 incidents daily and 2,000 per minute, with a 95% discovery rate and 3.5-minute P90 alert latency.
Benchmarks show improved signal-to-noise ratio and reduced false alerts, event mismerge, and event fragmentation.
The paper provides production case studies, a biz_code taxonomy example, and guidance on data skew, routing, and responsible LLM integration.
Compared with prior approaches, TingIS explicitly models persistent event identity and combines LLM-based adjudication with time-decayed similarity.
Its denoising funnel integrates semantic matching, dynamic baselines, and behavioral constraints, while waterfall routing isolates business domains across head and long-tail cases.

C Detailed Illustration of System Modules

Appendix C provides detailed illustrations of TingIS’s multi-stage event linking engine and multi-dimensional denoising pipeline.

Figure 5 illustrates the multi-stage event linking engine, M3.
Figure 6 illustrates the multi-dimensional denoising module, M5.

D B3 Evaluation Metrics

TingIS evaluates event-linking quality with B-Cubed metrics, which assess incident relationships without requiring meaningful cluster identifiers. Precision reflects cluster purity, while recall reflects convergence or completeness.

B-Cubed precision, recall, and F1 evaluate clustering when cluster IDs are arbitrary rather than predefined class labels.
For incident i, L(i) contains incidents sharing its ground-truth event label, while C(i) contains incidents assigned to its system-generated cluster.
B-Cubed precision and recall are computed per incident and averaged across all evaluation items.
B3-F1 is the harmonic mean of aggregate B3 precision and B3 recall.
High B3 precision indicates low mismerge through cluster purity, whereas high B3 recall indicates low fragmentation through event convergence.

E Resource and Cost Efficiency Analysis

TingIS’s computational footprint is quantified using one month of live production monitoring data, with a daily median input of 250,000 customer incidents.

One month of production monitoring data shows a daily median input of 250k customer incidents, with metrics reflecting live operational behavior.

E.1 Input Volume Reduction via Lightweight Preprocessing

Rule-based filtering removes low-signal customer incidents while preserving more than 99% recall for high-priority incidents, reducing downstream processing to approximately 50,000 incidents per day at near-zero computational cost.

E.1 Input Volume Reduction via Lightweight Preprocessing: More than 99% recall for high-priority incidents is preserved after rule-based filtering.The filter removes low-signal incidents while retaining the critical signal.
E.1 Input Volume Reduction via Lightweight Preprocessing: Approximately 50,000 customer incidents per day remain for downstream processing at near-zero computational cost.This establishes TingIS’s foundational efficiency layer.

E.2 LLM Token Consumption: Quantitative Breakdown

TingIS controls LLM usage through filtering, concise prompts, cluster-level invocation, and threshold gating, while producing approximately 29 validated high-confidence alerts per day.

E.2 LLM Token Consumption: Quantitative Breakdown: No LLM is invoked on filtered incidents, and algorithmic gating minimizes Kimi-K2 calls.These controls reduce token consumption across LLM-dependent stages.
E.2 LLM Token Consumption: Quantitative Breakdown: Approximately 50,000 filtered incidents processed by M1 generate 5.0M tokens per day through concise summaries.Prompt engineering limits summaries to approximately 100 tokens total.
E.2 LLM Token Consumption: Quantitative Breakdown: Kimi-K2 is invoked once per LSH-generated cluster, covering approximately 30,000 clusters per day.The stated batching structure is 250 batches × 10 business codes × 12 clusters per business code.
E.2 LLM Token Consumption: Quantitative Breakdown: More than 70% of historical matches bypass LLM adjudication under the s∗ > 0.95 threshold, containing total consumption at 3.0M tokens per day.The threshold is applied during steady-state operation.
E.2 LLM Token Consumption: Quantitative Breakdown: Approximately 29 high-confidence alerts per day are generated after M5 denoising and validated by SRE teams.The end-to-end cost metric covers processing from raw user-voice ingestion to human-actionable alerts.

E.4 Quantified Impact of Design Choices

TingIS’s scaling strategy combines lightweight pre-filtering, fixed-size batching, and threshold-gated adjudication to reduce LLM load and maintain tractable enterprise operation.

E.4 Quantified Impact of Design Choices: Rule-based pre-filtering eliminates approximately 200,000 low-signal incidents per day and reduces downstream LLM load by 80%.This is the first of three architecture decisions identified as enabling sustainable scaling.
E.4 Quantified Impact of Design Choices: Fixed-size batching stabilizes Kimi-K2 invocation at approximately 30,000 clusters per day, preventing RPM throttling and OOM risks.It replaces volatile time-window batching with a more stable invocation pattern.
E.4 Quantified Impact of Design Choices: More than 70% of historical matches bypass LLM calls under s∗ > 0.95, containing daily Kimi-K2 consumption at 3.0M tokens.Monitoring shows consumption declining from cold-start peaks to a stable baseline.
E.4 Quantified Impact of Design Choices: Sustained operation at 8.0M tokens per day is described as tractable for enterprise deployment despite API costs depending on commercial agreements.The system uses architectural controls to make LLMs a targeted component rather than a cost liability.
E.4 Quantified Impact of Design Choices: The scaling lessons were derived through iterative deployment and validation in a high-stakes production environment.The authors frame these lessons as empirically grounded guidance for industrial NLP system design.

F.1 Data Preprocessing: Rule Filtering Requires Recall-Aware Validation

TingIS addresses skewed, noisy, and semantically ambiguous incident streams with validated filtering, fixed-size batching, feedback-driven routing, and structured LLM summaries.

F.1 Data Preprocessing: Rule Filtering Requires Recall-Aware Validation: More than 50% of incidents contained low-information content, while 73% were concentrated across eight high-frequency business domains.Six configurable filtering rules were validated against historical fault logs to avoid degrading high-priority recall.
F.1 Data Preprocessing: Rule Filtering Requires Recall-Aware Validation: Traffic varied by more than 100× peak-to-trough, making time-window batching vulnerable to OOM risks, API throttling, and underutilization.Fixed-size batches of 200 provide constant computational load and natural backpressure.
F.1 Data Preprocessing: Rule Filtering Requires Recall-Aware Validation: More than 98% of valid incidents exhibited clear domain attribution, supporting retention of the top three business domains and filtering beyond that threshold.Keywords and incrementally updated vector indices balance automation with feedback from verified anomalies.
F.1 Data Preprocessing: Rule Filtering Requires Recall-Aware Validation: Pure embedding clustering confused incidents sharing problem tokens but differing in subjects, such as campaign rewards and NFC payment failures.LLM-generated subject-plus-problem summaries disentangled these semantics.
F.1 Data Preprocessing: Rule Filtering Requires Recall-Aware Validation: The authors identify validation over heuristics, continuous knowledge curation, and transparent failure analysis as principles for industrial NLP design.These principles emphasize historical validation, feedback-driven updates, and documenting limitations such as subject-blind clustering.