Source-linked AI summary
Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata
Meiwei Zhang, Eduardo Miranda, Bruce Baynes, Suvigya Jain, Wanlong Chen, Tao He, Sergey Borodavkin
TL;DR
Capability benchmarks provide limited evidence about how managed LLM services behave operationally after deployment. OpEmbed learns compact fingerprints from structured, privacy-preserving support metadata at model–time-window granularity, and the evaluation finds useful structure for forecasting, early-history prediction, and cross-model fault transfer. The paper also reports production lessons about privacy, workload composition, and downstream usefulness.
Problem
Capability benchmarks quantify model capability but do not address operational behavior such as support load, recurring failures, resolution speed, or version shifts after deployment.
Method
OpEmbed aggregates structured support metadata into model–time-window signatures and learns low-dimensional embeddings with temporal contrastive learning, cross-view reconstruction, and generational-ordinality constraints.
Results
OpEmbed organizes models into interpretable operational neighborhoods, improves forecasting over raw-signature and heuristic baselines, remains useful with limited early history, and transfers fault knowledge across models and vendors.
Takeaways & Limitations
Structured support metadata alone can provide a practical operational evaluation axis for model onboarding, support readiness assessment, and reliability monitoring under privacy constraints.
Takeaways & Limitations
The data come from a single support ecosystem, and different ticket schemas may require adapting the eight-channel signature rather than reusing it unchanged.
Abstract
from arXiv · showhide
Managed LLM services are now part of real production systems, but model selection and service planning still rely heavily on capability benchmarks that reveal little about operational behavior after deployment. We present Operational Embedding (OpEmbed), a framework for learning compact operational fingerprints of LLM cloud services from structured, privacy-preserving support-case metadata, without using case text. OpEmbed aggregates model--time windows into an eight-channel operational signature and learns a low-dimensional representation via temporal contrastive learning, cross-view reconstruction, and generational-ordinality regularization. Evaluated on more than 33,000 production support cases spanning seven LLM families over 26 months at Google Cloud, OpEmbed recovers interpretable family- and version-level structure, improves leave-one-model-out operational forecasting over non-learned baselines, remains useful under limited early-window data, and supports cross-model fault-type transfer. We report the practical lessons learned from building and evaluating this tool for model onboarding, support readiness assessment, and operational monitoring.
1. Introduction
OpEmbed addresses the operational reliability gap left by capability benchmarks by learning compact model-level fingerprints from privacy-preserving support metadata. The framework is evaluated for forecasting, early-history monitoring, and cross-model fault transfer to support onboarding and readiness planning.
- Capability benchmarks cannot answer how deployed models differ in support load, recurring failures, resolution speed, or version-shifting operational behavior.
- OpEmbed learns compact operational fingerprints from aggregated model–time windows using structured metadata without case text or customer content.
- The framework defines eight signal channels spanning 75 features and combines temporal contrastive learning, cross-view reconstruction, and generational-ordinality constraints.
- Evaluation covers more than 33,000 production support cases across seven LLM families over 26 months.
- The learned representation supports leave-one-model-out forecasting, limited-history prediction, and cross-model fault transfer for onboarding and support readiness planning.
2. Related Work
Prior work mainly analyzes individual alerts, incidents, or support tickets, while OpEmbed analyzes model–time windows to support cross-model operational comparison and forecasting.
- Capability benchmarks primarily quantify model capability rather than operational behavior after deployment.
- AIOps and customer-support methods typically diagnose, classify, or route individual cases using logs, traces, or ticket information.
- OpEmbed uses structured metadata at the model–time-window level to learn fingerprints for cross-model comparison, few-shot forecasting, and fault-pattern transfer.
3. Method
OpEmbed converts structured model-month support data into normalized low-dimensional embeddings designed to capture stable, discriminative, and evolution-aware operational behavior. It is evaluated through forecasting and fault-transfer tasks alongside non-learned baselines and practitioner workflows.
- Problem formulation and intuition: Each model–time window is represented as a fixed-length embedding intended to be comparable across models and stable over time.
- Signature construction: Each model-month signature contains 75 features across eight operational channels, including fault, severity, resolution, interaction, routing, and workload information.
- Data augmentation: Subsample augmentation repeatedly recomputes signatures from 80% of each pair’s cases to increase effective training data while reflecting sampling uncertainty.
- Preprocessing: PCA provides a lower-dimensional warm start before row-wise L2 normalization places embeddings on the unit hypersphere.
- Multi-objective embedding learning: Temporal contrastive learning, cross-view reconstruction, and generational ordinality jointly encode temporal continuity, support-lifecycle views, and within-vendor version ordering.
- Evaluation setup: The evaluation uses more than 33,000 cases across seven LLM families and compares OpEmbed with global-average, same-family-average, and Raw sig KNN baselines.
- Evaluation tasks: Leave-one-model-out forecasting measures MAE and R2 across six operational targets, while fault transfer uses JSD and Top-K hit rate.
- Practical use: OpEmbed is positioned as offline decision support for early workload characterization, troubleshooting-playbook retrieval, and reliability reviews rather than automated support replacement.
4. Results
OpEmbed organizes operational behavior into structured family- and version-level geometry, improves forecasting across targets and workload compositions, remains useful with short histories, and transfers fault-pattern knowledge across models.
- Operational geometry: OpEmbed forms family clusters and smooth version trajectories in a primarily qualitative t-SNE landscape.Google, Anthropic-Claude, and Meta-Llama models occupy distinct regions while version lines trace local trajectories.
- Forecasting: OpEmbed achieves lower forecasting error than all three non-learned baselines across all six targets.The comparison includes global average, same-family average, and Raw sig KNN references.
- Forecasting: 5,534 technical-support cases show that predictability changes with workload composition: Escalation R2 rises from 0.031 to 0.592, while Bug Rate R2 drops to 0.437.Filtering quota-increase requests reverses the relative strength of several operational targets.
- Early-history forecasting: At the earliest observation point, OpEmbed outperforms the global-average baseline across all four early-history forecasting targets.For Bug Rate, MAE declines from about 7.0 at 10 windows to below 5.0 with all 106 windows; windows are pooled model-month windows.
- Cross-model transfer: OpEmbed reaches median JSD ≈0.179 versus 0.335 for Raw sig KNN in cross-model transfer.At Top-3 and Top-5, OpEmbed reaches 0.69 and 0.77, compared with roughly 0.50–0.59 for baselines.
- Cross-model transfer: In five representative cases, OpEmbed identifies all three actual fault types for the target model, including cross-family nearest neighbors.For gemini-2.0-flash, it anticipates a shift from Quotas to 429 Errors that global and same-family baselines miss.
5. Conclusion
OpEmbed learns operational fingerprints that support forecasting, early-history monitoring, and cross-model fault transfer. Its practical value lies in decision support for onboarding, triage, and staffing rather than visual clustering alone.
- 5. Conclusion: Operational metadata alone supports useful representation learning under enterprise privacy constraints.The approach does not require case text.
- 5. Conclusion: OpEmbed remains informative with limited early histories and supports cross-model fault-pattern retrieval.Figure 3 covers partial early histories, while Figure 4 covers cross-model transfer.
- 5. Conclusion: The most useful embedding is the one that supports onboarding, triage, and fault-pattern retrieval, not necessarily the one with the most separated clusters.This frames OpEmbed as a practical decision-support layer rather than only an analysis artifact.
6. Limitations
The evaluation is bounded by a single support ecosystem, imbalanced and sparse data, unresolved objective attribution, and operational similarity that complements rather than replaces capability benchmarks.
- 6. Limitations: The single support ecosystem may encode local workflows, so other ticket schemas may require adapted operational channels.The authors caution that the eight channels should not necessarily be reused as-is elsewhere.
- 6. Limitations: Quota-related requests dominate the imbalanced corpus, reducing variance in several targets, while sparse fields limit some channels.These data properties constrain the information available for learning operational structure.
- 6. Limitations: The study does not isolate the marginal contribution of temporal contrastive learning, cross-view reconstruction, and generational-ordinality objectives.A controlled ablation is deferred to follow-up work.
- 6. Limitations: OpEmbed captures operational similarity rather than intrinsic model capability and should complement, not replace, benchmark-based evaluation.The learned representation addresses a different evaluation axis from capability benchmarks.
Data Availability
The study uses internal production support metadata from customer-facing cloud operations, but raw data cannot be publicly released because of governance, security, and confidentiality obligations.
- Data Availability: The raw production support metadata cannot be made publicly available under internal governance, security review, and customer confidentiality requirements.The dataset is derived from customer-facing cloud service operations.
Appendix A. Supply information
The learned space improves model-level forecasting and organizes operationally similar models across and within families. Online deviation monitoring highlights behavior shifts for inspection, but it is descriptive rather than a validation of degradation.
- Appendix A. Supply information: OpEmbed consistently improves model-level operational forecasting over non-learned baselines across six targets.The gain over Raw sig KNN reflects the learned projection as a whole, although PCA and objective contributions are not separated.
- Appendix A. Supply information: Within-family nearest neighbors preserve fine-grained operational similarity among closely related Anthropic and Gemini variants.Examples include claude-3.5-sonnet / claude-3.7-sonnet at 0.044 and gemini-2.0-flash / gemini-2.5-pro at 0.046.
- Appendix A. Supply information: Cross-family neighbors show that operational similarity can diverge from vendor lineage when observed support behavior is more alike.DeepSeek and Llama examples are closest to Claude models rather than to same-vendor models.
- Appendix A. Supply information: Online embedding deviation identifies windows where operational behavior moves away from recent history for inspection.The preceding-window reference makes the view suitable for monitoring without future data.
- Appendix A. Supply information: High deviation does not necessarily mean worse service quality because updates, workload redistribution, customer-mix changes, or emerging support patterns can also produce it.Figure 7 is therefore an operational monitoring view, not formal validation of degradation.
- Appendix A. Supply information: The paper’s practical motivation is that metadata-only operational fingerprints complement capability benchmarks for production reliability assessment.This privacy-oriented framing supports enterprise deployment without case text.
A B C
The review identifies missing methodological controls and uneven operational-target performance as central concerns. It also questions whether the reported gains can be attributed to the learned objectives rather than PCA.
- Methodological controls: The evaluation lacks a PCA-only KNN baseline, so gains over Raw sig KNN cannot be decomposed between PCA and the learned objectives.The authors attribute the comparison to the learned projection as a whole, but do not isolate the PCA step.
- Target dependence: Escalation and consultation R2 are near zero on the full dataset and recover only after quota-driven cases are filtered out.This makes aggregate averages sensitive to workload composition.
- Interpretability: The paper’s interpretability claims rely on a qualitative t-SNE plot, which is suggestive rather than evidentiary.
Reviewer 2
The review finds OpEmbed relevant and potentially novel, but argues that its practical advantage, evaluation validity, definitions, and presentation remain insufficiently established. Concerns include possible target leakage, incomplete baselines, unclear family grouping, overstated early-history results, and missing replication data.
- Summary: OpEmbed represents LLMs with signatures constructed from 75 features over model-month periods and uses them for operational prediction and fault transfer.
- Soundness and evaluation: The evaluation does not establish leakage-free forecasting or sufficient practical benefit over simpler approaches.
- Soundness and evaluation: The signature may include features corresponding directly to predicted targets, while target removal and strictly earlier-window prediction are not explicitly stated.
- Baselines: The review questions whether same-family averages are a sufficiently competitive comparator because finer generation-based groupings may retrieve closer models.
- Definitions: The paper does not clearly define its seven LLM families, leaving family counts and membership ambiguous across figures and tables.
- Results: Figure 3 is judged to overstate early-history performance because OpEmbed is worse than same-family averages for consultation rate and mixed for median response time.
- Presentation and reproducibility: The review reports that no replication data are provided and that the paper is difficult to follow because of missing definitions, confusing figures, and unconventional structure.
Reviewer 3
Reviewer 3 recommends acceptance and finds the operational-fingerprint contribution intriguing, especially its use of realistic production data and privacy-preserving metadata. The main concern is that the paper demonstrates useful analytical signal but does not substantiate operational decision support through real workflows or outcomes.
- Recommendation 4 (Accept); the reviewer also records “No use of AI: Yes.”
- OpEmbed is viewed as a solid cross-model, cross-vendor contribution distinct from case-level AIOps, with privacy-by-design supporting adoption.
- The evaluation uses 33,000+ real production support cases across seven LLM families over 26 months.
- The paper acknowledges reproducibility and objective-isolation limitations, while some acronyms are not defined on first use.The reviewer specifically mentions the missing PCA-only ablation and unisolated contributions of the three training objectives.
- The central weakness is the unsupported leap from useful signal to a decision-support layer for staffing, onboarding, and triage.The review notes there are no real decision examples, workflow thresholds, or outcome gains.