PaperlayerView exact revision on arXiv

Source-linked AI summary

TSG Suggester: Tree-Structured Knowledge-Graph Retrieval for Troubleshooting Guide Recommendation in Cloud Incident Management

Shawn Pan, PavanUttej Ravva, Walt Williams, CJ Barberan, Nutan Sahoo, Ziran Min, David Gross, Irene Shaffer

arXiv:2609.35780v1cs.IR

TL;DR

Troubleshooting guides describe solutions while incidents describe symptoms, creating a vocabulary gap that makes matching an incident to the applicable guide difficult. Tree + KG decomposes guides into sub-problems, rewrites subtrees as problem statements, and combines best-node embedding similarity with entity-overlap matching. 54.78% Top-1, 77.07% Top-3, and 82.48% Top-5: Tree + KG achieves the highest accuracy at every cutoff among the evaluated strategies.

  • Problem

    Troubleshooting guides describe solutions while incidents describe symptoms, creating a vocabulary gap that makes matching an incident to the applicable guide difficult.

    Source: A. Why TSG Retrieval Is Hard · p. 1

  • Method

    Tree + KG decomposes guides into sub-problems, rewrites subtrees as problem statements, and combines best-node embedding similarity with entity-overlap matching.

    Source: D. Tree + Knowledge Graph · p. 5, A. Why Tree + KG Outperforms Flat… · p. 8

  • Results

    54.78% Top-1, 77.07% Top-3, and 82.48% Top-5: Tree + KG achieves the highest accuracy at every cutoff among the evaluated strategies.

    Source: §XI · p. 9

  • Takeaways & Limitations

    Structural alignment is supported as a stronger retrieval direction than adding content through caption-and-inject enrichment, which falls 22.64 Top-5 points below text-only retrieval.

    Source: C. Visual Understanding Does Not… · p. 7, §XI · p. 9

    The corpus comes from one organization’s guide repository and incident-management toolchain, so authoring conventions and vocabulary-gap severity may vary elsewhere.

    Source: B. Limitations · p. 8

Abstract

from arXiv · show

On call engineers in large scale cloud services work under intense time pressure, yet locating the correct Troubleshooting Guide (TSG) for an incident remains a largely manual, keyword driven process, and prior empirical work finds that guide search consumes a substantial fraction of total mitigation time. We present TSG Suggester, a retrieval system that recommends relevant TSGs directly from an incident description. We evaluate five retrieval strategies: Text Only RAG, Image Augmented RAG, RAPTOR, Tree Structured Retrieval, and our proposed Tree + Knowledge Graph (Tree+KG), on 314 real world incidents spanning 112 unique TSGs drawn from 18 service teams on a production incident management system. Tree+KG converts each guide into a tree that preserves its native section hierarchy, attaches LLM generated problem abstractions to internal nodes to bridge the solution oriented language of guides and the problem oriented language of incidents, extracts a per guide entity knowledge graph, and fuses embedding similarity with entity level matching at query time. Tree+KG attains 54.78% Top 1 and 82.48% Top 5 accuracy, leading every baseline at every cutoff, with an 8.58 point Top 1 gain over text only RAG. Two findings are of independent interest. First, structural alignment dominates: methods that preserve or rebuild document structure outperform flat chunking where precise discrimination matters. Second, and contrary to our initial hypothesis, multimodal enrichment actively hurts. Captioning guide screenshots and injecting the captions costs 22.64 Top 5 points relative to the text only baseline because generic captions dilute embeddings rather than sharpen them. We report error analyses for both results and provide concrete deployment guidance.

I. Introduction

Finding the applicable troubleshooting guide is difficult because incident symptoms and guide procedures use different vocabularies, and long, overlapping guides further challenge retrieval. TSG Suggester uses guide structure and problem-oriented abstractions to recommend relevant guides from an incident.

A. Why TSG Retrieval Is Hard

  • Incident symptoms and guide procedures can share almost no surface tokens, while long, multi-topic and near-duplicate guides make precise retrieval harder.Fixed-size chunking can split a diagnostic step from the symptom it diagnoses.Source: A. Why TSG Retrieval Is Hard · pp. 1–2

I. Introduction

  • TSG Suggester recommends semantically relevant guides by preserving each guide’s native hierarchy and rewriting subtree content as problem-oriented abstractions.It also extracts a per-guide knowledge graph and fuses tree-node embedding similarity with entity matching at query time.Source: B. Our Approach · p. 2

B. Our Approach

  • The study compares text-only RAG, image-augmented RAG, RAPTOR, and tree-structured retrieval with the proposed approach.Its research questions test native hierarchy, entity matching, and whether captioning images provides retrievable signal.Source: B. Our Approach · p. 2

C. Contributions •

The paper compares five retrieval strategies and presents Tree + KG, which combines native guide structure with entity matching. Its contributions also include a negative result for caption-based visual enrichment and deployment recommendations.

C. Contributions •

  • Tree + KG leads at every cutoff, with its largest margin at Top-1, where discrimination is hardest.It pairs native-hierarchy trees carrying LLM problem abstractions with per-guide entity graphs and fuses embedding similarity with entity matching.Source: C. Contributions • · p. 2
  • Image-augmented retrieval falls 22.64 Top-5 points below text-only retrieval, despite screenshots and diagrams appearing in guides.The paper characterizes this as a retrieval-stage result, not a conclusion about multimodal understanding generally.Source: C. Contributions • · p. 2, A. Anatomy of a Troubleshooting Guide · p. 2
  • The deployment guidance is to return five candidates, invest in offline knowledge-graph construction, and use similarity structure to find and consolidate near-duplicate guides.Source: C. Contributions • · p. 2

A. Anatomy of a Troubleshooting Guide

  • Guides use a recognizable hierarchy of problem statements, symptoms, diagnostics, mitigations, and escalation, alongside images such as dashboards and diagrams.The images can carry information not expressed in surrounding prose.Source: A. Anatomy of a Troubleshooting Guide · p. 2

B. The Retrieval Setting

The retrieval setting calls for a short, accurately ranked list of guides because on-call engineers work under time pressure and a plausible but inapplicable guide can cost mitigation time. The paper positions guide retrieval as complementary to root-cause systems and varies document representation while holding the embedding model fixed.

B. The Retrieval Setting

  • On-call engineers need a short candidate list, and a wrong-but-plausible guide can cost mitigation time, motivating honest ranking over a single confident answer.The setting makes Top-1 through Top-5 the relevant metrics rather than deep-ranking metrics.Source: B. The Retrieval Setting · p. 2

A. Incident Management and AIOps

B. Troubleshooting Guide Recommendation

  • The proposed retrieval signal is the guide’s internal structure: subtree problem abstractions bridge vocabulary mismatch, while the knowledge graph supplies a discrete signal when continuous similarity is ambiguous.Source: B. Troubleshooting Guide Recommendation · p. 3

C. Dense Retrieval and Retrieval-Augmented Generation

  • The comparison holds the embedding model fixed across five methods, varying the embedded text and its organization to isolate representation structure from encoder quality.Sentence-BERT is the embedding model used in the pipeline.Source: C. Dense Retrieval and… · p. 3

D. Hierarchical and Structure-Aware Retrieval

Related approaches build hierarchies through clustering or combine native document trees with entity graphs, while this paper adapts structural and graph retrieval to guide recommendation. The section also limits its visual-retrieval claim to caption-and-inject enrichment and describes evaluation against recorded guide labels.

D. Hierarchical and Structure-Aware Retrieval

  • RAPTOR constructs a hierarchy by recursively clustering and summarizing chunks [20], whereas BookRAG pairs a document tree with an entity graph and tree-node mapping [22].Tree + KG shares that structural recipe but adds problem abstractions to address the vocabulary mismatch in troubleshooting-guide retrieval.Source: D. Hierarchical and Structure-Aware… · p. 3

E. Knowledge-Graph-Augmented Retrieval

  • Unlike corpus-wide graph retrieval methods [23], [24], [25], Tree + KG builds a separate graph for each guide and does not merge graphs.This keeps guide boundaries distinct and makes re-indexing one edited guide straightforward, but cannot support reasoning across guides.Source: E. Knowledge-Graph-Augmented Retrieval · p. 3

F. Multimodal Document Understanding and Retrieval

G. Evaluation

  • The evaluation reports Top-k accuracy against recorded guide labels, which may reflect the guide an engineer used rather than the best guide available.The paper notes that reference-free or judge-based evaluation [34], [35], [36] becomes relevant when there is no single correct answer.Source: G. Evaluation · p. 4

IV. Problem Formulation

The paper formalizes guide recommendation as ranking likely troubleshooting guides for an incident, then describes the evaluation corpus, system pipeline, and comparison methods.

IV. Problem Formulation

  • The task is to return a ranked list of k guides most likely to apply to an incident described by its title, severity, summary, and discussion log.The corpus includes guides with structured sections and may include dashboard screenshots, call-flow diagrams, and query-output visualizations.Source: §IV · p. 4, A. Dataset · p. 4
  • Top-k accuracy measures the fraction of incidents whose ground-truth guide appears in the returned list for k values of 1, 3, or 5.Ground truth is the guide recorded in the incident-management system by the resolving engineer.Source: §IV · p. 4

V. Methodology

  • The system indexes each guide as tree-structured embeddings plus a per-guide knowledge graph, then ranks incidents using embedding similarity and entity matching.The indexing phase is offline; retrieval scores an incoming incident against the resulting index.Source: §V · p. 4

A. Dataset

  • Over 30,000 production incidents were retrieved; the evaluation set contains 314 sampled incidents, 112 unique TSGs, and 18 service teams.The corpus statistics are reported in Table I.Source: A. Dataset · p. 4, Table I · p. 4
  • Each incident includes a title, severity, automatically generated summary, and chronological discussion log; guides contain problem, symptom, diagnostic, and mitigation sections.Guide content may also include dashboard screenshots, call-flow diagrams, and query-output visualizations.Source: A. Dataset · p. 4

B. Baseline Methods

  • Text-only RAG uses fixed-size chunks, image-augmented RAG adds VLM captions, and RAPTOR recursively clusters chunks and summarizes them at multiple levels.All three retrieve through the same broad chunk-based or clustered representation, unlike following each guide’s authored hierarchy.Source: B. Baseline Methods · p. 4

C. Tree-Structured Retrieval

Tree-Structured Retrieval preserves each guide’s authored hierarchy and adds problem abstractions to align guide content with incident descriptions. Tree + KG extends this representation with per-guide entity graphs and hybrid scoring.

C. Tree-Structured Retrieval

  • Native section hierarchy is retained as a tree, while image content is replaced with generated text descriptions.Each section maps to a node and its subsections to children.Source: C. Tree-Structured Retrieval · p. 4
  • For each internal node, descendant content is summarized as a problem description and triggering conditions, aligning solution-oriented guide text with problem-oriented incidents.The resulting node embeddings are intended to occupy the same embedding region as incident descriptions rather than procedure text.Source: C. Tree-Structured Retrieval · pp. 4–5
  • The design decision emphasized is to make structure-aware retrieval more than a reorganization of the same unhelpful text.The architecture depicts offline guide indexing and online incident retrieval; its image-captioning branch is a baseline, not part of final Tree + KG.Source: C. Tree-Structured Retrieval · p. 5, Figure 1 · p. 5

D. Tree + Knowledge Graph

  • Tree + KG builds a hierarchy, generates problem abstractions, extracts a separate entity graph for each guide, and fuses tree-node similarity with entity matching at query time.Stages 1–3 run once per guide offline; only Stage 4 is on the incident-response critical path.Source: D. Tree + Knowledge Graph · p. 5, Figure 2 · p. 5
  • The score combines the best-matching abstraction node with normalized entity overlap; embedding similarity handles paraphrase, while entity matching can distinguish specific named errors or services.The max over nodes avoids averaging across unrelated sub-problems, and the two signals fail in different ways.Source: D. Tree + Knowledge Graph · p. 5

E. Visual Understanding Pipeline

The visual pipeline captions guide images and inserts the descriptions before embedding. The paper also describes data-handling controls and the common evaluation setup used to compare retrieval methods.

E. Visual Understanding Pipeline

  • Because guide images may contain diagnostic information absent from prose, the visual pipeline captions images with a VLM and inserts descriptions into the corresponding guide sections.Early synthetic-data experiments were encouraging; real-data outcomes are reported elsewhere in the paper.Source: E. Visual Understanding Pipeline · p. 5

VI. Data Handling and Disclosure

  • Guides and incident records contain commercially sensitive material and personal data, so indexing and evaluation remain inside existing access controls without exporting content.LLM and VLM components use frozen, inference-only endpoints with no training retention, and content does not update model weights.Source: §VI · p. 6
  • Reporting is limited to corpus-level aggregates over 314 incidents and 112 guides, with no per-incident, per-team, or per-guide accuracies.Names and verbatim guide, incident, query, and image content are omitted or generalized; differences smaller than reported precision should not be treated as meaningful.Source: §VI · p. 6

VII. Experimental Setup

  • All five methods are evaluated on the same 314-incident dataset using Top-k accuracy at k values of 1, 3, and 5.A hit requires the engineer-recorded ground-truth guide to appear in the returned set.Source: §VII · p. 6
  • Sentence-BERT embeddings and one frontier commercial LLM are held across methods for representation comparisons, with additional analyses of rank, confusion, severity, and guide length.The LLM performs abstraction, entity and relation extraction, image captioning, and incident summarization.Source: §VII · p. 6

VIII. Results

Tree + KG leads retrieval accuracy at every cutoff, with gains from entity matching larger than those from preserving native guide structure. Native hierarchy also outperforms a constructed hierarchy, while tree-only retrieval has a Top-5 caveat that Tree + KG avoids.

A. Tree + KG Leads at Every Cutoff (RQ1, RQ2)

  • 54.78% Top-1, 77.07% Top-3, and 82.48% Top-5: Tree + KG leads all five methods at every cutoff, beating text-only RAG by 8.58 Top-1 points.Its Top-1 accuracy is also 6.43 points above tree-structured retrieval without the knowledge graph.Source: Figure 3 · p. 6, A. Tree + KG Leads at Every Cutoff… · p. 6
  • +2.15 Top-1 points come from native structure with problem abstractions, while adding per-guide entity matching contributes a further +6.43 points.The entity-matching gain is about three times larger and is the only one of the two large enough to be resolved on a corpus of this size.Source: A. Tree + KG Leads at Every Cutoff… · pp. 6–7, Figure 4 · p. 7

B. Native Structure Beats Constructed Structure

  • 37.66% Top-1 and 72.08% Top-5 for RAPTOR fall below text-only RAG, whereas native tree retrieval reaches 48.35% Top-1.The authors attribute the difference to RAPTOR summaries discarding discriminative details, while native-hierarchy problem abstractions change the language without averaging away specifics.Source: B. Native Structure Beats Constructed… · p. 7
  • 75.68% Top-5 for Tree Structure trails text-only RAG at 81.20%, although Tree + KG leads at Top-5 with 82.48%.The authors characterize native structure as sharpening the ranking head while narrowing its tail; entity overlap recovers guides when no node abstraction closely matches the incident.Source: B. Native Structure Beats Constructed… · p. 7

C. Visual Understanding Does Not Help—It Hurts (RQ3)

Caption-and-inject image augmentation reduces retrieval accuracy, with the authors attributing the decline to generic, redundant captions that further dilute embeddings. The representation also supports identifying near-duplicate guides as documentation-maintenance candidates.

C. Visual Understanding Does Not Help—It Hurts (RQ3)

  • 27.63% Top-1 and 58.56% Top-5; image-augmented retrieval trails text-only retrieval by 18.57 Top-1 points and 22.64 Top-5 points.The authors identify this as the study’s largest single effect and its only negative one.Source: C. Visual Understanding Does Not… · p. 7
  • Generic captions provide little discrimination, worsen embedding dilution in already-long guides, and can redundantly restate surrounding prose.These are the three causes identified in the error analysis.Source: C. Visual Understanding Does Not… · p. 7
  • The negative results for RAPTOR and image augmentation align with greater sensitivity to content structure than to added abstraction or content.The authors characterize both as sacrificing discriminative precision.Source: C. Visual Understanding Does Not… · p. 7

D. Near-Duplicate Guide Detection

  • Tree + KG’s representation can flag near-duplicate guides as candidates for consolidation or rewriting, turning retrieval confusion into a documentation-quality signal.Such overlaps can leave engineers unaware that another, potentially more current guide exists.Source: D. Near-Duplicate Guide Detection · p. 7

A. Why Tree + KG Outperforms Flat Retrieval

Tree + KG counters embedding dilution by preserving guide structure, matching problem-oriented abstractions, and adding entity signals. The section also explains why caption-injection underperforms and gives deployment recommendations.

A. Why Tree + KG Outperforms Flat Retrieval

  • Tree + KG decomposes guides at authored section boundaries, so its max-over-nodes score can use the best-matching subproblem instead of averaging the whole guide.Figure 5 contrasts this with flat retrieval’s diluted vector for a long, multi-topic guide.Source: A. Why Tree + KG Outperforms Flat… · p. 7, Figure 5 · p. 8
  • Problem abstractions bridge guide–incident vocabulary, while entity matching adds discrete signals for services, error codes, and tools that cosine similarity cannot express.The abstractions restate what a subtree solves and under what conditions it triggers.Source: A. Why Tree + KG Outperforms Flat… · p. 8

B. Why Visual Enrichment Underperformed

  • Generic captions can add length without discriminative information, worsening dilution; useful images for readers who have found a guide need not help retrieve it.The finding is specific to caption-and-inject retrieval; native visual retrieval is a different design that the authors describe as promising.Source: B. Why Visual Enrichment Underperformed · p. 8

A. Deployment Recommendations

  • Return five candidates: the Top-5 gain over Top-1 is 27.70 points, and an engineer can dismiss four wrong candidates quickly.The authors argue that showing one suggestion discards more than a quarter of the system’s achievable value.Source: A. Deployment Recommendations · p. 8
  • Use Tree + KG as the retrieval backbone: its per-guide graph is built offline when a guide is authored or edited, while the 8.58-point Top-1 gain over the flat baseline applies on every incident.Per-guide graphs also keep re-indexing an edited guide cheap.Source: A. Deployment Recommendations · p. 8
  • Use similarity detection proactively to flag near-duplicate guides, which degrade both human and machine retrieval.The same representation used for recommendations can identify these cases.Source: A. Deployment Recommendations · p. 8

B. Limitations

Tree + KG leads the five-strategy evaluation, while the authors qualify comparisons by sample size, recorded-guide labels, and the single-organization corpus. They outline extensions including cross-guide reasoning and alternative evaluation and retrieval approaches.

B. Limitations

  • With 314 incidents, the +2.15-point tree-only gain over flat chunking is suggestive rather than established; per-incident outcomes were unavailable for a more powerful paired test.The authors give an approximate 95% interval of ±5.5 points for accuracy near 50%.Source: B. Limitations · p. 8
  • Recorded guide labels reflect what engineers used, not necessarily the best available guide; the corpus is from one organization, and results lack per-team or per-guide breakdowns.LLM entity extraction may introduce spurious entities, which the authors did not separately quantify.Source: B. Limitations · p. 8

C. Future Work

  • Future directions include merging per-guide graphs for cross-guide reasoning, relation-path scoring, judge-based evaluation, native visual retrieval, and integration with guide-execution agents.Graph merging could enable cross-guide reasoning but would trade away some of the boundary sharpness that currently helps retrieval.Source: C. Future Work · pp. 8–9

XI. Conclusion

  • Tree + KG leads all five retrieval strategies at every cutoff, reaching 54.78% Top-1, 77.07% Top-3, and 82.48% Top-5 on 314 incidents.The evaluation spans 112 guides from 18 service teams.Source: §XI · p. 9
  • The authors’ central conclusion is that structural alignment beats semantic enrichment, with Tree + KG contributing 8.58 Top-1 points over the flat baseline.RAPTOR and caption-and-inject enrichment fall below that baseline.Source: §XI · p. 9

Related papers

Loading 2609.35780v1…