AutoTSG: Learning and Synthesis for Incident Troubleshooting

1 INTRODUCTION

• More than 50,000 TSGs support incident resolution at Microsoft, but on-call engineers manually interpret and execute their varied contents.TSGs can include commands, scripts, queries, natural-language instructions, and screenshots.Source: §1 · p. 1
• The empirical study finds that incidents linked to TSGs have reduced mitigation time, indicating that the guides are used in incident resolution.The study examines TSG usage and effectiveness at Microsoft.Source: §1 · p. 1
• Feedback from 400+ on-call engineers informs a taxonomy of TSG quality issues, including completeness, validation, and maintenance problems.The authors also identify readability and fragmentation as documentation issues.Source: §1 · p. 1
• AutoTSG combines machine learning and program synthesis to extract TSG components and parse them into executable workflow constituents.Its evaluation covers 50 TSGs, and a survey covers 10 Microsoft engineers.Source: §1 · p. 1

2.1 TSG Usage

• The usage analysis mapped actual TSG link click-throughs to over 1000s of incidents and ≈4800 TSGs collected over four months.This captures on-call use rather than recommended TSGs.Source: §2.1 · p. 2
• Over four months, ≈47% of TSGs were linked to at least two incidents, and 85% were linked to high- or medium-severity incidents.Six TSGs were linked to 100+ incidents each; the highest count was 184.Source: §2.1 · p. 2
• For Sev 1–2 incidents, mean TTM was ≈19 hours without TSGs versus ≈13 hours with TSG linkage, indicating lower mitigation effort.The analysis treats TTM as a proxy for on-call effort for these severities.Source: §2.1 · p. 2

2.2 TSG Quality

• The quality taxonomy was developed from 428 feedback items for 394 TSGs using open coding, with annotator agreement of Cohen’s kappa 0.907.The feedback included ratings and free-text suggestions; annotators resolved disagreements by selecting the first and most specific intent.Source: §2.2 · p. 3
• Completeness, Broken Link, and Correctness feedback together accounted for 56.77% of items; Readability and UX accounted for ≈20%.Completeness was the most frequent category at 32.24%.Source: §2.2 · p. 3
• Most feedback items were negative (367, or 85.7%), and every major category except Other had a majority of negative ratings.Empty had the highest negativity ratio, followed by Correctness (23:1), Broken Link (18:1), and Up-to-dateness (13:1).Source: §2.2 · p. 4

2.3 Implications & Recommendations

• The authors recommend maintenance and testing for content defects, alongside standard formats, a unifying platform, and user-in-the-loop review for presentation and UX issues.They note that TSGs are often siloed text documents and their executable content must be parsed.Source: §2.3 · p. 4
• The authors envision TSGs as verified semi-automated or fully automated workflows executable with minimal manual touches.They connect this direction to code review, regression testing, and version control for TSGs.Source: §2.3 · p. 5

3 AutoTSG: TOWARDS TSG AUTOMATION

• AutoTSG identifies TSG components, then parses them to extract constituents needed for execution, guided by components, control flow, and constituent structure.The framework targets limited labeled examples for component identification and the effort required to handcraft parsers.Source: §3 · p. 5

3.1 Component Identification

• Few-shot meta-learning addresses the need to classify component types from minimal labeled examples while supporting new types with few examples.The approach learns component similarity before using nearest-neighbor search to identify types.Source: §3.1 · p. 5, §3 · p. 5

3.1.1 Preprocessing.

• AutoTSG converts TSGs to Markdown, cleans and segments them into statements, then tokenizes them with rules for camel-case, URLs, commands, and queries.The conversion and cleaning step prunes difficult-to-parse images and tables; OCR is a planned extension for multimodal information.Source: §3.1.1 · p. 5

3.1.2 Meta-Learning Framework.

• The meta-task predicts whether a pair of statements belongs to the same component type, learning a similarity kernel that separates different types in an embedding space.This is framed as metric-based meta-learning, where similarity weights support-set labels.Source: §3.1.2 · pp. 5–6

3.1.3 Siamese Network.

• A Siamese Network uses twin, shared-weight networks to embed statement pairs and their L1 distance, which is converted to a probability for binary training.The model uses bag-of-words features and a convolutional network with a 100-dimensional embedding and a 128-dimensional dense layer.Source: §3.1.3 · p. 6

3.1.4 Nearest neighbour Search.

• Instead of comparing every statement with every training example, AutoTSG compares it with each component type’s mean embedding prototype to assign the nearest type.This reduces the stated search complexity from O(#training-examples × #statements) to O(#component-types × #statements).Source: §3.1.4 · p. 6

3.2 Component Parsing

• For component parsing, programming-by-example synthesis learns Python parsers from 1–5 user examples, avoiding large training datasets and allowing users to edit minor issues.For repeated constituents, AutoTSG learns a parser for the first, removes extracted constituents, and repeats extraction while constituents remain.Source: §3.2 · pp. 6–8
• PROSE can synthesize conditional parsers for complex formats, while constituency tagging supplies clause boundaries that avoid fragile punctuation-based rules for natural language.The tagger marks declarative clauses, subordinate clauses, and verb phrases; the parser uses those tags as anchor points and can handle missing punctuation.Source: §3.2 · p. 7

4.1 Component Identification Evaluation

• 0.89 average accuracy significantly exceeded the baselines, with F1 scores of 0.72–0.98 across component types, including difficult classes.The manually labeled evaluation set contained 661 examples drawn from 1,902 statements in 50 TSGs.Source: §4.1 · p. 8, Table 5 · p. 9

4.2 Component Parsing Evaluation

• 0.94 precision and 0.91 recall were the parsers’ average results, learned from 5–10 specification examples.The evaluation withheld specification examples from the test set; parsing also handled varied conditional statements.Source: §4.2 · pp. 8–9, Table 6 · p. 9

4.3 TSG Coverage Evaluation

• 21.24% of all TSG lines were correctly automatable on average; 12/15 TSGs had 100% of automatable lines correctly identified and parsed, with maximum coverage 41.2% for TSG#8.Most lines were not automatable, including section headers, dates, links, comments, and contact information.Source: Figure 4 · p. 9, §4.3 · p. 9

4.4 AutoTSG User Study

• Six participants used TSGs every other week, three monthly, and one daily; all reported TSG quality challenges, chiefly completeness, readability, and usability.Source: §4.4 · p. 10
• All participants said TSG automation would help, and rated AutoTSG 4.2 out of 5 on average.Six participants rated it 4, three rated it 5, and one gave a neutral rating of 3.Source: §4.4 · p. 10

• Incident-management research covered triage, mitigation, diagnosis, and mining knowledge from reports, while Jiang et al. studied TSG recommendations rather than TSG quality and automation.The authors position their work as a study of TSG quality and a framework for automating TSGs.Source: §5 · p. 10
• Software-documentation tools automate summarization or recommendation, whereas AutoTSG translates manual documentation into executable workflows.Prior documentation-quality studies examined software maintenance broadly; the authors focus on troubleshooting documentation.Source: §5 · p. 10
• AutoTSG uses few-shot learning with metric-learning: a Siamese network embeds TSG statements, then nearest-neighbor search classifies components.The approach draws on meta-learning and nearest-neighbor classification like Snell et al. [61].Source: §5 · p. 11
• Programming-by-examples has been used for code-related tasks, but AutoTSG applies PROSE to synthesize parsers for varied TSG components from minimal input-output specifications.The authors identify software documentation as a new domain for programming-by-examples.Source: §5 · p. 11

• On 50 TSGs, AutoTSG achieved 0.89 accuracy for identifying statements and 0.94 precision and 0.91 recall for parsing them.Source: §6 · p. 11
• The study of over 4K TSGs mapped to thousands of incidents found frequent TSG use and lower mitigation time and effort associated with TSGs.Source: §6 · p. 11
• Feedback from 400+ on-call engineers revealed TSG quality gaps, including completeness, maintainability, and readability, which the authors organize into a taxonomy.Source: §6 · p. 11
• The authors plan to extend AutoTSG toward inferring a queue of guides from multiple services for complex incidents.Source: §6 · p. 11