ASTRA - Agentic System for Ticket Resolution and Analysis

Shashidhar Reddy Javaji; Mohamed Trabelsi; Jin Cao; Huseyin Uzunalioglu

Source-linked AI summary

ASTRA - Agentic System for Ticket Resolution and Analysis

Shashidhar Reddy Javaji, Mohamed Trabelsi, Jin Cao, Huseyin Uzunalioglu

arXiv:2608.28790v1Aug 28, 2026cs.MAcs.AIcs.IRcs.LGcs.SE

TL;DR

Technical operations teams handle large volumes of tickets, while diagnostic evidence can be difficult to synthesize under constrained attention. ASTRA coordinates specialized agents to produce bounded, evidence-oriented fault analyses, achieving a mean quality score of 4.13/5.0 while revealing limitations in evaluation and text-based diagnosis.

Problem
Technical operations teams resolve large volumes of incident and support tickets under tight time constraints, while long evidence streams can dilute attention from relatively small numbers of critical signals.
Method
ASTRA is a multi-agent system that coordinates specialized agents and produces bounded artifacts for downstream verification rather than unconstrained narrative text.
Results
ASTRA achieves a mean quality score of 4.13/5.0 across five evaluation dimensions, with 59.9% of reports correctly identifying the fault area; Relevance is 4.88 and Clarity is 4.94.
Takeaways & Limitations
Presentation quality is consistently high, while the claim–evidence architecture limits fabricated technical details to under 3% of error cases.
Takeaways & Limitations
Evaluation relies on a single LLM judge, whose scores may not perfectly align with expert human judgment.

Abstract

from arXiv · show

Technical operations teams resolve large volumes of incidents by synthesizing fragmented evidence from ticket text, historical cases, system logs, and technical documentation. Existing automation often relies on monolithic generation without explicit evidence modeling or provenance, making outputs difficult to verify when critical signals are sparse across sources. We propose ASTRA, an agentic system for ticket resolution in which a central orchestrator coordinates three specialist information-gathering agents and drives a judge-orchestrator refinement loop to produce evidence-backed troubleshooting reports. TicketSimilarityAgent retrieves relevant historical precedents through dense retrieval and LLM reranking; LogAgent distills hundreds of thousands of log lines into structured, quote-grounded findings using deterministic filtering and constrained LLM analysis; and DomainKnowledgeAgent retrieves relevant technical knowledge via the Model Context Protocol (MCP). Their outputs are transformed into a claim-evidence representation linking each claim to a verbatim source passage, assigning a support level, and preventing cross-attribution. A JudgeAgent scores the report on five criteria, while the OrchestratorAgent converts low scores into targeted follow-up queries for bounded iterative refinement. Evaluated on 987 real-world telecom fault tickets across seven product lines, ASTRA achieves a mean quality score of 4.13/5.0, with 59.9% of reports identifying the fault area at the component-family level or better. Relevance and Clarity scores are 4.88 and 4.94, respectively, while fabricated technical details remain below 3% of error cases. Stratification by fault type reveals that hardware faults remain substantially harder than software or configuration faults (Cohen's d=0.80), pointing to a fundamental limitation of text-based evidence channels for hardware fault diagnosis.

1 Introduction

ASTRA addresses evidence-driven telecom ticket troubleshooting by coordinating specialized agents and making claims traceable to supporting evidence. Its judge–orchestrator loop targets weak sections through bounded, evidence-grounded refinement.

ASTRA targets end-to-end troubleshooting because existing approaches often optimize isolated stages rather than evidence-driven resolution.
The framework coordinates TicketSimilarityAgent, LogAgent, and DomainKnowledgeAgent to gather historical, runtime, and documentation evidence.
Each finding is represented as a claim paired with provenance-linked evidence, support levels, and cross-source consistency signals.This intermediate layer supports controllable generation, follow-up queries, and auditable traceability.
The judge–orchestrator loop evaluates drafts, identifies weak or under-supported sections, and triggers targeted follow-ups to relevant agents.Refinement is bounded to preserve operational usefulness while reducing unsupported conclusions.
The evaluation protocol measures report quality together with evidence grounding and retrieval faithfulness.

2 Related Work

Prior work spans ticket automation, multi-agent diagnosis, evidence verification, and realistic agent benchmarks, but persistent gaps remain in controllable, auditable troubleshooting. These gaps motivate ASTRA’s evidence-grounded decomposition and verification structure.

Ticket and incident automation: Ticket and incident systems automate routing, retrieval, aggregation, escalation, and report generation, but often address isolated workflow stages.
Agentic diagnosis: Multi-agent and RCA systems add specialized roles and tool interaction, yet may rely on brittle orchestration and expose few auditable intermediate representations.
Open challenges: Persistent challenges include distributed context, error propagation across long reasoning chains, and limited control over what systems assert versus justify.
Evaluation context: Realistic benchmarks show that robust autonomous IT and software-engineering agents remain difficult to build, with only modest solve rates in some systems.
Verification: Evidence-grounded generation and verification work emphasizes explicit claims, evidence, attribution, and validation outcomes rather than implicit end-to-end generation.

3 Problem

Automated telecom fault analysis must synthesize heterogeneous, noisy evidence while preserving conflicts and making every substantive report claim auditable. Long logs intensify the need for conservative evidence selection.

The task is automated fault-analysis report generation for telecom infrastructure, covering fault identification and the recommended fix.
Complete analysis combines the ticket with historical tickets, system logs, and technical documentation, each contributing a different signal.
Evidence quality is uneven: similar tickets may differ, logs are large and noisy, and documentation may not match the deployed configuration or version.
Sources must be cross-validated because historical, log-derived, and documentation signals can conflict or remain ambiguous when combined.
Log files can span tens of thousands of lines, so selection must retain critical evidence without hallucinating unsupported lines.
A report must link every substantive claim to its source and annotate support so findings can be distinguished by evidential strength.

4 ASTRA

ASTRA separates evidence gathering from report generation through three specialist agents and an explicit claim–evidence layer. A judge–orchestrator loop then supports targeted, bounded refinement.

ASTRA is a multi-agent framework combining three specialist information-gathering agents, claim–evidence construction, and judge–orchestrator refinement.
The central design separates evidence collection from report generation through an inspectable intermediate representation.Claims are mapped to source and confidence information before report generation.
TicketSimilarityAgent retrieves historical tickets through dense search and reranking, while DomainKnowledgeAgent queries technical documentation through MCP.
LogAgent processes full log streams with deterministic filtering, noise suppression, context expansion, and constrained analysis.The excerpt is capped at 2,000 lines, and 84% of evaluation tickets reach that limit.
The system integrates similar-ticket context into log analysis and returns structured findings including diagnostic evidence, causal narrative, and root-cause category.

5 Evaluation

ASTRA was evaluated on 987 telecom fault tickets using five judge-scored dimensions, revealing strong presentation quality but substantially weaker causal accuracy. Accuracy was especially limited for hardware and third-party faults, while refinement improved coverage more than diagnostic correctness.

Dataset and evaluation: 987 telecom fault tickets spanning seven product lines were evaluated using engineer-written fault analyses and live technical-analysis investigation chains.The technical-analysis chains contained logs, component names, parameter values, and conclusions, but were used only by the evaluation judge.
Dataset and evaluation: Five metrics—Accuracy, Completeness, Relevance, Clarity, and TechnicalDepth—were scored from 0–5, with their average forming Overall; scores ≥3 were considered useful.Accuracy assessed fault-area and mechanism alignment, with explicit anchors distinguishing exact, directional, and incorrect diagnoses.
Overall performance: Relevance scored 4.88 and Clarity 4.94, while Completeness scored 4.37 and TechnicalDepth 3.83, indicating consistently strong presentation quality.TechnicalDepth was more variable, with 92.0% of reports scoring at least 3, and refinement sometimes replaced low-level details with general explanations.
Overall performance: Accuracy averaged 2.63, with 48.2% of reports scoring 3 and only 11.7% scoring 4–5, showing that fault-area identification usually exceeded exact mechanism identification.The dominant pattern was correct symptom and fault area but a different specific mechanism from the ground truth.
Error analysis: The dominant low-accuracy failure was plausible but incorrect component attribution, affecting 271 tickets (68.4% of Accuracy ≤2 cases), while fabricated technical details remained below 3%.These errors often localized the correct product domain but selected the wrong responsible unit; the claim–evidence architecture constrained confabulation.

6 Conclusion

ASTRA coordinates heterogeneous evidence channels through a claim–evidence representation and iterative judge–orchestrator refinement, producing structured fault-analysis reports. Evaluation shows strong presentation quality and evidence-grounding safeguards, but hardware faults and component attribution remain difficult.

6 Conclusion: ASTRA coordinates historical tickets, system logs, and technical documentation through a structured claim–evidence representation and iterative judge–orchestrator refinement loop.The system is designed to connect report claims with evidence across heterogeneous channels.
6 Conclusion: 4.13/5.0 mean quality was achieved across five evaluation dimensions on 987 real-world telecom fault tickets.The evaluation covered real-world fault reports rather than a synthetic benchmark.
6 Conclusion: 59.9% of reports correctly identified the fault area, while Relevance reached 4.88 and Clarity reached 4.94.These results indicate strong report presentation alongside partial fault-localization performance.
6 Conclusion: Fabricated technical details remained below 3% of error cases, supporting the claim–evidence architecture’s grounding safeguard.The reported safeguard concerns fabricated technical details, not overall diagnostic accuracy.
6 Conclusion: Hardware faults were substantially less amenable to automated analysis than configuration and software faults (d=0.80).The authors associate this gap with a fundamental limitation of text-based evidence channels for hardware diagnosis.
6 Conclusion: Plausible but incorrect component attribution was the dominant failure mode, accounting for 68.4% of Accuracy ≤2 cases.This failure reflects unresolved ambiguity in identifying the responsible component despite technically plausible reports.

7 Limitations

ASTRA’s evaluation is constrained by judge dependence, capped log extraction, missing hardware-level evidence, and a single-organization corpus. These boundaries particularly affect hardware diagnosis and broader generalization.

A single LLM judge may not perfectly align with expert human judgment, motivating validation against engineer ratings.
84% of tickets saturate the 2,000-line diagnostic log cap, potentially hiding rare signals deeper in logs.
ASTRA lacks board sensor readings, hardware event logs, and physical component tests, limiting effectiveness on hardware faults.
Hardware faults receive the lowest reported accuracy, at 2.03.
The evaluation uses one organization’s ticket corpus, so generalization to other operators or domains requires further validation.

8 Future Work

Future work targets evidence coverage, refinement trade-offs, hardware diagnosis, human validation, and retrieval quality. Proposed changes address observed log saturation, specificity loss, and hardware-related misclassification.

Adaptive log budgets could adjust extraction limits using ticket complexity and initial signal strength for tickets exhausting refinement rounds.
Refinement improves Completeness by +0.10 but reduces TechnicalDepth by −0.54, motivating refinement-aware training.
Hardware-aware agents could add board sensor readings, hardware event logs, and component tests as evidence channels.
Adding hardware-level data could address domain-level misclassifications affecting 17.4% of low-accuracy cases.
Human-in-the-loop evaluation could compare LLM scoring with domain-engineer judgment and measure engineers’ time savings.
Retrieval-aware training could optimize ticket similarity for root-cause category overlap rather than surface symptom similarity.
This change could reduce the dominant “plausible but incorrect component” failure mode.

A ASTRA Prompts

The appendix documents ASTRA’s prompts for report generation, evidence handling, judging, refinement, and reference-aware evaluation. These prompts operationalize structured, evidence-grounded fault analysis.

A ASTRA Prompts: The appendix reproduces system-level prompts, while claim extraction, evidence mapping, and orchestrator planning prompts also appear in Section 4.
A ASTRA Prompts: All LLM calls use gpt-oss:120b at temperature 0 unless otherwise noted, with dynamic content assembled at runtime.
A.4 Report Generation Prompt: The prompt distinguishes verified from unverified claims, prohibits invented identifiers, and merges overlapping agent findings with both citations.
A.4 Report Generation Prompt: Report generation requires identification and resolution sections covering symptoms, diagnostic steps, causal chains, fixes, parameters, and verification.
A.4 Report Generation Prompt: When similarity evidence conflicts with logs, the prompt requires stating the conflict and prioritizing log evidence.
A.5 Judge Evaluation Prompt: The pipeline JudgeAgent scores reports on five criteria and triggers targeted revision when any metric scores 3 or below.
A.5 Judge Evaluation Prompt: The judge evaluates reports against agent outputs and claim-evidence mappings without access to ground truth.
A.6 TA-Aware Evaluation Prompt: The external evaluation judge compares predictions with ground-truth analyses and engineer investigation chains across five reported metrics.

A.1 TicketSimilarityAgent Analysis Prompts

TicketSimilarityAgent uses dense retrieval followed by two LLM reranking phases to select diagnostically useful historical tickets. Selection favors comparable mechanisms, actionable resolutions, confirmed causes, and diverse coverage.

TicketSimilarityAgent retrieves candidates with dense vector search before applying two-phase LLM reranking.
Phase 1 reviews historical tickets in batches of 20 using a Nokia 5G/LTE RAN fault-analysis prompt.
A ticket is useful when its subsystem, failure mechanism, and resolution provide directly transferable diagnostic or corrective information.
Tickets are rejected when they belong to another subsystem, lack a defined root cause, or share only surface keywords.
Embedding similarity is contextual only and does not itself indicate ticket usefulness.
Phase 2 prefers specific confirmed causes, complete actionable resolutions, diverse coverage, and exclusion of wrong-domain tickets.
The final selection returns at most five ranked tickets with usefulness labels, reasons, exclusions, and a collective fault-domain summary.
Both reranking phases receive target-ticket metadata and candidate details with similarity scores.

A.2 LogAgent Analysis Prompts

The LogAgent uses a constrained prompt to extract structured, quote-grounded observations from filtered logs while limiting speculation and optionally using similar resolved tickets as interpretive guidance.

Prompt construction: The LogAgent prompt combines ticket context, an enhanced description, acronym expansions, component context, an optional similar-ticket block, and filtered log content.The assembled inputs are passed to a single constrained LLM call.
Evidence constraints: The system message restricts the agent to directly observable information in the provided log lines and requires uncertainty to be reported as not determinable from logs.The prompt explicitly prohibits invented details.
Evidence constraints: Every finding must quote at least one log line, avoid unsupported paths, functions, and error codes, and distinguish observable events from speculative causes.When logs do not establish a root cause, the required response is that it is not determinable from logs alone.
Structured output: The output schema includes findings with type, component, log quote, and timestamp, plus frequency, severity, description, fault relevance, and relevance reasoning.It also includes temporal sequence, root-cause assessment, confidence, further-investigation status, and a summary based only on log evidence.
Historical references: Similar resolved tickets guide classification of matching component or error patterns, ownership of ambiguous tokens, and root-cause plausibility without permitting verbatim copying of resolutions.The prompt requires such interpretations to remain corroborated by the log lines.

A.3 DomainKnowledgeAgent Analysis Prompts

The DomainKnowledgeAgent uses other agents’ findings and ticket context to construct a bounded documentation query, then retrieves fault-relevant technical guidance through MCP.

Query construction: The DomainKnowledgeAgent queries technical documentation through the Model Context Protocol using a constrained system message and an enriched task query.The query is constructed from ticket context and outputs from other agents.
Retrieval focus: The documentation assistant focuses on root causes, error meanings, configuration parameters, resolution procedures, fixes, workarounds, and fault-relevant content.It is instructed to distinguish identification-relevant findings from resolution-relevant findings.
Evidence constraints: The agent must cite a specific document or section when available and clearly state when no retrieved document directly addresses the query rather than generalizing.This constrains documentation-based claims to retrieved support.
Runtime task: The runtime query targets a Nokia fault’s root cause, known issues, and resolution, including alarm triggers, configuration values, and step-by-step procedures.The query incorporates the ticket title and extracted fault identifiers.
Query construction: Log analysis findings and similar historical tickets are supplied as context to narrow the documentation search toward the likely fault domain.The included agent outputs are truncated to 2,000 and 1,500 characters, respectively, to remain within retrieval limits.