Source-linked AI summary
Before the Ticket: Grounding HPC User Support in Documentation and Resolved Tickets with an Agentic LLM Assistant
Elham J. Barezi, Primus C Kabuo, Niveditha Nerella
TL;DR
HPC support answers are split between current but generic manuals and specific ticket history that users cannot search. This paper combines documentation retrieval with a fine-tuned model over resolved tickets; on held-out tickets, fine-tuning agreed with staff resolutions far better than the base model, while merging did not improve on either tool.
Problem
Method
An orchestrator routes queries to documentation retrieval and a fine-tuned model over resolved tickets, then synthesizes their outputs.
Source: §1 · p. 1Results
Takeaways & Limitations
Returning the fine-tuned answer, with the documentation answer alongside as a cited second opinion, is safer than the tested merges.
Source: §5 · p. 7The central correctness comparison relies on reference metrics and LLM judges that disagree and have not been validated against human ratings for this task.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Most questions that reach an HPC help desk have already been answered in the center's user guides or in a ticket someone else filed last semester; users still file new tickets because the guides are hard to navigate and the ticket history is not theirs to search. We present an assistant that answers from both sources: an orchestrator calls two tools over the Model Context Protocol, retrieval-augmented generation (RAG) over the live documentation and a Llama-3-8B model fine-tuned with QLoRA on 11,244 query-resolution pairs mined from historical tickets, and merges their answers. Every model runs on-premises or on the university's hosted LLM service because ticket data may not leave the institution. On 400 held-out tickets from Purdue's Rosen Center for Advanced Computing, disjoint from training and free of near-duplicates, and evaluated by two judges from model families that generated none of the answers, fine-tuning on resolutions rewritten into self-service guidance is the clear winner in agreement with RCAC's resolutions (BERTScore F1 improves from 0.06 to 0.27 over the base model) and is judged most correct on average. The end-to-end system does not improve on its tools, although they are complementary: choosing the better answer per question would exceed either tool (judged correctness 3.41 versus 3.20, an oracle bound), while the merge follows the documentation tool in 78% of answers. Returning one tool's answer instead recovers the fine-tuned tool's quality (3.19 versus 2.66), but neither a learned selector nor an LLM verifier beats that tool alone.
1 Introduction
HPC support knowledge is split between current but generic guides and specific but inaccessible ticket history. The paper combines these sources in a pre-ticket assistant; fine-tuning improves agreement with staff resolutions, but the end-to-end synthesis does not improve on its tools.
1 Introduction
- A large public-university center logged more than 20,000 tickets and nearly 95,000 messages from 2017 to 2023, with an average resolution time of 11.8 business days.Support staff across industries report spending 60–70% of their time on previously solved problems.Source: §1 · p. 1
- Guides are authoritative and current but generic, while ticket history captures practical fixes but is inaccessible to users and ages as systems change.Support answers often need both sources: what the manual says and what fixed the issue previously.Source: §1 · p. 1
- The work addresses combining documentation and ticket knowledge under the constraint that ticket data cannot leave the institution and models must run on local or institutionally hosted hardware.Prior LLM work on IT support mostly automates tasks after ticket filing; this assistant intervenes before a ticket.Source: §1 · p. 1
- The assistant routes support queries to documentation retrieval and a fine-tuned ticket model, then synthesizes their answers with precedence and source attribution.The training-data pipeline extracts query–resolution pairs, rewrites staff actions as self-service guidance, filters answers, masks residual PII, and splits by ticket while removing near-duplicates.Source: §1 · p. 1
- Fine-tuning on ticket resolutions wins on agreement with staff resolutions, while the end-to-end system does not improve on its tools.The tools are complementary, but synthesis and selectors do not exceed the fine-tuned tool.Source: §1 · p. 2
2 Setting and Data
The study draws on RCAC’s public guides and active ticket systems, then converts ticket threads into self-service training pairs. Its pipeline removes unresolved or unsafe examples, masks personal information, and separates tickets and near-duplicates across splits.
2.1 The center
- RCAC serves researchers, students, and research software engineers through a central service desk, with public guides and a private ticket history.The center runs several shared clusters, and its TicketHub platform already indexes ticket history with NLP.Source: §2.1 · p. 2
2.2 Knowledge sources
- The evaluated documentation crawl contains 849 pages, while the two active ticket systems contribute 15,268 tickets from 2023 onward.The larger legacy Footprints corpus was excluded because much of it concerns retired hardware and obsolete documentation.Source: §2.2 · p. 2
2.3 Turning threads into training data
- The pipeline cleans and pseudonymizes ticket records, then uses an LLM to extract first-person queries and direct-instruction resolutions while preserving technical tokens.Prompted Gemma 3 was selected for extraction; unresolved and failed extractions were removed, leaving 13,332 pairs.Source: §2.3 · p. 2
- Staff actions and transient-state claims are rewritten into user-facing guidance or filtered out, leaving 12,692 pairs after 632 examples are removed.The strict filtering rules also catch some useful answers.Source: §2.3 · p. 2
- Residual names can remain after masking, so the data and fine-tuned model are not released; the final ticket-level split contains 11,244 training, 600 validation, and 400 test pairs.Exact duplicates and questions with cosine similarity ≥0.92 to training questions or another kept question in the same split were removed.Source: §2.3 · pp. 2–3
3 System
The system keeps documentation and ticket knowledge in separate tools, with the orchestrator deciding how to reconcile their answers. It combines live guide retrieval, a locally served fine-tuned model, and routing and synthesis over MCP.
3 System
- The design keeps documentation and ticket knowledge in separate tools so each answer component is grounded in its own material, leaving reconciliation to the orchestrator.Ticket-data governance requires models and embeddings to remain on-premises or institutionally hosted.Source: §3 · p. 3
3.1 Tool 1: RAG over the user guides
- Documentation retrieval searches 849 pages with embedding and BM25 candidates, reranks them, and sends up to six relevant chunks to Llama 3.1 with instructions to answer only from those sources.If no chunks pass the relevance threshold, the tool says the documentation does not cover the question and avoids calling the model.Source: §3.1 · p. 3
3.2 Tool 2: what the tickets know
- The ticket tool fine-tunes Llama-3-8B-Instruct with QLoRA on curated query–answer pairs and serves a quantized model locally; a second candidate retrieves ticket pairs for Llama 3.1.The fine-tuned model receives only the question, with follow-ups rewritten as standalone questions.Source: §3.2 · p. 3
3.3 Orchestration over MCP
- The orchestrator rewrites follow-ups, calls both tools concurrently for technical queries, and merges answers while preferring documentation on conflicts over concrete facts, commands, paths, or URLs.The architecture returns source links and directs unresolved users to the help desk; the interface also lets users compare the full system with either tool alone.Source: §3.3 · p. 4, Figure 1 · p. 4, Figure 2 · p. 5
4 Evaluation
On 400 held-out tickets, fine-tuning best matched staff resolutions, but the end-to-end merge did not improve on either tool. The tools were complementary, while judges disagreed about correctness and answer length complicated completeness scores.
4.1 Protocol
- The evaluation compares five systems on the same 400 tickets using reference metrics and two independent LLM judges; every answer was produced.Judges came from model families that generated none of the answers, and comparisons with Tool 2 used paired bootstrap intervals.Source: §4.1 · p. 5, Table 3 · p. 5
4.2 Results
- BERTScore F1 rises from 0.06 to 0.27 with fine-tuning over the base model, although the authors note that training and reference-answer registers align.Fine-tuned answers were 76 words on average versus 255 for the base model.Source: §4.2 · p. 5, Table 3 · p. 5
- Ticket-RAG trails fine-tuning on reference metrics and judged correctness, but is judged most complete; documentation retrieval answers a different question from ticket-based references.Ticket-RAG can produce long answers with inapplicable steps, while documentation may not cover account or allocation actions recorded in references.Source: §4.2 · pp. 5–6
- The merged agent does not improve on either tool; the per-question oracle reaches 3.41 judged correctness versus Tool 2's 3.20, but is only an upper bound.The oracle selects the answer judges rated more correct on each question, so its score is not an achievable system result.Source: §4.2 · p. 6, Table 4 · p. 6
- Neither the learned selector nor the LLM verifier beats Tool 2, while the oracle remains an optimistic upper bound.The learned selector found little usable signal; the verifier's decision varied with answer order.Source: §4.2 · pp. 6–7, Table 6 · p. 6
- The judges disagree about correctness: Gemma rates Tool 2 above other systems, while Qwen slightly favors the base model and finds no difference from either RAG system.Their correctness-score agreement is Spearman ρ=0.63.Source: Table 5 · p. 6, §4.2 · p. 7
- Answer length correlates with judged completeness (Spearman ρ=0.37), but not correctness (−0.04), so completeness scores should be read alongside correctness.Length correlates negatively with BERTScore (−0.40).Source: §4.2 · p. 7
5 Discussion and Limitations
The discussion recommends selecting rather than merging tool answers, while highlighting data-cleaning risks and limits in how the systems were evaluated. It also argues that judge disagreement and answer length require dimension-by-dimension reporting.
Data preparation
- Rewriting staff actions as self-service guidance and removing stale status updates addressed confidently false answers, but ticket-data cleaning still required iteration.Residual usernames remained after pseudonymisation and could be learned and repeated by a fine-tuned model.Source: §5 · p. 7
Discussion
- Fine-tuning beat ticket retrieval on reference metrics, but the merge tracked documentation and lost complementary strengths; selecting an answer recovered fine-tuned quality without beating it.The authors recommend returning the fine-tuned answer, with documentation alongside as a cited second opinion, and identify better selection signals as future work.Source: §5 · p. 7
Evaluation
- The judges disagreed about fine-tuning’s correctness benefit, and both rewarded length on completeness, motivating separate judge and dimension reporting.The authors report each judge and size claims to what holds under both.Source: §5 · p. 7
Limitations
- Evaluation relied on reference metrics and LLM judges that disagreed on the central correctness comparison and had not been validated against human ratings.The references were LLM-rewritten staff resolutions, which the authors note favors the fine-tuned model on reference metrics.Source: §5 · p. 7
- The test-label oracle is an upper bound, while selectors tuned on validation labels from the same judges may learn those judges’ preferences.The authors also note that the hosted model tags can change and that no frontier model was compared because ticket text could not leave the institution.Source: §5 · p. 7
6 Related Work
Prior HPC support work includes guide-based chatbots, domain-adapted retrieval systems, and automation that acts after tickets are filed. This paper distinguishes itself by using real ticket resolutions in a separate tool and by examining how documentation and ticket answers combine.
HPC LLM assistants
- AskHPC retrieves from user guides and manuals but not ticket history, while HPC-LLM [23] uses QLoRA and retrieval with public documentation and synthetic training pairs.AskHPC includes modality-aware parsing and a user study reporting gains over vanilla RAG.Source: §6 · p. 8
- Compared with Zheng et al., this work keeps documentation and ticket history in separate orchestrated tools and finds its simple merge rule does not combine their strengths.Zheng et al. compare fine-tuning on historical tickets, document RAG, and in-context examples across eight combinations on roughly 60,000 TACC tickets.Source: §6 · p. 8
Fine-tuning versus retrieval
- Ovadia et al. favor RAG for new factual knowledge, whereas Balaguer et al. [2] find fine-tuning and retrieval complementary; here, merging did not realize that complementarity.The paper reports that fine-tuning beat retrieval over the same tickets on agreement with staff resolutions.Source: §6 · p. 8
Agentic and tool-using LLMs
- The orchestrator uses one classification, two parallel tool calls, and one synthesis; the authors identify synthesis as the design stage needing most care.The Model Context Protocol standardizes the tool interface used in this work.Source: §6 · p. 8
Ticket automation
- Earlier ticket automation mainly handles classification, routing, escalation, or categorization after a ticket exists; this work intervenes before ticket filing.The cited literature includes classical-ML subtask automation and newer LLM applications.Source: §6 · p. 8
Documentation usability
- Studies report that inadequate documentation impedes learning and HPC users rely on facilitators and peers, but do not evaluate guides and ticket history together.Source: §6 · p. 8
7 Conclusion
Most HPC support answers already exist across current manuals and inaccessible ticket histories. Fine-tuning rewritten resolutions works well, but merging sources does not improve on the tools, leaving answer selection unresolved.
7 Conclusion
- Fine-tuning a small model on rewritten and filtered ticket resolutions agreed with RCAC’s resolutions far better than the base model or ticket retrieval on 400 held-out tickets.One of two judges also rated it clearly the most correct.Source: §7 · p. 8
- The documentation and ticket-history sources are complementary, but their documentation-first merge improved on neither tool.Selecting one answer recovers the fine-tuned tool’s quality without exceeding it; choosing the better source in advance remains open.Source: §7 · p. 8
- Rewriting staff actions into self-service guidance, filtering missed actions, and masking residual PII were key steps for a trustworthy assistant.Source: §7 · p. 8
- The conclusion identifies expert validation of the judges and better signals for choosing an answer source as next steps.Source: §7 · p. 8
- The underlying support knowledge is divided between current but generic manuals and specific but inaccessible ticket history.Source: §7 · p. 8