Source-linked AI summary
Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering
Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, Zheng Li
TL;DR
Customer-service RAG systems commonly treat issue tickets as plain text, losing intra-issue structure and inter-issue relations needed for retrieval. This paper builds a knowledge graph over historical tickets, parses queries, and retrieves related subgraphs for answer generation. The method improves benchmark retrieval and question-answering metrics and reduces median production resolution time by 28.6%.
Problem
Conventional RAG retrieval treats historical issue tickets as plain text, overlooking intra-issue structure and inter-issue relations.
Method
The method constructs a knowledge graph from historical tickets, preserves tree and relational structure, parses queries, and retrieves related subgraphs for answer generation.
Results
The method surpasses the baseline by 77.6% in MRR and by 0.32 in BLEU, while production use reduced median resolution time per issue by 28.6%.
Takeaways & Limitations
Integrating RAG with a knowledge graph improved retrieval, answering metrics, and overall customer-service effectiveness in the reported evaluation and deployment.
Abstract
from arXiv · showhide
In customer service technical support, swiftly and accurately retrieving relevant past issues is critical for efficiently resolving customer inquiries. The conventional retrieval methods in retrieval-augmented generation (RAG) for large language models (LLMs) treat a large corpus of past issue tracking tickets as plain text, ignoring the crucial intra-issue structure and inter-issue relations, which limits performance. We introduce a novel customer service question-answering method that amalgamates RAG with a knowledge graph (KG). Our method constructs a KG from historical issues for use in retrieval, retaining the intra-issue structure and inter-issue relations. During the question-answering phase, our method parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration of a KG not only improves retrieval accuracy by preserving customer service structure information but also enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on our benchmark datasets, utilizing key retrieval (MRR, Recall@K, NDCG@K) and text generation (BLEU, ROUGE, METEOR) metrics, reveal that our method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. Our method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%.
1 INTRODUCTION
Customer-service support depends on rapidly retrieving relevant past issues, but conventional text-based retrieval loses issue structure and connections. The proposed graph-based approach preserves these relationships and avoids answer incompleteness caused by segmentation.
- Rapidly retrieving similar past issues is crucial for efficiently resolving customer inquiries.Customer inquiries frequently resemble previously resolved issues, making accurate retrieval important for technical support.
- Conventional text-chunk retrieval ignores issue structure and inter-issue relations, losing vital information.Issue tickets contain inherent structure and links such as related-to, copied-from, or caused-by relationships.
- Graph-based parsing represents tickets as trees and connects them into an interconnected graph that maintains relationships among entities.The approach is designed to retain both within-ticket organization and links across tickets.
- Text segmentation can separate an issue description from its solution, producing incomplete answers.The graph-based method preserves the logical coherence of ticket sections instead of relying solely on fixed-length segments.
2 RELATED WORK
Prior knowledge-graph question-answering methods include retrieval-based, template-based, and semantic-parsing approaches. Recent work also integrates knowledge graphs with LLMs for reasoning and question answering.
- Knowledge-graph question answering is broadly divided into retrieval-based, template-based, and semantic-parsing methods.These categories differ in how they derive answers, encode queries, or map text to logical forms.
- Retrieval-based methods use relation extraction or distributed representations but struggle with questions involving multiple entities.
- Template-based methods encode complex queries manually but are limited by the scope of available templates.
- Semantic-parsing methods map text to logical forms containing predicates from knowledge graphs.
- Recent research integrates knowledge graphs with LLMs as predictors, encoders, aligners, and components of graph-based reasoning and question answering.Examples include Think-on-Graph, Reasoning-on-Graph, Mindmap, and related approaches.
3 METHODS
The paper integrates retrieval-augmented generation with a knowledge graph built from historical customer-service tickets. It parses queries and navigates related graph subgraphs to generate answers.
- The system constructs a knowledge graph from historical tickets using tree-structured issue representations and relational links between issues.Each node also receives an embedding to support later semantic search.
- During question answering, the method parses consumer queries to identify named entities and intents.
- The system navigates the knowledge graph to identify related subgraphs for answer generation.
3.1 Knowledge Graph Construction
Knowledge-graph construction separates intra-ticket trees from inter-ticket connections. It combines explicit ticket links with semantic similarity and supports embedding-based retrieval over graph nodes.
- 3.1 Knowledge Graph Construction: Each ticket is modeled as an intra-issue tree whose nodes represent ticket sections and whose edges encode hierarchical and relational structure.
- 3.1 Knowledge Graph Construction: The inter-issue graph combines explicit ticket links with implicit connections derived from semantic similarity.Explicit links come from issue-tracking records, while implicit links connect semantically similar tickets.
- 3.1 Knowledge Graph Construction: Graph construction has separate intra-ticket parsing and inter-ticket connection phases.Parsing uses rules for predefined fields and an LLM guided by a YAML graph template for other text.
- 3.1 Knowledge Graph Construction: Node embeddings support online semantic retrieval and are stored in a vector database.Embeddings target text-rich sections such as issue summaries, descriptions, and reproduction steps.
- 3.1 Knowledge Graph Construction: Figure 1 contrasts knowledge-graph construction on the left with retrieval and question answering on the right.The example includes ticket nodes and their connections, including a direct clone link and implicit semantic links.
3.2 Retrieval and Question Answering
The method parses user queries into named entities and intents, identifies relevant tickets through embedding-based retrieval, and extracts answer-relevant subgraphs for LLM response generation.
- Query Entity Identification and Intent Detection.: The system parses each query into named-entity key-value pairs and an intent set using an LLM, graph template, and prompt.
- Embedding-based Retrieval of Sub-graphs.: It identifies the top K relevant historical tickets by comparing query entity values with graph nodes in matching sections using pretrained text embeddings.Node-level cosine-similarity scores are summed across nodes belonging to the same ticket.
- LLM-driven Subgraph Extraction.: The system reformulates the query with retrieved ticket IDs and translates it into a graph-database query for subgraph extraction.The approach uses languages such as Cypher for Neo4j and can retrieve information across subgraphs from the same or distinct trees.
- Answer Generation.: An LLM synthesizes responses from retrieved graph data and the initial query, with text-based retrieval serving as a fallback when query execution fails.
4 EXPERIMENT
The experiment compares the knowledge-graph method with conventional text-based embedding retrieval using the same GPT-4 and E5 models. Across retrieval and question-answering metrics, the proposed method consistently improves performance.
- Experimental Setup: The evaluation uses a curated golden dataset containing typical queries, support tickets, and authoritative solutions, comparing conventional text-based EBR with the proposed method.Both groups use GPT-4 and E5; retrieval is measured with MRR, recall@K, and NDCG@K.
- Result and Analysis: Across all reported retrieval and question-answering metrics, the proposed method demonstrates consistent improvements over the baseline.Retrieval and question-answering results are presented in Tables 1 and 2.
5 PRODUCTION USE CASE
The method was deployed within LinkedIn’s customer service team across multiple product lines and compared with traditional manual methods. The system-using group achieved a lower median issue-resolution time.
- Production Deployment: The deployment covered multiple LinkedIn product lines and randomly divided the customer service team between the proposed system and traditional manual methods.
- Production Outcome: 28.6% lower median resolution time per issue was achieved by the group using the proposed system.The result is reported in Table 3.
6 CONCLUSIONS AND FUTURE WORK
The paper concludes that integrating RAG with a knowledge graph improves customer-service retrieval, answering metrics, and overall service effectiveness. Future work targets graph-template extraction, dynamic graph updates, and use beyond customer service.
- Conclusion: The authors conclude that integrating retrieval-augmented generation with a knowledge graph improves retrieval, answering metrics, and overall service effectiveness.
- Future Work: Future work will automate graph-template extraction, investigate query-driven dynamic knowledge-graph updates, and explore applications beyond customer service.
7 COMPANY PORTRAIT
LinkedIn is a global professional network founded in 2003, with more than 1 billion members worldwide and a diversified business model.
- LinkedIn connects professionals worldwide to make them more productive and successful.
- Founded in 2003, LinkedIn has more than 1 billion members worldwide.
- LinkedIn generates revenue through Talent Solutions, Marketing Solutions, Sales Solutions, and Premium Subscriptions.
8 PRESENTER BIO
Zhentao Xu is a Senior Software Engineer at LinkedIn whose research focuses on large language models and natural language generation.
- Zhentao Xu is a Senior Software Engineer at LinkedIn.
- He earned an M.S. in Robotics and a B.S. in Electrical Engineering and Computer Science from the University of Michigan.
- His research interests include large language models and natural language generation.