Source-linked AI summary
HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction
Bhaskarjit Sarmah, Benika Hall, Rohan Rao, Sunil Patel, Stefano Pasquali, Dhagash Mehta
TL;DR
Financial documents pose extraction challenges for LLMs because specialized terminology, complex formats, and inconsistent retrieved context limit existing RAG approaches. The paper introduces HybridRAG, combining vector-based and knowledge-graph retrieval, and reports stronger overall performance than either approach alone, with a context-precision trade-off.
Problem
Existing LLM and RAG approaches face challenges extracting consistent, accurate information from specialized and intricate financial documents.
Method
HybridRAG combines VectorRAG and GraphRAG to retrieve external context from both vector databases and knowledge graphs for financial-document queries.
Results
HybridRAG outperforms VectorRAG and GraphRAG on key evaluation dimensions, excelling in faithfulness, answer relevancy, and context recall despite lower context precision.
Takeaways & Limitations
The approach supports more effective analysis of complex financial documents and may extend to AI-assisted financial decision-making tools.
Takeaways & Limitations
Future work must extend HybridRAG to multimodal inputs, numerical analysis, richer evaluation metrics, and real-time financial data streams.
Abstract
from arXiv · showhide
Extraction and interpretation of intricate information from unstructured text data arising in financial applications, such as earnings call transcripts, present substantial challenges to large language models (LLMs) even using the current best practices to use Retrieval Augmented Generation (RAG) (referred to as VectorRAG techniques which utilize vector databases for information retrieval) due to challenges such as domain specific terminology and complex formats of the documents. We introduce a novel approach based on a combination, called HybridRAG, of the Knowledge Graphs (KGs) based RAG techniques (called GraphRAG) and VectorRAG techniques to enhance question-answer (Q&A) systems for information extraction from financial documents that is shown to be capable of generating accurate and contextually relevant answers. Using experiments on a set of financial earning call transcripts documents which come in the form of Q&A format, and hence provide a natural set of pairs of ground-truth Q&As, we show that HybridRAG which retrieves context from both vector database and KG outperforms both traditional VectorRAG and GraphRAG individually when evaluated at both the retrieval and generation stages in terms of retrieval accuracy and answer generation. The proposed technique has applications beyond the financial domain
1 INTRODUCTION
Financial documents contain valuable information but are difficult for general-purpose LLMs to interpret because of domain-specific language, varied formats, and complex contextual relationships. HybridRAG combines VectorRAG and GraphRAG to improve information retrieval and answer generation for financial Q&A.
- Financial analysts need to extract information from unstructured news, earnings reports, and other documents relevant to investment decisions and market analysis.
- Pre-trained LLMs struggle with financial documents outside their training data because terminology, formats, and contextual relationships vary across documents.
- VectorRAG retrieves relevant textual information from vector databases to provide context for language-model generation.
- GraphRAG uses knowledge graphs to support context-aware Q&A but generally underperforms on abstractive questions or queries without explicit entities.
- HybridRAG combines VectorRAG and GraphRAG to retrieve relevant external information and generate more accurate answers.
- The study presents a hybrid approach and a ground-truth Q&A dataset derived from financial call transcripts of Nifty-50 companies.
2 METHODOLOGY
The methodology section outlines the study’s progression from VectorRAG, through knowledge-graph construction and GraphRAG, to the proposed HybridGraph technique.
- The methodology first discusses VectorRAG, then knowledge-graph construction, GraphRAG, and finally the HybridGraph technique.
2.1 VectorRAG
VectorRAG retrieves semantically relevant document chunks from a vector database and supplies them to a generative model alongside the original query. This combines retrieved external information with the model’s existing knowledge.
- Traditional RAG searches an external repository for passages relevant to a query and feeds them to the LLM as additional context.
- VectorRAG divides documents into chunks, converts them into embeddings, stores them in a vector database, and ranks them by similarity to the query.
- The generative model combines the retrieved context with the original query to synthesize a contextually relevant response.
- Figure 1 depicts the vector-database creation stage of the VectorRAG process, including explicit metadata references.
2.2 Knowledge Graph Construction
Knowledge-graph construction represents document information as entities and relationships, then extracts and refines graph triplets from unstructured text. The paper uses prompt-engineered LLM chains to support this process.
- A knowledge graph represents entities, attributes, and relations as graph nodes, edges, labels, and attributes; triplets encode subject–predicate–object facts.
- Knowledge-graph construction commonly involves knowledge extraction, knowledge improvement, and knowledge adaptation, although this study treats its graphs as static.
- Knowledge improvement addresses redundancies and missing information, while knowledge fusion reconciles conflicts across multiple sources.
- The proposed pipeline creates knowledge-graph triplets from corporate earnings-call transcripts through interconnected extraction and refinement stages.
- A two-tiered LLM chain first abstracts each document chunk and then extracts entities and relationships for triplet construction.
- Both stages use prompt engineering on a pre-trained LLM, with implementation details deferred to a later section.
2.3 GraphRAG
GraphRAG retrieves entities and relationships from a knowledge graph to form a subgraph that supplies structured context for language-model responses. HybridRAG combines this graph-derived context with VectorRAG’s similarity-based retrieval.
- 2.3 GraphRAG: GraphRAG searches a knowledge graph for query-relevant entities and relationships, extracts a subgraph, and integrates it with the language model.The subgraph provides graph-structured context for response generation.
- 2.3 GraphRAG: GraphRAG uses prompts first to clean data and then to create knowledge triplets with metadata.
- 2.4 HybridRAG: HybridRAG combines VectorRAG’s broad similarity-based retrieval with GraphRAG’s structured, relationship-rich context before generating final responses.The combined contexts are provided as input to a language model.
2.4 Evaluation Metrics
The evaluation framework separately assesses retrieval context and generated answers using faithfulness, answer relevance, context precision, and context recall. These metrics compare generated content with its context, question, or ground-truth answer.
- Evaluation framework: The study compares VectorRAG, GraphRAG, and HybridRAG responses to assess response quality, accuracy, and comprehensiveness.
- Faithfulness: Faithfulness measures the extent to which generated answers can be inferred from the provided context.The score is F = |V|/|S|, where supported statements are divided by total statements.
- Answer relevance: Answer relevance measures how well a generated answer addresses the original question regardless of factual accuracy.The implementation generates questions from the answer and averages their embedding similarity to the original question.
- Context precision: Context precision measures the proportion of relevant retrieved items appearing at top ranks relative to ground truth.
- Context recall: Context recall measures the proportion of ground-truth answer sentences attributable to the retrieved context.Values range from 0 to 1.
3 DATA DESCRIPTION
The study uses a purpose-built dataset because existing public financial datasets did not provide the required combination of documents, ground-truth Q&A pairs, and knowledge-graph context. It comprises Nifty 50 earnings-call transcripts from Q1 of India’s financial year 2024.
- Dataset motivation: Existing public financial datasets were unsuitable because they targeted specific use cases, lacked source documents, or were not publicly available.
- Dataset motivation: No publicly available benchmark dataset compared VectorRAG and GraphRAG in financial or general domains, motivating construction of a new dataset.
- Source documents: The dataset contains earnings-call transcripts from Nifty 50 constituents for the quarter ending June 2023, corresponding to Q1 of India’s financial year 2024.
- Source documents: The 50 transcripts span companies across sectors including infrastructure, healthcare, banking, automobiles, energy, telecommunications, pharmaceuticals, and information technology.
- Ground-truth Q&A: Earnings-call transcripts naturally contain analyst-management question-answer pairs, from which 400 questions and their exact responses were selected.
4 IMPLEMENTATION DETAILS
The implementation preprocesses earnings-call PDFs, extracts structured knowledge triplets, and builds separate vector- and graph-based retrieval pipelines. It evaluates generated answers from VectorRAG, GraphRAG, and their combined HybridRAG configuration using structured Q&A outputs.
- Preprocessing: PDF transcripts are loaded with PyPDFLoader and split into 2024-character chunks with 204-character overlap.The overlap preserves context across chunk boundaries.
- Knowledge Graph Construction: Knowledge-graph extraction uses a two-tiered language-model chain and represents each triplet as [’h’, ’type’, ’r’, ’o’, ’type’, ’metadata’].The fields encode head and object entities, entity categories, relationships, and contextual metadata.
- Knowledge Graph Construction: The extracted triplets are aggregated across document chunks into a document-level knowledge graph and persisted as Python data structures in a pickle file.
- VectorRAG: VectorRAG embeds transcript text with OpenAI’s text-embedding-ada-002 model and stores the representations in a Pinecone vector database for semantic similarity search.
- VectorRAG: The Q&A pipeline retrieves relevant chunks, formats them into context, and uses GPT-3.5-turbo to generate responses.Retrieval can be filtered by company name, quarter, and year.
- Evaluation: Each system processes the Q&A dataset, compiles questions, generated answers, retrieved contexts, and ground truths, then computes faithfulness, answer relevance, context precision, and context recall.
- GraphRAG: GraphRAG constructs a NetworkxEntityGraph from subject-predicate-object triplets and uses GraphQAChain with GPT-3.5-turbo to traverse the graph and generate answers.Relevant information is extracted with a depth-first search constrained to depth one from the specified entity.
5 RESULTS
The study evaluates VectorRAG, GraphRAG, and HybridRAG across retrieval and generation metrics. HybridRAG generally provides the strongest overall balance, although combining contexts reduces context precision.
- Comparative performance: 0.96 faithfulness is achieved by both GraphRAG and HybridRAG, exceeding VectorRAG’s 0.94.
- Comparative performance: 0.96 answer relevancy makes HybridRAG the leading pipeline, ahead of VectorRAG at 0.91 and GraphRAG at 0.89.
- Comparative performance: 0.96 context precision makes GraphRAG highest on this metric, while HybridRAG scores 0.79 and VectorRAG scores 0.84.The lower HybridRAG score is attributed to additional combined context that may not align precisely with ground truth.
- Comparative performance: 1 context recall is achieved by both VectorRAG and HybridRAG, compared with 0.85 for GraphRAG.
- Comparative performance: HybridRAG achieves the strongest overall balance across the evaluated RAG pipelines, particularly in faithfulness, answer relevancy, and context recall.Its combined retrieval strategy also introduces a trade-off in context precision.
- Question-type behavior: GraphRAG performs better on extractive questions, VectorRAG on abstractive questions, and HybridRAG combines these complementary strengths.The reported fallback behavior uses GraphRAG when VectorRAG fails on extractive questions and VectorRAG when GraphRAG fails on abstractive questions.
6 CONCLUSION AND FUTURE DIRECTIONS
The paper presents HybridRAG as a system for extracting and interpreting complex information from unstructured financial documents. It reports advantages over vector-only and KG-based RAG while identifying broader evaluation and modality extensions as future directions.
- Conclusion: HybridRAG combines vector-based RAG, knowledge graphs, and language models to improve information extraction from complex unstructured financial texts.
- Conclusion: HybridRAG shows superior retrieval accuracy and answer generation compared with its vector-based and KG-based counterparts.
- Conclusion: HybridRAG excels in faithfulness, answer relevancy, and context recall despite potential trade-offs in context precision.
- Implications: The system could support more sophisticated AI-assisted financial decision-making tools and broader access to financial insights.The stated implications extend beyond financial analysis, although the passage frames these outcomes as potential applications.
- Future directions: Future work includes multimodal inputs, numerical-data analysis, more nuanced evaluation metrics, and integration with real-time financial data streams.