PaperlayerView exact revision on arXiv

Source-linked AI summary

From SQL to Knowledge Graphs: An LLM-Driven Multi-Agent Approach with Data Schema Improvement

Dinh-Khanh Pham, Quy-Anh Dang, Lam Mai Thanh, Khanh Bui, Truong-Son Hy

arXiv:2608.26117v1cs.DBcs.AI

TL;DR

The paper addresses limitations of relational databases and the underexplored use of LLMs for reducing ambiguity in relational-to-graph schema generation. It proposes an end-to-end multi-agent ETL and schema-refinement pipeline, and reports higher accuracy and lower latency for graph-based query answering on BFSI data.

  • Problem

    Relational databases face costly joins, rigid relational semantics, and slower handling of complex interconnected data, while effective LLM-generated graph schema design remains underexplored.

  • Method

    An LLM-powered multi-agent system standardizes structured data, maps tables and relationships into graph schemas, and iteratively scores designs for accuracy, groundedness, and faithfulness.

  • Results

    Cypher achieves 85.57% mean accuracy versus SQL’s 73.45%, with 68.8% faster average latency across all complexity levels on the BFSI dataset.

  • Takeaways & Limitations

    The results support graph-based approaches as a suitable foundation for LLM-driven query generation in domains with complex interconnected data.

Abstract

from arXiv · show

RDBMS (Relational Database Management System) databases face several limitations, including slow execution with multi-hop queries and a lack of explainability by graphical interpretations. In contrast, Graph database offers a more intuitive and efficient data schema that performs faster execution on large datasets. Most existing RDBMS conversion pipelines focus on running traditional loading commands and relying on Cypher queries. However, the efficiency of using an LLM to generate an effective graph data schema, significantly reducing the ambiguity of the graph database, remains underexplored in the current research literature. This paper presents a novel algorithm that bridges RDBMS and graph database by using a novel LLM-powered ETL agent to standardize table and column names before saving them to the Data Mart. A Multi-Agent System generates a looping discussion between ETL, Analyzer, and Graph agents to optimize the final design through an iterative process of suggesting and scoring the graph database schema. We ensure that the final graph database meets three criteria before being accepted for data conversion: Accuracy, Groundedness, and Faithfulness. This system demonstrates an effective pipeline to automatically convert a tabular database into a graph database through a comprehensive end-to-end process. Our study highlights notable efficiency in using the converted graph database, which is measured on 1,081 samples of the BFSI dataset across three levels of complexity (easy, medium, and hard). Specifically, CypherAgent achieves an 85.6% accuracy for Q&A tasks using a Graph database, which is 12.12% higher than the accuracy achieved by an SQLAgent on the RDBMS database type PostgreSQL, for all queries. Additionally, the Graph database demonstrates faster performance, reducing latency by approximately 3 times.

I. INTRODUCTION

The paper targets relational-to-graph conversion challenges by introducing an LLM-based multi-agent pipeline that standardizes data, iteratively improves graph schemas, and supports end-to-end ingestion.

  • Graph databases support interconnected-data workloads through semantic relationships and efficient traversal, whereas relational systems face costly joins and rigid relational semantics.
  • Existing relational-to-graph conversions rely mainly on loading commands that cannot interpret target-graph entity relationships, and no standardized solution is widely accepted.
  • The proposed system automatically transforms an unnormalized structured database into a normalized graph database using its structured schema knowledge.
  • The paper contributes relational–knowledge-graph integration and an end-to-end data-ingestion pipeline intended to improve accuracy and efficiency.
  • Its end-to-end multi-agent system assigns distinct roles to ETL, Analyzer, and Graph agents that collaboratively design graph nodes and relationships.
  • The pipeline iteratively evaluates and improves graph structures to make them more precise and aligned with domain knowledge.

II. RELATED WORK

The paper positions its approach against relational and graph database trade-offs, then describes an automated ETL-to-graph pipeline whose schema is iteratively generated and scored.

  • II. RELATED WORK: Relational databases use fixed table-based schemas, while knowledge graphs provide richer semantics, flexible hierarchies, and dynamic relations at the cost of more complex schema design.
  • II. RELATED WORK: The proposed architecture automates database lifecycle stages including ETL, standardization, transformation, and data mining.
  • II. RELATED WORK: Graph databases are presented as better suited to complex relational structures because interconnected nodes and edges facilitate semantic enrichment and graph traversal.
  • II. RELATED WORK: ETL standardizes table names, field names, and formats from SQL, CSV, and raw data before graph transformation.
  • II. RELATED WORK: Graph Agent maps tables to nodes, relationships to directed edges, and metadata to properties, then repeatedly tunes the schema until its score reaches the threshold.
  • II. RELATED WORK: Analyzer Agent scores candidate schemas for accuracy, groundedness, and faithfulness using checks tied to SQL representation, graph principles, and business logic.

B. Overall Architecture

The overall architecture transforms structured SQL data into a validated knowledge graph through specialized agents whose combined functionality covers the conversion process.

  • The system models total agent functionality as Atotal = AGraph ∪ AAnalyzer ∪ AETL and produces a validated GraphDB for querying and analytics.

1) System Prompt Pooling:

System Prompt Pooling coordinates specialized Graph, Analyzer, and ETL agents by assigning task-specific prompts and parameters.

  • The coordinator distributes prompt instructions from Ppool to Graph, Analyzer, and ETL agents for precise data-conversion execution.

IV. DATASET

The study evaluates CypherAgent and SQLAgent on 1,081 BFSI business queries spanning easy, medium, and hard complexity levels. Across these levels, CypherAgent achieves higher accuracy and lower latency than SQLAgent.

  • Dataset: The BFSI dataset contains 1,081 business queries divided into easy, medium, and hard levels.The partitions contain 512 easy, 333 medium, and 236 hard queries.
  • Evaluation setup: CypherAgent and SQLAgent answer the same business queries using graph and PostgreSQL databases, respectively.EvalAgent compares their answer accuracy and latency under matched inputs including schemas, relationships, and metadata errors.
  • Overall results: Cypher-based querying consistently outperforms SQL-based querying across evaluation metrics and query-complexity levels.The comparison covers accuracy and computational efficiency for easy, medium, and hard queries.
  • Easy queries: 90.43% accuracy is achieved by Cypher on easy queries versus SQL’s 72.66%.Cypher also reduces latency from 5.63 seconds to 1.73 seconds in this category.
  • Medium queries: 81.98% accuracy is achieved by Cypher on medium queries versus SQL’s 77.78%, with latency of 1.94 seconds versus 6.91 seconds.Medium queries involve moderate relationship traversals and multi-entity interactions.
  • Aggregate results: 85.57% mean accuracy is achieved by Cypher versus SQL’s 73.45% across all complexity levels.The average latency improvement is 4.39 seconds, corresponding to 68.8% faster performance.

VI. ABLATION STUDY

The ablation study examines meta-error correction and few-shot learning within the multi-agent graph-querying system. Both techniques improve effectiveness or efficiency over the baseline, while the full system outperforms the baseline in accuracy, token usage, and latency.

  • Ablation results: Meta Error improves accuracy by 28.84% and reduces token usage in the BFSI evaluation.It stores common syntax and table-or-column selection errors for inclusion in agent prompts.
  • Error handling: The baseline often produces Cypher syntax errors that require retry turns, increasing latency and token consumption.Meta-error prompting targets these recurring errors and increases the percentage of runnable code.
  • Overall ablation conclusion: The full system combining Meta Error correction with few-shot grounding outperforms the baseline in accuracy, token usage, and latency.The conclusion attributes the comparison to the combined system rather than either technique alone.
  • System contribution: The proposed multi-agent system converts structured databases into knowledge graphs while preserving critical fields and relationships across multiple tables.CypherAgent is reported to outperform SQLAgent for business-query efficiency and accuracy through graph traversal and enhanced relationship semantics.
  • Evaluation dimensions: Table I compares CypherAgent variants using meta error, few-shot learning, and correct error against SQLAgent baselines on accuracy, latency, and token count.The caption identifies the GPT-4o configuration with all three techniques as having the best overall accuracy.

APPENDIX

The appendix describes a closed, heuristic-driven multi-agent pipeline for converting relational schemas into graph databases. ETL, Graph, and Analyzer agents iteratively generate, evaluate, and refine graph schemas before Cypher-based querying.

  • Pipeline overview: The end-to-end system consists of ETL, graph generation, and question-answering stages.Specialized agents collaborate within a closed pipeline.
  • ETL: The ETL Agent extracts, transforms, and loads relational data while preserving semantic relationships, producing standardized tables and an initial graph schema.Its transformation maps DSQL into an initial graph representation ready for GraphDB loading.
  • Graph generation: The Graph Agent maps structured tables to nodes and relationships to edges, then repeatedly improves the graph schema before executing migration scripts.The schema description supports the data-ingestion step.
  • Schema analysis: The Analyzer Agent scores the graph database schema for trustworthiness and robustness while proposing a suitable schema and standardizing inconsistencies.It operates jointly with the Graph Agent in a unified refinement loop.
  • Motivation: Direct or rule-based relational-to-graph conversion often omits semantic validation and structural refinement.The resulting schemas may inadequately represent domain semantics and relationships.
  • Iterative refinement: Iterative schema review modifies entity types, attributes, and relationship semantics across multiple iterations toward a higher-quality representation.The process is intended to validate and progressively refine the graph rather than generate it in one pass.
  • Refinement criteria: The refinement criteria include consistency with relational constraints, graph modeling principles, and preservation of semantic and business logic.These criteria cover keys and dependencies, node granularity and redundancy, and downstream analytical support.
  • Refinement outcome: The refined graph is reported to improve structural accuracy, semantic expressiveness, practical utility, and downstream application support relative to the unmodified initial graph.The passage states that the process yields higher schema-level accuracy.

C. Data Conversion

The ETL engine converts PostgreSQL data into graph-ready data, while specialized agents generate, validate, and execute against a graph schema. The pipeline also formats retrieved graph results into accessible business answers and augments Cypher generation with examples and error corrections.

  • The ETL engine integrates PostgreSQL, a processing module, and a large language model to drive SQL-to-GraphDB conversion.
  • Data extraction exports PostgreSQL data into manageable formats such as CSV, and processing cleanses, normalizes, and transforms it to align with the graph schema.
  • The Graph Agent and processing module generate a graph schema represented by nodes, edges, and properties that reflect SQL data structure and semantics.
  • The Load stage imports transformed data into GraphDB, while the Analyzer Agent iteratively refines it until the graph reaches the acceptance threshold.
  • Cypher queries are executed against GraphDB to retrieve relevant nodes, edges, and properties, which the Cypher Agent formats into clear business-context answers.
  • Few-shot question–Cypher pairs and Meta Error knowledge are added to the Cypher Agent’s prompts to avoid common query errors and improve accuracy.

E. End-to-end Pipeline Orchestration

The end-to-end system first standardizes relational inputs and proposes an initial graph schema, then uses iterative Graph–Analyzer interaction to produce an accepted final database for question answering. The Cypher Agent uses the resulting schema together with augmented knowledge to generate answers.

  • End-to-end Pipeline Orchestration: The pipeline combines ETL, Analyzer, and Graph agents in a sequential end-to-end process for data conversion and answer generation.
  • ETL pipeline: The ETL Agent standardizes raw relational schemas, loads data into Graph Database, and proposes an initial graph schema for graph generation.
  • ETL pipeline: Raw relational databases are represented as DSQL, standardized relational schemas as DStandardSQL, and initial graph schemas as Ginit.
  • Graph Gen: The Graph Agent and Analyzer Agent repeatedly interact to finalize the standard graph database through an iterative graph-generation loop.
  • Graph Gen: The loop selects the first iteration whose score reaches the acceptance threshold τ, and otherwise uses feedback to generate a new graph schema.
  • Generate Answer: The Cypher Agent generates answers from the final graph database using the graph schema, few-shot examples, and Meta Error corrections.

F. SQL Database Dataset

The BFSI benchmark uses a banking-sector relational database with interconnected tables and 1,081 business queries spanning three difficulty levels. Its schema is repeatedly evaluated and modified to improve graph-database clarity.

  • SQL Database Dataset: The SQL database contains small, medium, and large versions with 10^3, 10^5, and 10^6 records, sharing nine banking-sector tables.
  • SQL Database Dataset: The nine tables cover customers, branches, deposits, deposit types, loans, loan types, credit cards, accounts, and transactions.
  • SQL Database Dataset: Foreign keys connect table identifiers across the relational model, including one-to-many relationships from customers to accounts and accounts to transactions.
  • Schema improvement: The Knowledge Graph Scoring algorithm repeatedly evaluates and modifies the input SQL schema to meet the graph-quality threshold.
  • Benchmark queries: Figure 2 presents the end-to-end comparison between a Cypher-based Knowledge Graph agent and an SQL-based relational-database agent.

G. Graphical Database

The transformed Neo4j database represents SQL tables as graph nodes and their relationships as semantically named edges, improving interpretability and preserving the original data structure. Its performance is compared with SQL across hard, medium, and easy business queries, while graph quality is screened using three criteria.

  • Graphical Database: The Graph Agent transforms relational data into a Knowledge Graph with semantically meaningful relationships and additional graph-oriented features.
  • Graphical Database: Neo4j represents the nine SQL tables as nodes with properties and connects them through relationship edges such as “HAS LOAN TYPE” and “BELONGS TO CUSTOMERS.”
  • Graph quality evaluation: Graph quality is scored on Accuracy, Groundedness, and Faithfulness, and only designs receiving the maximum score of 5 on all three metrics are accepted.
  • Schema improvement: Iterative GraphAgent suggestions change node and relationship labels to match business logic, making the Neo4j schema clearer than the original unstandardized SQL schema.
  • Graphical Database: The Neo4j dataset preserves original SQL data integrity, enriches graph elements with natural-language explanations, and supports visual interpretation of complex relationships.
  • Query performance: Queries in Neo4j are reported as faster than SQL for hard, medium, and easy business-query levels on the same benchmarking dataset.

H. Experimental Setup

The evaluation uses a banking-domain question set spanning three query-complexity levels to compare SQLAgent and CypherAgent. It measures accuracy and latency while assessing schema standardization, graph design, query-generation support, and execution performance.

  • Dataset: The banking-domain evaluation comprises 1,089 questions covering transaction analysis, account management, customer relationships, and financial-product queries.Questions are distributed across hard, medium, and easy levels at 47.36%, 30.80%, and 21.83%, respectively.
  • Query complexity: Easy queries use single-entity operations and basic filters, while medium and complex queries require relationship traversals, joins, nested conditions, or aggregates.Medium queries involve two to three relationship traversals across multiple entities.
  • Query generation: CypherAgent uses few-shot examples, common error patterns, and graph-schema relationships to generate more accurate Cypher syntax and relationship directions.The prompt also incorporates top-k similar examples to reinforce business-logic understanding.
  • Schema design: The schema workflow contrasts an unstandardized BFSI relational schema with a standardized schema scored for accuracy, groundedness, and faithfulness, while GraphAgent represents table relationships as graph edges.The graphical design links multiple entities to simulate relationships in the original SQL tables.
  • Metrics: Accuracy counts responses as correct when their numeric values match the ground truth, regardless of syntactic variation.The formal score compares each system response with the corresponding ground truth through a matching function.
  • Comparative evaluation: The evaluation compares SQLAgent-generated SQL with GraphAgent-generated Cypher using an Evaluation Agent that measures accuracy and performance across easy, medium, and hard queries.Execution-time comparisons attribute CypherAgent’s speed advantage to an optimized schema and fewer request rounds than SQLAgent’s ReAct process.
Loading 2608.26117v1…