PaperlayerView exact revision on arXiv

Source-linked AI summary

Schema-Free Text-to-SQL: Learning Enterprise Database Structure from Query Logs

Ömer Faruk Akgül, Nikhil Kapahi, Prajit Reddy Muppidi

arXiv:2610.33946v1cs.CL

TL;DR

Enterprise text-to-SQL methods assume complete schema documentation, which is often absent in databases with cryptic names and implicit relationships. This paper builds a knowledge graph from historical query logs and uses entity-driven GraphRAG retrieval. It reports higher Beaver execution validity than embedding-based retrieval and expert preference in an industrial evaluation.

  • Problem

    Enterprise text-to-SQL is difficult to formulate from documented schemas because those schemas are often unavailable and database names and relationships can be ambiguous.

    Source: §1 · p. 1

  • Method

    The framework mines historical query logs into a knowledge graph and uses domain-entity matching and graph traversal to retrieve relevant tables and join patterns.

    Source: §1 · p. 2

  • Results

    95.7% vs. 60.8% execution validity on Beaver; the framework also substantially outperforms embedding-based baselines.

    Source: §1 · p. 2, §7 · p. 6

  • Takeaways & Limitations

    The results demonstrate effective text-to-SQL in schema-ambiguous enterprise environments.

    Source: §7 · p. 6

    The BEAVER adaptation uses schema-derived rather than log-derived edges, which lack frequency information and may include relationships absent from actual usage patterns.

    Source: §8 · p. 7

Abstract

from arXiv · show

Text-to-SQL systems achieve remarkable accuracy on academic benchmarks, yet performance drops dramatically in enterprise deployments. This gap stems from a fundamental assumption: existing approaches require complete schema documentation, but enterprise databases typically lack comprehensive documentation, featuring cryptic naming and implicit relationships. We present the first text-to-SQL framework that operates without schema documentation, instead learning database structure from historical query logs. Query logs encode implicit structural knowledge: which tables are queried together, how they join, and what columns map to business concepts. We construct a knowledge graph from parsed query patterns and introduce entity-driven GraphRAG retrieval that grounds user questions in domain terminology rather than relying solely on embedding similarity. Validated on both an industrial deployment with expert evaluation and the Beaver benchmark, our approach substantially outperforms similarity-based retrieval methods, the dominant industry approach, in both executability and answer quality. On Beaver, we achieve 95.7% execution validity compared to 60.8% for embedding-based retrieval, with expert data engineers preferring our system 70% to 10% on internal deployment. Our work demonstrates that text-to-SQL can operate effectively in schema-ambiguous enterprise environments, enabling natural language database interfaces "in the wild".

1 Introduction

Enterprise text-to-SQL faces schema ambiguity: systems assume complete, interpretable schemas, while enterprise databases often have cryptic names and implicit joins. The paper uses historical query logs to build structure and entity-driven retrieval, reporting gains on Beaver and an industrial deployment.

1 Introduction

  • Enterprise systems often lack meaningful table and column descriptions, explicit foreign keys, and current documentation across schemas that may span hundreds to thousands of tables.The paper describes these conditions as common rather than edge cases.Source: §1 · p. 1
  • Historical query logs encode co-queried tables, practical joins, and mappings between columns and business concepts, providing structural knowledge without manual documentation.The paper proposes learning this knowledge from observed usage patterns.Source: §1 · p. 2
  • Entity-driven GraphRAG extracts domain entities from questions and traverses a query-log knowledge graph to find relevant tables and structurally coherent joins.This addresses vocabulary mismatches such as business terms that do not resemble cryptic schema identifiers.Source: §1 · p. 2
  • Because open-ended questions can have multiple valid SQL formulations, the evaluation combines execution metrics, expert-validated blind LLM comparisons, and rubric-based SQL analysis.The LLM judge achieved 83% agreement with expert data engineers.Source: §1 · p. 2
  • 95.7% vs. 60.8% execution validity on Beaver, with validation on an industrial deployment of 240 tables and 70% expert preference.The introduction attributes the results to the proposed approach and reports ablations identifying entity matching as the key contributor.Source: §1 · p. 2
  • The paper concludes that text-to-SQL can operate effectively in schema-ambiguous enterprise environments, supporting natural-language database interfaces in the wild.It also reports that generated SQL quality scores match or exceed human-written gold standards on open-ended questions.Source: §1 · p. 2

2 Related Work

Prior text-to-SQL methods achieve strong benchmark results but generally assume complete schema documentation. Retrieval and graph methods address parts of the problem, but the paper identifies query-log mining for semantic structure as its distinction.

2 Related Work

  • In-context learning and fine-tuning methods, including CHASE-SQL and MCS-SQL, reach near-perfect benchmark accuracy but assume complete schema documentation.Enterprise benchmarks including Spider 2.0 and BEAVER show degradation under realistic conditions; RubikSQL highlights the need for domain knowledge in industrial settings.Source: §2 · p. 2
  • Embedding-based retrieval narrows candidate schema elements but treats them independently, while graph methods model structure without leveraging query logs.The passage cites embedding approaches (Eben et al., 2025), relational encodings (Wang et al., 2021; Cao et al., 2021), and Graph RAG (Edge et al., 2025).Source: §2 · p. 2
  • The paper adapts database workload analysis to mine semantic rather than physical structure from logs.This contrasts the paper’s focus with workload analysis of physical database structure.Source: §2 · pp. 2–3

3 Task Formulation

Traditional text-to-SQL generates a query from a natural-language question and a documented schema. This paper instead assumes historical query logs, builds a knowledge graph, and retrieves context to support query generation.

3 Task Formulation

  • Traditional text-to-SQL takes a natural-language question and a complete schema of tables, columns, and foreign-key relationships as inputs to generate SQL.The formulation presupposes that the schema is fully documented and provided.Source: §3 · p. 3
  • In the paper’s setting, schema documentation is unavailable and historical query logs are the available database evidence.Source: §3 · p. 3
  • The proposed process constructs a knowledge graph from logs, retrieves relevant context for each question, and generates SQL from that context.The graph encodes table metadata, associated domain entities, and join relationships weighted by their frequency in historical queries.Source: §3 · p. 3
  • The log-based formulation includes the documented-schema setting, while logs additionally reveal practical joins, co-occurring columns, and mappings from domain terminology to schema elements.The paper states that schema documentation alone cannot recover this implicit usage knowledge.Source: §3 · p. 3

4 Method

The framework builds a knowledge graph from historical query logs, then uses entity-driven retrieval to assemble context for SQL generation. Its implementation combines metadata generation, graph traversal, and query examples to ground generated SQL.

4 Method

  • The method separates offline graph construction from online retrieval and SQL generation for each question.The three stages are knowledge graph construction, entity-driven retrieval, and SQL generation.Source: §4 · p. 3, Figure 2 · p. 4

4.1 Knowledge Graph Construction

  • The framework constructs a graph from query logs, annotating table nodes with generated metadata and business entities and weighting join edges by historical frequency.Construction includes structural extraction, metadata generation, and entity annotation.Source: §4.1 · p. 3

4.2 Entity-Driven Retrieval

  • Entity-driven retrieval matches question entities to table entities, then traverses and ranks join paths to find relevant tables, including indirect bridge tables.Personalized PageRank identifies tables that serve structural roles even when they do not directly match question entities.Source: §4.2 · pp. 3–4

4.3 Context Assembly and SQL Generation

  • The retrieved context combines table metadata, join paths, and similar historical queries, which an LLM uses to generate SQL grounded in the provided schema elements.The generation prompt instructs the model to follow identified join paths and avoid hallucinating schema elements.Source: §4.3 · p. 4

4.4 Implementation

  • The enterprise implementation processes logs spanning 240 tables, uses Claude 3.7 Sonnet for metadata and entity extraction, and limits graph traversal to k = 5 hops.Entity matching combines exact match (α = 0.5), substring match (β = 0.3), and semantic similarity (γ = 0.2) using E5-base-v2.Source: §4.4 · p. 5

5 Evaluation Framework

The evaluation framework addresses underspecified enterprise questions with execution-based metrics, pairwise LLM judging, and rubric-based scoring. It validates automated judgments against data-engineer annotations and compares against embedding-based RAG under the same constraints.

5 Evaluation Framework

  • The evaluation combines execution validity, blinded LLM pairwise comparisons, and rubric-based scoring to assess SQL quality across multiple dimensions.Pairwise judging covers executability, schema validity, filter correctness, join relevance, and result fidelity.Source: §5 · p. 5
  • Two senior data engineers' annotations agree with the automated evaluation on 83% of cases with clear human preference.The evaluation uses randomized A/B assignment for pairwise comparisons.Source: §5 · p. 5
  • The study compares against embedding-based RAG under identical constraints because schema-dependent text-to-SQL systems require unavailable complete schemas.The authors also note that reconstructed documentation does not resolve those systems' difficulty with historical usage patterns.Source: §5 · p. 5

6 Experiments

Across an internal deployment and BEAVER, the evaluation compares the system with retrieval-augmented generation (RAG), expert judgments, and gold SQL. The reported results favor the approach on execution validity and expert preference, while showing that gold SQL is not always the best reference for open-ended questions.

6.1 Experimental Setup

  • The evaluation used 35 questions on an Amazon database with 240 frequently queried tables and about 500K historical queries, plus BEAVER’s 209 questions across 463 tables.The internal deployment used 1,847 join edges; the BEAVER adaptation produced 1,310 graph edges.Source: §6.1 · p. 5

6.2 Main Results

  • 95.7% execution validity versus 60.8% for RAG on BEAVER, a 35-point improvement.The authors attribute the result to knowledge-graph structure producing coherent table-column combinations.Source: §6.2 · p. 5, Table 1 · p. 5
  • On BEAVER, blind LLM judges preferred the system in 45.5% of cases versus 34.0% for RAG; rubric judges preferred it 47.8% versus 30.6%.Rubric-dimension wins were largest for executability (+74 vs +1) and schema validity (+60 vs +17).Source: §6.2 · pp. 5–6, Table 2 · p. 6

6.3 Human Validation

  • 70% versus 10% of Amazon cases favored the system among data engineers; LLM judges preferred it in 80% versus 16%.On cases with clear human preferences, LLM judgments agreed in 83% of cases.Source: §6.3 · p. 6, Table 3 · p. 6

6.4 Rethinking Gold Standards

  • On open-ended BEAVER questions, generated SQL scored 9.00 versus 8.85 for gold, while gold dominated on deterministic questions.Judges found the generated SQL adequate or better in 46% of open-ended cases.Source: §6.4 · p. 6, Table 4 · p. 6

6.5 Ablation Studies

  • Removing entity matching dropped scores by 5.4% and lost 43% of pairwise comparisons, the largest ablation impact.KG traversal contributed 2.1%; PPR ranking provided modest gains, with 60.8% ties.Source: §6.5 · p. 6, Table 5 · p. 6

7 Conclusion

The paper presents a text-to-SQL framework that mines query logs to build a knowledge graph and uses entity-driven GraphRAG retrieval. It reports that the approach works without schema documentation and concludes that evaluation should go beyond single-gold matching.

7 Conclusion

  • The framework operates without schema documentation by mining query logs to build a knowledge graph of table relationships and domain entities.Entity-driven GraphRAG retrieval connects business terminology with cryptic schema identifiers.Source: §7 · p. 6
  • The approach substantially outperforms embedding-based baselines in the reported results.The conclusion describes effective text-to-SQL in schema-ambiguous enterprise environments.Source: §7 · p. 6
  • The authors highlight evaluation beyond single-gold matching as necessary for schema-ambiguous enterprise text-to-SQL.The conclusion links this need to the framework's reported enterprise setting.Source: §7 · p. 6

8 Limitations

The stated limitation concerns the BEAVER adaptation: its graph edges come from schema files rather than query logs. Consequently, those edges lack usage-frequency information and may not reflect relationships found in actual usage.

8 Limitations

  • BEAVER’s schema-derived edges lack frequency information and may include relationships absent from actual usage patterns.This limits how closely the adaptation’s graph represents query-log-derived enterprise structure.Source: §8 · p. 7

D.1 Motivation for Adaptation

  • BEAVER does not provide the query logs central to the approach, so the benchmark adaptation generates equivalent metadata from schema files.The authors frame this adaptation as a test of generalization beyond the log-based setting.Source: §D.1 · p. 10, D BEAVER Benchmark Adaptation · p. 10

D.2 Schema-to-Metadata Conversion

  • The schema-based adaptation discovered 1,113 implicit joins beyond 127 declared relationships, producing 1,310 graph edges across 463 tables.Candidate joins were identified through column-name matching and validated by an LLM.Source: §D.2 · p. 10

D.4 Limitations of Adaptation

The schema-based BEAVER adaptation has limitations, but its results are reported as evidence that entity-driven GraphRAG can provide value without query logs. Across the later analyses, the system adapts table selection to query complexity and shows advantages on open-ended questions and across databases.

D.4 Limitations of Adaptation

  • BEAVER adaptation uses equal-weight edges, synthetic joins that may not occur in real queries, and no historical query patterns for few-shot examples.Despite these constraints, its results suggest the approach provides value without query logs.Source: §D.4 · p. 11

E.1 Dataset Statistics

  • Table 7 summarizes dataset statistics for both evaluation settings, without reporting the values in this passage.Source: Table 7 · p. 11

E.2 Table Selection Statistics

  • Our system selects 5–16 tables according to query complexity, while RAG uses fixed top-k=12.Table 8 compares table-selection statistics on BEAVER.Source: §E.2 · p. 11, Table 8 · p. 11

E.3 Ablation Configuration Details

  • Entity matching has the largest impact on result fidelity in the ablation breakdown.The configurations remove entity matching, KG traversal, or PPR ranking one component at a time.Source: §E.3 · p. 11, Table 9 · p. 12

F.1 Generated vs Gold by Question Type

  • On deterministic questions, gold SQL dominates; on open-ended questions, our system achieves comparable or superior quality.The comparison supports the concern that gold-based evaluation can miss valid alternative formulations.Source: §F.1 · p. 12, Table 10 · p. 12

F.2 Ours vs RAG by Question Type

F.3 Per-Database Results

  • Our system improves execution validity across BEAVER databases, with the largest gains on the complex data warehouse (dw).Table 12 reports execution validity by database.Source: §F.3 · p. 12

G Example Queries with Rubric Analysis

The examples show that the system can select correct schemas and joins for some questions, but ambiguity and incomplete entity coverage can still limit answer quality. The failure analysis identifies entity coverage and disconnected graph regions as recurring issues.

G.1 High Quality: Entity Matching Enables Correct Schema Selection

  • 10/10: Entity matching grounded material-status, subject, and school terms in the schema, while the graph supplied a bridge-table join path.The answer received full scores for executability, schema validity, filtering, joins, and result fidelity.Source: §G.1 · p. 12

G.2 High Quality: KG Traversal Discovers Non-Obvious Tables

  • 10/10: Graph traversal found MOIRA tables not named in the question and supported a three-table join path with grand totals.The answer was assessed as executable, schema-valid, and comprehensive.Source: §G.2 · p. 13

G.3 Medium Quality: Reasonable Interpretation of Ambiguous Question

  • 8/10: The room query was executable and structurally sound, but its department-based interpretation differed from the gold answer to an ambiguous question.The passage notes that several interpretations of “CS students can enroll” are plausible, and exact match would miss practical utility.Source: §G.3 · p. 13

G.4 Low Quality: Missing Entity Coverage

  • The instructor query partially executed but failed to answer: it missed MOIRA_LIST and MOIRA_LIST_MEMBER, treated “keeper-zephyr” as a department filter, and returned NULL for publication years.Entity extraction failed to connect “mailing list” to the MOIRA schema; questions spanning academic and mailing-list domains may have incomplete entity coverage.Source: §G.4 · p. 14

G.5 Failure Pattern Summary

  • Entity coverage failures account for 40% of low-scoring queries, while disconnected graph regions account for 33%.The analysis identifies iterative refinement of entity extraction as a potential improvement target.Source: §G.5 · p. 14

H Comparison with Schema-Aware Methods

On the Amazon deployment, our system outperformed DIN-SQL in blind pairwise comparisons despite DIN-SQL receiving human-readable schema documentation. The authors attribute the difference to the knowledge graph’s use of historical table co-occurrence and terminology mappings.

Loading 2610.33946v1…