PaperlayerView exact revision on arXiv

Source-linked AI summary

text2ql: Multi-Target Natural Language Querying via a Language-Agnostic Intermediate Representation

Ritesh Kumar

arXiv:2609.02115v1cs.CLcs.AIcs.DB

TL;DR

Natural-language database interfaces remain constrained by SQL-only targeting, required LLM inference, and absent runtime warnings for semantic errors. text2ql addresses these gaps with a language-agnostic QueryIR, pluggable renderers, and a shared seven-stage pipeline. Its deterministic mode reaches 100% execution accuracy at 3.2 ms p50 latency, while LLM results and schema-aware ablations are evaluated on Spider and BIRD samples.

  • Problem

    Natural-language database interfaces traditionally target SQL exclusively, depend on LLM inference for every query, and lack runtime signals for semantically incorrect generated queries.

  • Method

    text2ql uses a language-agnostic QueryIR with pluggable renderers, a shared seven-stage pipeline for SQL and GraphQL, deterministic and LLM-backed modes, and runtime confidence scoring.

  • Results

    The deterministic mode achieves 100% execution accuracy with zero errors at 3.2 ms p50 latency; LLM mode reaches 62.0% Spider and 70.0% BIRD exact match, while schema-aware prompting adds +18.4 pp over no schema.

  • Takeaways & Limitations

    QueryIR and renderer plugins support multiple query targets, while schema configuration is identified as the highest-leverage available accuracy lever without fine-tuning or hardware investment.

  • Takeaways & Limitations

    Experiments use gpt-4o-mini and N = 50 per benchmark split, with indicative samples and no fine-tuning or query-specific few-shot selection.

Abstract

from arXiv · show

Natural language interfaces to databases have traditionally suffered from three structural limitations: exclusive targeting of relational SQL, unconditional dependence on large language model (LLM) inference at query time, and absence of any runtime signal when generated queries are semantically incorrect. This paper presents text2ql, an open-source Python framework that addresses all three limitations through a language-agnostic Intermediate Representation (QueryIR) and a pluggable renderer architecture. A single seven-stage detection pipeline serves both SQL and GraphQL targets; a zero-LLM deterministic mode delivers 100% execution accuracy at a median latency of 3.2 ms with no API cost; and every generated query carries a runtime confidence score in [0.15, 0.97] computed from an additive signal model. Evaluated on 50-query random samples from the Spider and BIRD benchmarks (indicative results; full-set evaluation is planned), the LLM-backed mode achieves 62-70% exact match and 84-91% execution accuracy; the deterministic mode achieves 100% execution accuracy with zero parse errors across all 100 test cases. An ablation study isolates schema-aware prompting as the dominant accuracy lever, contributing +18.4 percentage points of exact-match gain over the schema-free baseline on both benchmarks. text2ql is publicly available at https://pypi.org/project/text2ql/ under the Apache 2.0 license.

1. INTRODUCTION

text2ql addresses SQL-only targeting, unconditional LLM dependence, and silent semantic failures through a shared QueryIR-based framework. Its contributions include multi-target rendering, deterministic execution, schema-aware mapping, runtime confidence scoring, and benchmark infrastructure.

  • text2ql parses one natural-language utterance into language-agnostic QueryIR and renders equivalent GraphQL and SQL queries in under five milliseconds without an external API call.
  • The framework targets three limitations: SQL monoculture, unconditional LLM dependence, and semantically incorrect queries that provide no runtime warning.
  • QueryIR decouples natural-language parsing from rendering, allowing new target languages through a single IRRenderer subclass without changing the engine.
  • The deterministic engine achieves 100% execution accuracy on the test corpus with sub-5 ms p50 latency and zero API cost.
  • The project provides hybrid schema mapping with provenance tracking and benchmark infrastructure covering Spider, BIRD, three evaluation modes, and schema-aware prompting ablations.
  • A runtime confidence model combines additive signals and validation penalties, clips scores to [0.15,0.97], and supports dynamic cascade decisions.

2. RELATED WORK

Related work developed natural-language database interfaces through domain-specific systems, intermediate representations, schema-linking methods, and increasingly capable LLMs. Existing systems nevertheless remain largely SQL-only, LLM-dependent, or lacking runtime uncertainty signals, while GraphQL work remains sparse.

  • Early systems required substantial manual engineering or user correction, and pre-neural approaches did not generalize across domains without re-engineering.
  • SemQL introduced an intermediate representation for text-to-SQL, while later systems improved schema linking through relation-aware encoding and graph neural networks.
  • GPT-based systems raised Spider performance through self-correction, few-shot selection, prompt structure, and other strategies, but target SQL exclusively and require LLM inference for every query.
  • GraphQL research remains sparse, and cited systems lack either schema-aware prompting, uncertainty quantification, LLM-free operation, or simultaneous SQL support.

3. SYSTEM ARCHITECTURE

text2ql uses a four-layer architecture in which a facade dispatches to target-specific engines, engines produce QueryIR, and pluggable renderers serialize query strings. Shared schema and engine utilities let one configuration and common detection stages serve multiple targets and modes.

  • The facade dispatches to language-specific engines, which produce QueryIR passed to a pluggable renderer for final query serialization.
  • Adding a new target such as Cypher requires implementing IRRenderer.render() without changing engines or detection stages.
  • The facade supports synchronous and asynchronous APIs and selects deterministic, LLM, or function-calling mode per call using one schema configuration.
  • Shared engine utilities provide schema normalization, confidence computation, retries, fallback chaining, and structured logging across GraphQL and SQL engines.
  • NormalizedSchemaConfig encodes aliases, typed fields, filters, relations, defaults, arguments, and keyword-intent routing as the primary production extension point.
  • QueryIR stores entities, fields, filters, aggregations, joins, nested selections, ordering, pagination, distinctness, and grouping-related fields in a typed dataclass.
  • SQLIRRenderer serializes SELECT statements with joins and clauses, while GraphQLIRRenderer constructs arguments, selection sets, aggregations, and nested sub-selections.

4. MULTI-STAGE DETECTION PIPELINE

The deterministic pipeline sequentially enriches a shared QueryIR through entity, field, filter, aggregation, relation, ordering, validation, and confidence stages. It uses schema-aware heuristics and records failures as confidence penalties rather than hard exceptions.

  • Seven independently testable stages execute sequentially, enriching the shared QueryIR while recording failures as validation issues rather than raising hard exceptions.
  • Entity resolution: Entity resolution follows a priority cascade from aliases and schema names through keyword intents, field overlap, filter values, column mentions, and BFS fallback.
  • Field detection: Fields come from explicit mentions after alias expansion, configured defaults when no fields are detected, or aggregation keywords that introduce target fields.
  • Filter detection: The regex engine handles equality, inequalities, ordered comparisons, ranges, set membership, and null checks, with aliases mapping business vocabulary to schema values.
  • Aggregation: Aggregation keywords trigger COUNT, SUM, AVG, MIN, or MAX with implicit GROUP BY, while postaggregation conditions generate HAVING clauses.
  • Relations: SQL joins use configured ON-column pairs, whereas GraphQL nested detection traverses relations to depth three with cycle protection; both share relation resolution logic.
  • Pagination and ordering: Pagination scans numeric tokens with intent keywords, and ordering derives from superlatives, directional keywords, or configured default sort fields and directions.
  • Validation and confidence: Confidence sums signals for resolution quality, field coverage, filter richness, and structural complexity, subtracts capped validation penalties, and clips the result to [0.15,0.97].

5. GENERATION MODES

text2ql offers deterministic, LLM, and function-calling generation modes, with deterministic fallback and confidence-based cascading. The deterministic mode avoids LLM calls while preserving semantic correctness and low latency.

  • 5.1 Deterministic Mode: 100% execution accuracy with zero errors across 100 test cases is achieved by deterministic mode at sub-5 ms p50 latency and zero API cost.Its conservative output form causes 0% exact match on standard benchmarks because gold annotations prefer compact equivalents.
  • 5.2 LLM Mode: The LLM pipeline schema-validates responses and automatically falls back to the deterministic result when validation fails.This produced a zero error rate across observed test cases regardless of LLM output quality.
  • 5.3 Function-Calling Mode: Structured-output function calling deserializes directly into QueryIR and achieves 2 pp higher exact match than completion mode.Deterministic fallback maintains zero observed errors.
  • 5.4 Cascade Strategy: The recommended cascade returns deterministic results when confidence ≥0.75 and invokes LLM mode below that threshold.The threshold is derived from the test-corpus confidence distribution and should be tuned per schema.

6. EXPERIMENTAL EVALUATION

Evaluation on 50-query random samples from Spider and BIRD compares text2ql’s generation modes, schema-aware prompting, latency, and related systems. Deterministic mode provides perfect execution accuracy at millisecond latency, while schema information substantially improves LLM exact match.

  • 6.1 Experimental Setup: 50-query random samples from Spider and BIRD provide indicative benchmark results with an estimated ±7 pp margin at 95% confidence.Experiments use gpt-4o-mini without fine-tuning or query-specific few-shot selection.
  • 6.2 Main Results: 100% execution accuracy with zero errors at 3.2 ms p50 latency is achieved by deterministic mode across both benchmarks.LLM mode reaches 70.0% exact match on BIRD and 62.0% on Spider; structural accuracy exceeds exact match in all modes.
  • 6.2 Main Results: 2 pp higher exact match on both benchmarks is achieved by function-calling mode compared with completion mode.The result is consistent with reduced output parsing variance from JSON-structured output.
  • 6.3 Ablation Study: +18.4 pp exact-match gain on both benchmarks results from full schema configuration over the no-schema baseline.The full configuration includes field aliases, filter value aliases, and keyword_intents; schema-free prompting yields 43.6% Spider EM and 51.6% BIRD EM.
  • 6.4 Latency Analysis: 260× faster median latency is measured for deterministic mode than LLM completion, with 3.2 ms versus 840 ms p50 latency.The comparison supports real-time, embedded, and latency-critical use cases where LLM round-trips are prohibitive.
  • 6.5 Comparison with State of the Art: 62.0% versus 86.6% exact match separates text2ql-LLM from DAIL-SQL, while text2ql uniquely combines multi-target extensibility, LLM-free operation, and runtime confidence scoring among compared systems.The comparison attributes text2ql’s lower exact match to the absence of fine-tuning and query-specific few-shot selection.

7. SYSTEM DEPLOYMENT

text2ql is distributed as a Python package with deterministic and LLM-backed usage modes, configurable providers, and CI-oriented benchmarking guidance.

  • Distribution: text2ql is published on PyPI with a Python ≥3.10 requirement, a base install containing both pipelines, and optional SQL and application extras.The CLI is registered automatically, and the package reached v0.2.6 across 14 releases in 10 days.
  • Usage modes: The deterministic quick-start mode constructs Text2QL from a schema and generates queries with zero cost and sub-5 ms latency.The example returns a generated query and a confidence value of 0.88.
  • Usage modes: LLM mode is activated through a provider configuration passed to the Text2QL constructor, and supports OpenAI-compatible endpoints and local models.Structured-output mode can be enabled with use_structured_output=True.
  • Operational guidance: Deployment guidance recommends versioning schema and mapping files, establishing a deterministic baseline, and gating LLM mode below confidence 0.75.The guidance also recommends execution benchmarking in CI before each schema release.

8. LIMITATIONS

The evaluation is constrained by indicative benchmark samples, SQL-only gold standards, manual schema configuration, automated metrics, and known query-generation failure patterns.

  • 8.1 Benchmark Sample Size: 50-query random samples from Spider and BIRD make the reported evaluation indicative rather than definitive, with full-set evaluation planned.Spider and BIRD contain 1,034 and 1,534 queries respectively.
  • 8.2 SQL-Only Benchmark Bias: Spider and BIRD provide SQL gold annotations only, so GraphQL generation is assessed structurally rather than against benchmark gold standards.A GraphQL-specific evaluation corpus is not currently available in the literature and is planned as a future contribution.
  • 8.3 Schema Configuration Overhead: Complete NormalizedSchemaConfig is required for both schema-aware accuracy gains and deterministic 100% execution accuracy, creating manual configuration overhead.Small schemas typically require 1–2 hours, while large schemas remain burdensome and error-prone despite scaffold generation.
  • 8.4 No Human Evaluation: The evaluation uses automated metrics and does not assess domain-expert judgments of whether generated queries match user intent.The paper notes that automated metrics can diverge from human preference on complex analytical queries with ambiguous intent.
  • 8.5 Exact-Match Metric Adequacy: Exact match penalizes semantically equivalent queries, illustrated by deterministic mode’s 0% exact match alongside 100% execution accuracy.The paper recommends execution-based or semantic-equivalence metrics for future production evaluation.
  • 8.6 Common Failure Patterns: LLM-mode errors include aggregation misattribution, omitted implicit joins, and hallucinated filter values, while deterministic failures involve unsupported sub-selects or correlated predicates.On the Spider sample, the reported error shares are approximately 8%, 12%, and 5% for the three LLM categories.

9. FUTURE SCOPE

Future work prioritizes broader query-target support, scalable schema handling, automatic schema discovery, and full benchmark evaluation.

  • Target and scale extensions: A planned Cypher renderer would extend QueryIR to property graphs with an estimated 150 lines and no engine changes.QueryIR already encodes entity–relation semantics needed for this renderer.
  • Target and scale extensions: Future directions include vector-store entity retrieval using dense approximate nearest-neighbor search for schemas with thousands of entities.
  • Schema automation: Auto-schema discovery from live database introspection or GraphQL SDL is planned to eliminate the manual configuration burden.
  • Evaluation: Full-set benchmark evaluation on Spider and BIRD is identified as the most immediate priority for the next release.

10. CONCLUSION

text2ql addresses SQL-only targeting, mandatory LLM dependence, and silent semantic failures through a unified multi-target framework. Its deterministic and LLM-backed modes achieve strong execution accuracy, while schema-aware prompting emerges as the main accuracy lever and several limitations define future work.

  • Results: 100% execution accuracy in deterministic mode and 84–91% execution accuracy with LLM backing were achieved across SQL and GraphQL test cases.The framework uses a seven-stage detection pipeline, three generation modes, and a pluggable renderer interface, with zero observed errors across all test cases.
  • Contribution: QueryIR decouples natural language understanding from query rendering, enabling SQL and GraphQL support from a single codebase.The typed intermediate representation is the core mechanism for addressing multi-target querying and the limitations of prior NL2QL systems.
  • Ablation: +18.4 pp exact match identifies schema-aware prompting as the dominant accuracy lever over the schema-free baseline.The conclusion frames schema quality, rather than model scale, as the most productive accuracy investment for production deployments.
  • Limitations: Five limitations include evaluation sample size, missing GraphQL gold standards, schema configuration burden, absent human evaluation, and exact-match inadequacy.These limitations define a concrete research agenda for extending and evaluating the framework.

CODE AND DATA AVAILABILITY

text2ql is distributed through PyPI under the Apache 2.0 license, with an interactive playground and benchmark loaders included. The underlying Spider and BIRD datasets remain under their original authors’ licenses and are not redistributed.

  • Distribution: The text2ql package is distributed on PyPI under the Apache 2.0 license.The package is available at https://pypi.org/project/text2ql/.
  • Resources: An interactive playground is available at https://text2ql.streamlit.app.
  • Benchmark data: Benchmark loaders for Spider and BIRD ship with the package, but their datasets are not redistributed.The benchmark datasets remain distributed by their original authors under their own licenses.
Loading 2609.02115v1…