PaperlayerView exact revision on arXiv

Source-linked AI summary

TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases

Fei Lyu, Zhiyi Peng, Jiaming Liu, Yixuan Yang, Changjian Chen, Zhuo Tang, Jiapeng Zhang, Kenli Li

arXiv:2609.34783v1cs.CL

TL;DR

Text-to-query capabilities for TSDBs remain largely unassessed, and existing benchmarks do not capture their varied syntaxes, domains, and time-specific intents. TQTS-BENCH provides a human-reviewed multi-syntax benchmark, and evaluation finds a substantial gap between the best-performing model and humans.

  • Problem

    Text-to-query progress on relational databases has not established performance on TSDBs, and existing benchmarks do not capture their diverse syntaxes, domains, and time-specific intents.

    Source: §1 · pp. 1–2

  • Method

    TQTS-BENCH combines 6,125 QA pairs across 97 TSDBs and 23 syntaxes with human-centric AI-assisted construction and cross-validation.

    Source: §1 · p. 2, §3.2.2 · p. 5

  • Results

    Claude-Opus-5 achieves 48.98% execution accuracy, compared with 87.34% for humans; analysis identifies syntax heterogeneity, intent misinterpretation, and schema-linking errors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations show that existing methods still struggle with TSDB tasks, particularly syntax handling, intent understanding, and schema linking.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have significantly advanced natural language querying over relational databases, yet their ability to query time-series databases (TSDBs) remains largely unassessed. Existing benchmarks fail to adequately capture the non-unified query syntaxes, diverse application domains, and unique time-specific query intents inherent to TSDBs. To address this gap, we introduce TQTS-BENCH, a multi-syntax benchmark for evaluating text-to-query capabilities over TSDBs. TQTS-BENCH contains 6,125 high-quality question-answering (QA) pairs spanning 97 TSDBs, 23 distinct query syntaxes, 22 application domains, and 4 types of time-specific query intents. It is constructed through a human-centric AI-assisted workflow, where all QA pairs are carefully reviewed and revised by domain experts to ensure quality and correctness. Extensive evaluations of advanced LLMs and state-of-the-art text-to-query methods reveal challenges in querying TSDBs. Even the best-performing model evaluated, Claude-Opus-5, achieves only 48.98% execution accuracy, while humans reach 87.34%. Error analysis reveals that this performance gap mainly stems from the heterogeneous query syntaxes across different TSDBs, misinterpretation of time-specific intents, and incorrect schema linking. These findings highlight new opportunities to narrow the gap between current LLM capabilities and the requirements of TSDB queries in real-world applications. The benchmark is available at: https://anonymous.4open.science/r/TQTS-Bench-00CD.

1 INTRODUCTION

Text-to-query progress on relational databases has not established how well models handle TSDBs, whose syntaxes, domains, and time-specific intents differ. TQTS-BENCH addresses this gap, and evaluation finds that current methods still struggle.

1 INTRODUCTION

  • Existing text-to-query methods for RDBs have approached human-level results on RDB benchmarks, but TQTS progress remains unclear without a comprehensive TSDB benchmark.The introduction identifies Spider and BIRD as examples of RDB benchmarks.Source: §1 · p. 1
  • TSDB queries differ from RDB queries in syntax, domain coverage, and intent: they include non-uniform languages, high-frequency-data domains, and time-specific analysis.Flux uses pipeline-style syntax, PromQL nested function-oriented expressions, and QuestDB SQL adds temporal operations.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • TQTS-BENCH contains 6,125 QA pairs spanning 97 TSDBs, 23 query syntaxes, 22 application domains, and four time-specific intents, with expert collaboration and cross-validation.The workflow begins with 23 representative TSDB systems, 427 datasets, and 1,405 seed QAs.Source: §1 · p. 2
  • Claude-Opus-5 achieves 48.98% execution accuracy versus 87.34% for humans; error analysis identifies syntax heterogeneity, time-specific intent misinterpretation, and incorrect schema linking.The evaluation also finds limited performance among existing RDB and TSDB methods.Source: §1 · p. 2

2 RELATED WORK

Existing text-to-query benchmarks cover varied relational-database settings, while TSDB-focused benchmarks have generally targeted specific systems, syntaxes, or domains. TQTS-BENCH is designed to evaluate across these varied TSDB settings.

2 RELATED WORK

  • RDB benchmarks address unseen databases, real-world data, enterprise workflows, specialized applications, and reasoning or robustness, but do not capture time-series requirements.Examples include Spider (Yu et al., 2018), BIRD, Spider 2.0 (Lei et al., 2025), ScienceBenchmark (Zhang et al., 2023), SParC (Yu et al., 2019b), and CoSQL (Yu et al., 2019a).Source: §2 · p. 3
  • Prior TSDB benchmarks have focused on individual systems, syntaxes, or domains, limiting comprehensive evaluation across TSDB environments and scenarios.Examples include TEFD, Dranca et al. (2026), and PromCopilot (Zhang et al., 2026).Source: §2 · p. 3
  • TQTS-BENCH spans diverse TSDBs, query syntaxes, and application domains to support systematic evaluation of text-to-query capabilities over TSDBs.Source: §2 · p. 3

3 DATASET CONSTRUCTION

The paper defines TQTS as generating an executable query from a question, schema, and target TSDB, then constructs and verifies a benchmark spanning diverse systems, syntaxes, domains, and temporal intents.

3.1 TASK DEFINITION

  • TQTS generates an executable query from a natural-language question, database schema, and target TSDB to retrieve the requested data.Schemas describe timestamp, value, and metadata organization; TSDBs may represent these as tables or measurements, columns or fields, and tags or labels.Source: §3.1 · p. 3

3.2 CONSTRUCTION PIPELINE

  • The construction pipeline has two stages: curate 97 TSDBs, then construct and verify 6,125 QA pairs through a human-centric AI-assisted workflow.Source: §3.2 · p. 3, Figure 2 · p. 4

3.2.1 TSDB CONSTRUCTION

  • From 427 candidate datasets and 1,405 seed QAs, the authors manually select 97 representative datasets based on authenticity and schema complexity, then instantiate them in their TSDB systems.Source: §3.2.1 · p. 4

3.2.2 QA CONSTRUCTION AND VERIFICATION

  • Domain experts identify nine representative query intents; LLMs generate questions and executable queries for annotator review, while two annotators independently hand-write and may execute queries, escalating unresolved cases to adjudication.Verification covers all 6,125 QA pairs; annotators spent more than 600 hours and executed 30,774 queries.Source: §3.2.2 · pp. 4–5

3.3 DATA STATISTICS

  • The benchmark spans 23 syntaxes across 97 TSDBs; the same question can require pipeline transformations, metric selectors and range-vector functions, or SQL temporal operations.It covers all 22 application domains in the comparison, and has the highest average numbers of intents, time-specific intents, and time-specific operations per QA pair among compared benchmarks.Source: §3.3 · p. 5, Figure 3 · p. 5, Figure 4 · p. 6

4 EXPERIMENT

Evaluations find that LLMs and existing text-to-query methods struggle on TQTS-BENCH, with errors spanning syntax, intent understanding, and schema linking. Converting TSDBs to RDBs improves execution accuracy, pointing to diverse query syntaxes as a major source of difficulty.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses execution accuracy, classifies question difficulty by intent count, and measures human performance on a randomly sampled 10% subset.Difficulty bands are 1–3 intents (easy), 4–5 (medium), and 6–9 (hard).Source: §4.1 · p. 6

4.2 QUANTITATIVE EVALUATION RESULTS

  • DeepEye-SQL achieves 5.83% execution accuracy on TQTS-BENCH, while most evaluated RDB methods score below 10.00%.Some RDB methods score as low as 0.54%.Source: §4.2 · p. 7, Table 1 · p. 7
  • Claude-Opus-5 achieves 48.98% execution accuracy, below the 87.34% human baseline; its performance remains limited despite outperforming existing RDB methods.Claude-Opus-5 also consistently outperforms other models at medium and hard difficulty levels.Source: §4.2 · p. 7, Table 1 · p. 7
  • PromCopilot achieves 0.69% execution accuracy overall, and its performance falls from 3.03% to 0.00% when the domain shifts to others.PromCopilot targets PromQL and is primarily designed for cloud services and AIOps.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 ERROR ANALYSIS

  • 75.29% of identified errors are query syntax errors, while 61.43% are intent-understanding errors; time-specific intents account for 65.41% of intent errors.Syntax errors include structural and function/keyword misuse; intent errors include missing window aggregation and resampling.Source: §4.3 · p. 8, Figure 5 · p. 8, Figure 7 · p. 9
  • 17.09% of identified errors involve schema linking, with error cases averaging 381k schema tokens versus 115k for other erroneous cases.The analysis identifies diverse metric, label, and tag structures and large schemas as schema-linking challenges.Source: §4.3 · pp. 8–9

4.4 ABLATION STUDY

  • Converting TSDBs into RDBs raises execution accuracy from 0.00% to 32.21%, below BIRD’s 56.91% but above Spider 2.0-lite’s 22.94%.The authors attribute the original poor performance to diverse query syntaxes rather than question difficulty.Source: §4.4 · p. 9

5 CONCLUSION

TQTS-BENCH is a benchmark for evaluating text-to-query capabilities over TSDBs, designed around their distinct syntaxes, domains, and time-specific intents. Evaluations find that existing methods still struggle with these tasks, especially syntax handling, intent understanding, and schema linking.

5 CONCLUSION

  • TQTS-BENCH is a comprehensive benchmark for evaluating text-to-query capabilities over TSDBs.Source: §5 · p. 9
  • Evaluations find that existing methods still struggle with TSDB tasks, particularly query syntax handling, intent understanding, and schema linking.Source: §5 · p. 9
  • The analysis highlights opportunities to develop more effective and generalizable methods for real-world time-series analysis.Source: §5 · p. 9

AI USAGE DISCLOSURE

The paper reports limited uses of LLMs, with human experts reviewing and finalizing AI-assisted QA candidates. The benchmark appendix describes system selection and the temporal query functions illustrated for selected systems.

AI USAGE DISCLOSURE

  • LLMs assisted with related-work coverage, candidate QA generation, and grammar refinement; human experts reviewed and finalized candidate pairs.Source: AI USAGE DISCLOSURE · p. 10

A DETAILS OF TQTS-BENCH

  • The appendix details the TSDB systems selected, their characteristics, and their domains, as well as the benchmark's query intents and their proportions.Source: A DETAILS OF TQTS-BENCH · p. 13

A.1 IDENTIFYING TSDB MANAGEMENT SYSTEMS AND THEIR QUERY SYNTAX

  • The authors selected 23 systems based on popularity and accessibility; some highly ranked systems, such as kdb+, were excluded because licensing limits access.Source: §A.1 · p. 13
  • Examples for two intents illustrate system-specific functions and operators for window aggregation and resampling or temporal change analysis.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 IDENTIFYING TSDBS AND THEIR DOMAINS

The appendix describes the benchmark's databases, intents, schemas, TSDB contexts, and human verification workflow. It covers diverse real-world workloads and uses expert-defined intents and iterative review to check QA quality.

A.2 IDENTIFYING TSDBS AND THEIR DOMAINS

A.3.1 THE IDENTIFIED QUERY INTENTS

A.3.2 STATISTIC OF QUERY INTENTS

  • Time-specific intents occur more often in hard questions, and queries tend to involve more intents as difficulty increases.Time-specific intents show the most pronounced increase with difficulty.Source: §A.3.2 · p. 15, Figure 9 · p. 16

A.4 NON-UNIFIED SCHEMA STRUCTURES ACROSS TSDBS

  • The appendix illustrates seven ways TSDBs can organize the same time-series data, including relational tables, metric–label series, device trees, and property graphs.Source: §A.4 · p. 16, Figure 10 · p. 17

A.5 DETAILS OF THE TSDB CONTEXT

  • Each TSDB receives customized context covering its background, characteristics, and usage scenarios; the Prometheus example includes a fixed benchmark time and cloud-service telemetry.Source: §A.5 · p. 17

B.1 DESIGN OF VISUAL ANALYTICS ANNOTATION TOOL

  • The annotation tool is designed to check executable-answer correctness, question naturalness and clarity, and alignment with query intents and TSDB context.Source: §B.1 · pp. 18–19

B.2 DETAIL OF QA VERIFICATION

  • Two annotators independently construct queries while unable to see the candidate answer or peer solution, then inspect execution results and refine their answers.Source: §B.2 · pp. 19–20
  • An adjudicator reviews inconsistent or unanswerable pairs, revises them, and repeats verification until both annotators obtain results consistent with the revised answer.Source: §B.2 · p. 20

C.1 EVALUATION CRITERIA FOR TQTS-BENCH

The evaluation criteria adapt execution accuracy to TSDB outputs that can differ in format despite representing the same data. Human experts manually write queries from the supplied question, schema, and context, which are then evaluated for execution accuracy.

C.1 EVALUATION CRITERIA FOR TQTS-BENCH

  • Execution Accuracy primarily checks whether predicted output contains all columns returned by the gold query, but equivalent TSDB data can appear in different formats.Source: §C.1 · p. 20
  • An additional transformation step addresses format-related false negatives while avoiding an increase in false positives.Source: §C.1 · p. 20, Figure 14 · p. 20

C.2 DETAILED INFORMATION FOR HUMAN EVALUATION

  • For human evaluation, experts receive the question, database schema, and TSDB context, manually write queries, and have their queries evaluated for EX.Source: §C.2 · pp. 20–21

C.3 WHY THE RDB METHODS STRUGGLE WITH THE TQTS TASK

RDB methods generally perform poorly on TQTS, so the authors examine DeepEye-SQL, a strong text-to-query method on BIRD. Its SQL-focused prompts make it difficult to generate queries in the diverse syntaxes TSDBs require.

C.3 WHY THE RDB METHODS STRUGGLE WITH THE TQTS TASK

  • RDB methods generally perform poorly on TQTS, prompting further analysis of DeepEye-SQL, a representative high-performing method on BIRD.Source: §C.3 · p. 21
  • DeepEye-SQL’s query-generation prompts are heavily constrained by SQL syntax.The excerpt in Figure 15 illustrates the prompt design.Source: §C.3 · p. 21, Figure 15 · p. 21
  • DeepEye-SQL’s SQL-specific query-generation prompts limit its ability to produce the different syntaxes and structures required by TSDBs.The example shows it generating SQL when PromQL was required, leading to execution failure.Source: §C.3 · p. 21, Figure 16 · p. 22

C.4 WHY THE TSDB METHODS STILL STRUGGLE WITH TQTS TASK

PromCopilot still achieves limited execution accuracy on TQTS-BENCH. Its Kubernetes- and Prometheus-focused knowledge graph and prompts restrict transfer beyond cloud services and AIOps.

C.4 WHY THE TSDB METHODS STILL STRUGGLE WITH TQTS TASK

  • TSDB methods still achieve limited execution accuracy on TQTS-BENCH, motivating analysis of PromCopilot, a method designed for Prometheus-based monitoring.Source: §C.4 · p. 22
  • PromCopilot’s knowledge graph encodes Kubernetes cloud-native entities and relations, making direct transfer to other domains difficult.Figure 17 contrasts its domain-focused prompts with the original paper’s external-validity statement.Source: §C.4 · p. 22, Figure 17 · p. 23
  • PromCopilot achieves only 0.69% EX on TQTS-BENCH, and its Kubernetes- and Prometheus-specific designs may explain this limited performance.Its system prompts are tailored primarily to cloud services and AIOps, while its knowledge graph encodes Kubernetes components and relations.Source: §C.4 · p. 22

C.5 MORE EXAMPLES OF ERROR ANALYSIS

The error analysis organizes error types and illustrates how query syntax mistakes can alter execution or returned results. The examples distinguish structural and operator errors.

C.5 MORE EXAMPLES OF ERROR ANALYSIS

  • Table 6 summarizes the error taxonomy and links each subcategory to representative cases; the accompanying examples focus on time-specific intent errors.Source: §C.5 · p. 24, Table 6 · p. 24
  • Figure 18 shows two syntax-error patterns: placing LAG() directly in WHERE causes a window-function execution error, while substituting delta for subtraction changes the computation and can produce incorrect values and rankings.Source: Figure 18 · p. 24

C.6 DETAILS OF THE ABLATION STUDY

The syntax-only ablation converts selected TSDB cases to an RDB setting to examine the role of query syntax. The construction prompts guide question and query generation using intent labels, schema, and database context.

C.6 DETAILS OF THE ABLATION STUDY

  • Because 75.29% of syntax-error cases also involve other error types, the ablation selects cases where syntax errors are the sole error type.The selected TSDBs are mapped to RDBs, with schemas formatted as in BIRD.Source: §C.6 · p. 24
  • 32.21% execution accuracy, up from 0%, was achieved on syntax-only error cases after conversion to an RDB setting.The questions stayed unchanged, received manually annotated SQL gold queries, and were checked for answer consistency; the result suggests TSDB syntax handling is a primary difficulty.Source: §C.6 · p. 26

D PROMPTS USED IN TQTS-BENCH CONSTRUCTION PROCESS

  • The construction taxonomy distinguishes time-specific intents I1–I4 from time-agnostic intents I5–I9, including window aggregation, temporal change, localization, and relationship analysis.Source: D PROMPTS USED IN TQTS-BENCH… · p. 27
  • Question-generation prompts require meaningful, unambiguous questions grounded in schema and context, and prohibit unsupported assumptions or goals beyond querying the TSDB.Source: D PROMPTS USED IN TQTS-BENCH… · p. 27
  • The query-generation prompt requires executable syntax for the target TSDB, schema-grounded objects, and verification of successful, valid results when the database is accessible.It also instructs the model not to guess when required information is missing or the question conflicts with the schema or context.Source: D PROMPTS USED IN TQTS-BENCH… · p. 28

Related papers

Loading 2609.34783v1…