PaperlayerView exact revision on arXiv

Source-linked AI summary

RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases

Yao Cheng, Siqiang Luo

arXiv:2610.21382v1cs.LG

TL;DR

Natural-language interfaces to relational databases must answer both current-state questions and future-oriented predictions, but existing benchmarks address these goals largely in isolation. RDL-Bench combines both in a task-aware benchmark and evaluates native solvers alongside routed modular systems. Its findings include limited native reasoning performance, format-sensitive LLM prediction, and routing errors tied to understanding the required capability.

  • Problem

    Existing benchmarks largely study current-state reasoning and future-oriented prediction separately, although users may ask both kinds of questions over the same database.

    Source: §I · p. 1

  • Method

    RDL-Bench combines current-state reasoning and temporally constrained prediction in one natural-language benchmark, evaluating native solvers and modular systems that route questions to task-appropriate modules.

    Source: §I · p. 2

  • Results

    The best seven-domain native reasoning macro-average is 0.49, while LLM predictive performance is format-sensitive and routing can expose failures to distinguish reasoning from prediction.

    Source: §I · p. 2

  • Takeaways & Limitations

    The benchmark evaluates task inference alongside execution and separates routing errors from execution errors in modular relational QA systems.

    Source: §I · p. 2

Abstract

from arXiv · show

Relational databases (RDBs) remain the primary infrastructure for storing and managing structured data across enterprise and scientific domains. Natural-language (NL) interfaces to RDBs aim to let users ask questions without writing SQL, but real user needs go beyond SQL-style lookup: some questions require reasoning over the observable database state, while others require predictive inference from historical relational records under temporal constraints. Existing benchmarks address these goals in isolation. To address this gap, we introduce RDL-Bench, a benchmark for task-aware question answering (QA) over real RDBs. RDL-Bench organizes diverse user questions into reasoning-oriented questions, whose answers are determined by the current database state, and prediction-oriented questions, whose answers depend on future outcomes under temporal constraints. Each instance contains an NL question, task metadata, schema information, temporal context, either executable evidence or target-construction logic, and denotation-level ground truth. RDL-Bench supports two evaluation settings: native solvers under their intended capabilities, and task-aware modular systems where a router dispatches each question to an appropriate execution module.

I. INTRODUCTION

Relational QA must handle both questions about the current database state and predictions about future outcomes, but existing benchmarks largely study these capabilities separately. RDL-Bench unites them in a task-aware benchmark and evaluates both native solvers and routed modular systems.

I. INTRODUCTION

  • RDL-Bench covers current-state reasoning and future-oriented prediction over real multi-table databases through a shared natural-language workload and instance schema.Instances include task metadata, schema information, temporal context, and executable evidence or target-construction logic; prediction inputs are restricted to records before the reference time to avoid information leakage.Source: §I · p. 2
  • Its two-level evaluation measures native solver performance and modular QA performance, including the routing-induced gap between an LLM-based router and an oracle router across seven domains.The motivating workload includes questions requiring current-state reasoning and future-oriented prediction, while existing benchmarks mostly evaluate these goals in isolation.Source: §I · pp. 1–2
  • The best seven-domain native reasoning macro-average is 0.49, with performance varying across domains and answer forms.Current methods struggle to align user intent, schema grounding, relational composition, and answer normalization.Source: §I · p. 2
  • Two-hop context helps in some domains but does not consistently improve over schema-only prompting, and more local records alone do not resolve query-planning and answer-form errors.Source: §I · p. 2
  • LLMs can be competitive on Boolean prediction from schema and historical statistics but remain unstable on scalar regression, so Boolean predictive QA can overestimate relational understanding.Source: §I · p. 2
  • Strong LLMs route accurately, whereas Llama8B can identify relevant tables yet fail to distinguish reasoning from prediction.Routing requires recognizing the execution capability implied by the question.Source: §I · p. 2

II. RELATED WORK

Earlier structured-data QA benchmarks largely answer questions from existing database states, while relational deep learning benchmarks focus on future prediction; RDL-Bench evaluates both under a shared NL interface and includes text-to-SQL methods as reasoning solvers.

A. Question Answering over Structured Data

  • Structured-data QA benchmarks cover current-state questions, while relational deep learning benchmarks cover future prediction; RDL-Bench brings both into a shared NL workload and evaluates routing.Earlier benchmarks do not evaluate whether systems recognize the required capability and route accordingly.Source: A. Question Answering over Structured… · p. 2, Figure 1 · p. 3
  • RDL-Bench complements Table QA, text-to-SQL, and relational deep learning benchmarks with explicit temporal constraints and separate native-solver and task-aware QA evaluation.The benchmark evaluates current-state reasoning alongside future-oriented prediction under a shared NL interface.Source: A. Question Answering over Structured… · p. 2

B. Text-to-SQL Methods

  • Text-to-SQL methods generate executable queries for answers in an existing database snapshot, but are not designed for future-oriented prediction.RDL-Bench evaluates them as native solvers for reasoning-oriented questions and characterizes their cross-domain limitations.Source: B. Text-to-SQL Methods · pp. 2–3

C. Relational Deep Learning and Predictive Modeling

Relational deep learning benchmarks evaluate prediction over relational structure, while RDL-Bench frames future prediction and current-state reasoning as NL questions and evaluates task-aware routing. Its instances represent databases, questions, optional reference times, and task metadata, with predictive context separated from future targets.

C. Relational Deep Learning and Predictive Modeling

  • RDL-Bench extends relational prediction benchmarks by expressing predictive tasks as NL questions and evaluating them jointly with current-state reasoning in a unified QA framework.Relational deep learning methods model multiple entity types, relations, and temporal dependencies, using either entity-level tabular features or graph structure.Source: C. Relational Deep Learning and… · p. 3

D. Positioning RDL-Bench

  • Prior structured-data QA benchmarks lack predictive-track and routing evaluation, while relational deep learning benchmarks lack a user-facing NL interface and routing component.RDL-Bench combines current-state answering and future prediction, with native-solver and end-to-end task-aware evaluation.Source: D. Positioning RDL-Bench · p. 3

III. PRELIMINARIES AND PROBLEM DEFINITION

  • The task is to answer an NL question over an RDB by identifying its required capability and invoking an appropriate execution mechanism, rather than assuming one model family solves every question type.The benchmark assesses both native solvers and modular systems routing questions to specialized modules.Source: §III · p. 3

A. Relational Databases and QA Instances

  • An instance includes an RDB and schema, an NL question, optional reference time, and task metadata; predictive context uses records at or before that time while targets come from later records.This separation is specified to prevent temporal leakage in predictive evaluation.Source: A. Relational Databases and QA Instances · p. 3

B. Capability Tracks

RDL-Bench distinguishes questions answered from the observable database state from questions requiring estimates of unobserved outcomes. Task-aware systems route each question to an execution module, with task metadata guiding evaluation across answer types.

B. Capability Tracks

  • Reasoning Track questions cover retrieval, aggregation, and multi-hop relational reasoning, with answers obtained by executing a query or program over the observable context.The track's answers are determined by the observable database state.Source: B. Capability Tracks · pp. 3–4
  • Predictive Track questions require estimating targets constructed from records after the reference time, using only the observable context available before it.Targets may be future labels, scores, scalar values, counts, or rankings, and cannot be answered by directly querying the context.Source: B. Capability Tracks · p. 4

C. Task-Aware Relational QA

  • A task-aware system's router predicts a route containing the capability track and metadata such as query type, answer form, target entity, and relevant schema components.Native solvers use a fixed route, whereas modular systems predict it, enabling measurement of both capability-specific and end-to-end performance.Source: C. Task-Aware Relational QA · p. 4, D. Evaluation Objective · p. 4
  • Task metadata selects the evaluator for each instance, while the overall objective is expected task loss across the benchmark's heterogeneous instances.Evaluation may use exact match for scalar answers, execution-based correctness, or task-specific predictive metrics.Source: D. Evaluation Objective · p. 4

IV. DATASET CONSTRUCTION

RDL-Bench is constructed from real multi-table databases with varied relational structure and organizes instances by the capability needed to answer them. Its taxonomy separates current-state retrieval, analysis, and relational reasoning from prediction using temporally bounded context.

IV. DATASET CONSTRUCTION

  • The benchmark uses seven real-world databases spanning domains including e-commerce, finance, sports, healthcare, transportation, social platforms, and government.The databases vary in schema size, table count, relational depth, temporal coverage, and entity distributions.Source: §IV · pp. 4–5

B. Task Taxonomy

  • Reasoning-oriented tasks comprise retrieval, analytical aggregation or ranking, and relational reasoning involving composition across tables or conditions.The categories range from filtering and projection to joins, nested queries, set operations, and indirect constraints.Source: B. Task Taxonomy · p. 5
  • Prediction-oriented instances use context observed before a reference time and derive targets from later records, including labels, scores, values, counts, or rankings.The taxonomy distinguishes these future-outcome questions from answers determined by the observable database state.Source: B. Task Taxonomy · p. 5

C. Unified Instance Schema

RDL-Bench represents each question with a shared schema that records its task, database context, evidence or target logic, temporal information, and denotation-level answer. Its generation process profiles databases, creates candidates across task types, and applies temporal constraints where needed.

C. Unified Instance Schema

  • Each instance has six components: the question, task metadata, schema information, evidence or target-construction logic, temporal context, and denotation-level ground truth.The question may include paraphrases, while ground truth stores a raw denotation and may also include formatted answers and variants.Source: C. Unified Instance Schema · pp. 5–6
  • Task metadata specifies query type, answer form, complexity, SQL features, involved tables, and reference time when applicable.Schema information records relevant table names, attributes, data types, primary keys, and foreign-key relations.Source: C. Unified Instance Schema · p. 5
  • Evidence is a validated SQL query for reasoning questions and target-construction logic, reference time, prediction window, and label rule for prediction questions.Temporal context distinguishes records observable before the reference time from the future window used to construct prediction targets.Source: C. Unified Instance Schema · p. 5, D. Instance Generation · p. 6

D. Instance Generation

  • Per-database schema and data profiles record table structure and statistics such as row counts, distributions, cardinalities, null ratios, representative values, and temporal coverage.These profiles constrain generation to valid database elements and realistic values.Source: D. Instance Generation · p. 6
  • Candidate instances are generated separately for retrieval, analytical, relational-reasoning, and predictive questions, with task-specific evidence or target-construction fields.Prediction candidates include a reference time and observable-context definition; reasoning candidates may include SQL evidence.Source: D. Instance Generation · p. 6
  • Prediction questions require a reference time, while reasoning questions include temporal conditions only when the question requires them.Selected time ranges must fall within the corresponding database's temporal coverage.Source: D. Instance Generation · p. 6

E. Ground-Truth Construction

Ground truth is constructed from executed, validated SQL for reasoning questions and from future records under a pre-reference-time context for prediction questions. Automated checks and manual review filter benchmark instances for correctness and temporal validity.

E. Ground-Truth Construction

  • Reasoning answers are denotations from validated SQL executed against the database, with results stored as raw_value and normalized by answer type.Complete set-valued outputs are retained for evaluation, even when display examples are truncated.Source: E. Ground-Truth Construction · p. 6
  • Prediction labels are derived from future records relative to tref, while input features and relational context use only records before tref.Separating context from future labels prevents temporal leakage.Source: E. Ground-Truth Construction · p. 6

F. Quality Control

  • 4,040 validated instances remain after automatic and manual checks, which revised or removed 382 candidates.Checks cover schema validity, executable SQL, answer consistency, temporal constraints, metadata, formatting, semantic correctness, and ambiguity.Source: F. Quality Control · p. 6
  • The hepatitis domain has no prediction-oriented instances because it is the only non-temporal database and lacks timestamp attributes.Source: F. Quality Control · p. 6

V. EXPERIMENTS AND ANALYSIS

The experiments assess native solvers on their intended tracks and modular systems that route questions to reasoning or prediction modules. Results show persistent limits in denotation-level reasoning, with performance varying across answer types and domains.

V. EXPERIMENTS AND ANALYSIS

  • Native-solver evaluation isolates execution on each method's intended track, while modular QA tests whether a router can dispatch questions to the appropriate execution module.The analysis examines routing alongside reasoning and prediction, with an oracle router serving as a routing upper bound.Source: §V · p. 7, A. Experimental Setup · p. 8

A. Experimental Setup

  • 0.49 is the best reasoning macro-average (Claude-Schema), and only 52.1% of its executable SQL outputs match the gold denotation.94.7% of Claude-Schema outputs are executable; the error audit identifies schema grounding, joins, filters, aggregations, and relational composition as failure areas.Source: Table IV · p. 8, B. RQ1: Native Reasoning over… · p. 9
  • 0.81 EM (Claude-2Hop) on scalar answers contrasts with near-zero ranking scores; C3's Set-F1 is 0.83 and Set-EM is 0.65, while its Boolean Accuracy is 0.55 versus DIN-SQL's 0.53.Aggregate domain scores mask substantial variation by answer form, and specialized strengths do not carry uniformly across types.Source: Table V · p. 9, B. RQ1: Native Reasoning over… · p. 10
  • Reasoning is scored against denotation-level answers using answer-type-specific metrics, while predictive classification and regression use Accuracy and nMAE, respectively.Routing metrics assess track, query type, answer form, involved tables, and module selection.Source: A. Experimental Setup · p. 9
  • DIN-SQL leads the LLM-based methods on NCAA, Airline, and Stats but scores 0.16 on Dallas; solver advantages therefore vary across domains.CHESS and XiYanSQL-32B improve over DIN-SQL on the four domains evaluated for those methods.Source: B. RQ1: Native Reasoning over… · p. 9
  • Claude-Schema remains ahead on Claude-generated questions (0.57 vs. 0.54) and Qwen-generated questions (0.58 vs. 0.51) in a preliminary two-domain comparison.The comparison uses Olist and NCAA and provides no evidence of a Claude-specific home-field advantage.Source: B. RQ1: Native Reasoning over… · p. 10, Table VIII · p. 10

C. RQ2: Predictive QA over Future Outcomes

The predictive results distinguish binary classification from scalar regression: LLMs are competitive on Boolean prediction using aggregate statistics but remain unstable on numerical estimation. Trained relational models achieve stronger aggregate regression performance.

C. RQ2: Predictive QA over Future Outcomes

  • 0.71 macro-average Accuracy (Claude-only) on Boolean prediction rises to 0.74 with Claude-GraphCtx, a marginal gain over schema and aggregate statistics alone.The passage attributes the competitiveness to binary questions' coarse polarity judgment and says local records add limited signal beyond aggregate profiling.Source: C. RQ2: Predictive QA over Future… · pp. 10–11
  • 0.73 is the best macro-average nMAE (RelGNN) for scalar regression, while LLM-based methods show high nMAE and cross-domain instability.Llama8B-only reaches 50.04 nMAE on Airline; adding local records offers only marginal improvement over LLM-only.Source: C. RQ2: Predictive QA over Future… · p. 11
  • LLMs can exploit coarse historical priors for Boolean prediction but remain unstable on scalar regression; trained relational models perform better in aggregate on regression.The results contrast direct estimates from prompted summaries or local records with task-specific learning from relational histories.Source: C. RQ2: Predictive QA over Future… · p. 11

D. RQ3: Task-Aware Routing and End-to-End QA

Routing errors affect end-to-end QA when execution modules are fixed, while strong routers approach the oracle and schema grounding alone does not ensure correct capability selection.

D. RQ3: Task-Aware Routing and End-to-End QA

  • Track Accuracy is 0.97 for Claude-Router and 0.95 for Qwen32B-Router, near the Oracle upper bound, while Llama8B-Router scores 0.29 versus Random-Router’s 0.49.Source: D. RQ3: Task-Aware Routing and… · p. 11
  • Llama8B-Router’s Table-F1 of 0.78 alongside Track Accuracy of 0.29 shows that schema identification can succeed while capability selection fails.The passage attributes routing to understanding the question’s temporal and predictive intent, not schema linking alone.Source: D. RQ3: Task-Aware Routing and… · p. 11
  • Llama8B-Router scores 0.03 on Reasoning, compared with 0.44 for Claude-Router and Oracle-Router; Claude-Router also trails Oracle-Router on Predictive Bool. Acc. (0.84 vs. 0.87) and Scalar nMAE (0.98 vs. 0.91).With execution modules fixed, capability-selection errors affect end-to-end performance.Source: D. RQ3: Task-Aware Routing and… · p. 11

E. RQ4: Diagnostic Analysis

Diagnostic results vary by difficulty and query type, with different failure patterns across reasoning and prediction methods; predictive solvers are generally cheaper than LLM-based reasoning pipelines.

E. RQ4: Diagnostic Analysis

  • Annotated difficulty is not monotonic: Text-to-SQL degrades on Hard Reasoning instances, while LLM methods stay relatively stable; predictive trends vary, especially for scalar regression.Scalar regression depends on temporal signal strength and label distribution as well as structural complexity.Source: E. RQ4: Diagnostic Analysis · p. 11, F. Efficiency Analysis · p. 12
  • DIN-SQL with Claude scores 0.490 on Analytical queries but 0.253 on Relational Reasoning; graph and RDL models are generally more reliable on prediction, especially scalar regression.LLMs remain competitive on some Boolean prediction tasks; failure modes vary across difficulty and query type.Source: E. RQ4: Diagnostic Analysis · p. 11, Figure 10 · p. 12

F. Efficiency Analysis

  • Predictive solvers are substantially cheaper than LLM-based reasoning pipelines, with tabular methods fastest and graph or RDL methods adding moderate overhead.Among LLM methods, runtime depends on model size and prompting strategy, and multi-stage Text-to-SQL pipelines such as C3 are especially expensive.Source: F. Efficiency Analysis · p. 11, Figure 11 · p. 12

VI. CONCLUSION AND FUTURE WORK

RDL-Bench covers current-state reasoning and future-oriented prediction over relational databases. The conclusion identifies capability mismatch and several open problems, and the authors describe their use of language tools in manuscript preparation.

VI. CONCLUSION AND FUTURE WORK

  • RDL-Bench is a benchmark for task-aware question answering over relational databases, covering current-state reasoning and future-oriented prediction.Source: §VI · p. 12
  • A mismatch between the question’s required capability and the invoked execution mechanism produces fundamentally wrong answer types.Source: §VI · p. 12
  • Open problems include relational reasoning that jointly handles schema grounding, multi-hop composition, and answer-form control, plus predictive QA beyond binary classification and multi-turn QA.Ranking, set-valued prediction, and scalar regression under weak temporal signal are described as largely unsolved; lightweight routers combining schema grounding with reliable capability selection are also motivated.Source: §VI · p. 12

AI-GENERATED CONTENT ACKNOWLEDGEMENT

  • The authors report using Claude and GPT tools for language clarity, phrasing, presentation, and code implementation, with the code verified by the authors.They state that the research ideas, benchmark design, experiments, analysis, and conclusions were developed and verified by the authors.Source: AI-GENERATED CONTENT ACKNOWLEDGEMENT · p. 13

Related papers

Loading 2610.21382v1…