STQA: A Benchmark for Stock-Focused Tabular Question Answering over Historical and Forecasted Data

• Existing benchmarks and systems rarely evaluate the unified workflow of historical retrieval, forecasting, and reasoning over predicted values, leaving composite tool coordination difficult to study.Forecast-based reasoning specifically requires invoking a forecaster, interpreting its outputs, and making high-level judgments.Source: §1 · pp. 1–2
• STQA unifies historical query reasoning, numerical forecasting, and forecast-based reasoning in a benchmark of 31,400 question–answer pairs grounded in data from 4,417 stocks.The benchmark also provides intent and slot annotations for mapping natural-language questions to structured analytical specifications.Source: §1 · p. 2
• SQFRS provides a unified, agent-based baseline that parses intent and slots, routes tasks to SQL and forecasting workflows, and generates answers grounded in tool evidence.The paper positions SQFRS as a transparent operational baseline, not a novel algorithmic contribution.Source: §1 · p. 2
• Experiments quantify task difficulty and report that SQFRS improves existing LLMs’ performance on STQA.The supplied introduction does not provide task-specific performance values.Source: §1 · p. 2

2.1 QA datasets

• Existing QA datasets generally lack unified support for historical retrieval, forecasting, and forecast-based reasoning, and rely on limited intent diversity.STQA integrates these three task types in one benchmark with explicit intent annotations; the supplied table caption identifies a dataset comparison but provides no cell values.Source: §2.1 · pp. 2–3, Table 1 · p. 3

2.2 QA methods

• Prior stock QA and agent methods remain limited in task coverage and cross-scenario generalization, while fixed pipelines hinder complex queries requiring coordinated database and forecasting tools.SQFRS addresses this by planning multi-step workflows that coordinate SQL retrieval with time-series forecasting (Li et al., 2025; Zeng et al., 2025).Source: §2.2 · p. 3

3.1 Stock Data Collection and Preprocessing

• STQA retains five daily fields from FNSPID stock tables, aligns eligible series to a shared trading calendar, and extracts 700 trading days per stock.Preprocessing excludes tables that do not cover 2021–2023 and removes weekend dates; 4,417 tables remain after preprocessing.Source: §3.1 · p. 3

3.2 Question–Answer Generation

• The benchmark instantiates 72 templates with concrete stocks, dates, and values; questions and answers are workday-based because the underlying series contain trading days only.The dataset provides intent, slots, ground-truth answers, and task-specific SQL or historical data for the three task types.Source: §3.2 · p. 3, §3.4 · p. 5

3.2.1 Query Reasoning

• Query reasoning uses 42 templates spanning single- and multi-table retrieval, temporal expressions, comparisons, and multi-step or numerical reasoning.Multi-table templates include cross-stock and multi-time-point questions.Source: §3.2.1 · pp. 3–4

3.2.2 Numerical Forecasting

• Numerical forecasting uses 16 templates for opening, closing, high, and low prices, requiring temporal parsing, historical-series retrieval, and future-price prediction.Trading volume is excluded as an unstable forecasting target.Source: §3.2.2 · p. 4

3.2.3 Forecast-based Reasoning

• Forecast-based reasoning uses 14 templates that combine retrieval, forecasting, and logical judgments over predicted values, including trends, extrema, and threshold conditions.The templates cover single-point, fuzzy-time, multi-time-point, and more complex logical reasoning.Source: §3.2.3 · p. 4

3.3 Intent and Slot Annotation

• STQA annotates ten intents and five slot types, covering historical queries and future prices, trends, extrema, tickers, and temporal expressions.Fuzzy dates are normalized against the global trading calendar.Source: §3.3 · p. 4

3.4 Query Rewriting and Quality Control

• GPT-oss-120B paraphrases template questions while preserving meaning-critical entities and conditions, and annotators correct or discard cases with drift or ambiguity.The corpus contains 31,400 QA pairs split 7:2:1 across training, validation, and test; manual review checks grammaticality, naturalness, and semantic equivalence.Source: §3.4 · pp. 4–5
• 3.36/4 was the average automatic naturalness score after rewriting, up from 3.18/4; human ratings also rose from 2.48 to 2.84.The human evaluation used eight volunteers rating 50 randomly sampled test questions; the automatic scorer was Qwen3-30B.Source: §3.4 · p. 4

4.1 Overview

• SQFRS routes parsed stock questions among query reasoning, numerical forecasting, and forecast-based reasoning workflows through an LLM-based dynamic routing agent.The SLU module predicts intent and slots, which guide workflow selection and tool dispatch.Source: §4.1 · p. 5, Figure 1 · p. 6

4.2 SLU Module

• The SLU module supplies intent and slot labels that represent the user’s goal and arguments such as stock ticker, year, month, or date.These labels provide additional guidance to the routing agent.Source: §4.2 · p. 5

4.3 Dynamic Routing Agent

• The Dynamic Routing Agent uses the intent, slot values, and original query to select and execute a predefined Query, Forecast, or Forecast-Reasoning workflow.A structured prompt specifies tool capabilities and admissible calling orders for each workflow.Source: §4.3 · p. 5

4.4 SQL Generation Module

• Query2SQL generates SQL for structured retrieval, while History2SQL retrieves historical time-series data for downstream forecasting and reasoning.The SQL generation module uses the SLU intent and slot information.Source: §4.4 · pp. 5–6

4.5 Database Query Module

• The PostgreSQL Database Query module executes SQL from Query2SQL or History2SQL and returns standardized results.Source: §4.5 · p. 6

4.6 Time Series Forecasting Module

• The forecasting module converts retrieved historical dates and prices into a numerical series and predicts prices for the next 15 trading days.Weekend dates are assigned zero prices during calendar alignment and excluded from forecasting metric calculations.Source: §4.6 · p. 6

4.7 Forecast Decision Module

• Forecast Selection extracts a requested value from the prediction sequence, whereas Forecast-based Reasoning uses both historical and forecast series to answer judgment or extreme-value queries.The reasoning module’s examples include a canonical reasoning trajectory to demonstrate the intended path from inputs to conclusion.Source: §4.7 · pp. 6–7

5.1 Evaluation Metrics and Baselines

• Reasoning tasks use exact-match Accuracy and item-level F1, while numeric forecasting uses MAE, MSE, and MRE.Baselines include four open-source LLM backbones; Time-MoE is selected as the default forecaster after comparison with five alternatives.Source: §5.1 · p. 7

5.2 Results and Analysis

• GEMMA 4-31B leads reasoning tasks with 72.38% QR accuracy and 62.43% FR accuracy, while GPToss-120B has the lowest NF MRE at 0.0590.Lower NF MRE does not necessarily correspond to higher FR accuracy.Source: §5.2 · p. 7
• Removing the workflow mechanism drops average accuracy by 8.8425% on QR and 12.2575% on FR, and increases NF MRE by 0.01202.SQFRS also achieves average routing accuracy above 99.9%.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Ablation Study

• With SLU enabled, SQFRS scores 71.23% QR, 0.0612 NF MRE, and 59.68% FR, versus 66.15%, 0.0704, and 54.77% without SLU.The authors attribute the gains to intent labels helping workflow allocation and slot labels constraining SQL generation.Source: Table 3 · p. 8, §5.3 · p. 8
• Ground-truth workflow routing yields the smallest gains, while ground-truth SQL produces a larger FR gain of 2.55% and NF MRE reduction of 0.0013.The study injects ground-truth labels in execution order across SLU, routing, SQL generation, and forecast data.Source: §5.3 · p. 8
• Ground-truth forecast data yields the largest FR gain at 19.85%, but even ground-truth labels at every intermediate step leave FR accuracy at 86.14%.The authors identify logical reasoning as a primary remaining bottleneck.Source: §5.3 · p. 8
• Manually written questions reduce average QR performance by 17.06% and FR performance by 10.51% relative to template-generated questions.The evaluation uses 100 human-written questions: 30 QR, 30 NF, and 40 FR.Source: §5.3 · p. 8, Table 4 · p. 9
• SQFRS with GEMMA 4-31B reaches 68.33% FR accuracy, compared with 69.17% for stock investors and 63.33% for noninvestors.Under direct reasoning without forecasting tools, its FR accuracy is 56.67%.Source: Table 5 · p. 9, §5.3 · p. 9

6 Conclusion

• STQA contains 31,400 question-answer pairs grounded in 4,417 stock tables and unifies query reasoning, numerical forecasting, and forecast-based reasoning.The benchmark targets end-to-end research integrating multiple paradigms.Source: §6 · p. 9
• SQFRS is proposed as a strong baseline approach for the benchmark.The conclusion reports that experiments validate the method while underscoring the benchmark’s difficulty.Source: §6 · p. 9

Limitation

• The benchmark primarily uses stock-market data, a defined evaluation context whose measuring methodologies the authors say apply to other financial tasks.Examples named include investment strategy and credit risk assessment.Source: Limitation · p. 9
• The selected evaluation models are representative rather than exhaustive, and the authors identify testing additional models as future work.They describe the selection as targeting models relevant to a reliable baseline.Source: Limitation · p. 9
• The work prioritizes constructing STQA over improving a specific method, with its focus on benchmark utility and SQFRS as a baseline.The authors distinguish this emphasis from method-oriented research.Source: Limitation · p. 9
• Template-generated questions may not fully represent real-world inquiry complexity, although LLM rewriting and model and human assessments are used to increase expression diversity.The authors describe the rewritten expressions as richer than mere template-based presentation.Source: Limitation · p. 10

A.2 Experimental Results of SQL Accuracy

• 99.40% average pass@1 is reported for History2SQL, compared with 71.23% for Query2SQL.The passage says QR correctness is determined directly by SQL execution, while NF and FR depend largely on retrieval quality.Source: §A.2 · p. 12, Table 6 · p. 13

A.3 Detailed Results of the SLU Module

• 72.16% QR F1 and 71.23% QR accuracy, 0.0612 NF MRE, and 62.22% FR F1 and 59.68% FR accuracy are reported for SQFRS, versus 66.82%, 66.15%, 0.0704, 54.77%, and 56.38%, respectively, without SLU.The authors attribute workflow allocation to intent labels and more precise SQL-generation constraints to slot labels.Source: §A.3 · p. 13

A.4 Robustness Analysis via In-Context Learning

• The ICL setting replaces fine-tuned BERT with few-shot prompting to generate SLU labels without additional training.The authors report generally higher NF MRE and slightly worse QR and FR performance than SQFRS, linking the shortfall to numerical extraction and reasoning over predicted values.Source: §A.4 · p. 13
• 70.08% QR accuracy, 0.0686 NF MRE, and 58.47% FR accuracy are reported for ICL-based LLM-SLU, with QR and FR accuracy lower than BERT-fine-tuned SQFRS and NF MRE generally higher.The study reports this pattern across backbone models.Source: §A.4 · p. 13

A.5 Fine-Grained Analysis of Forecast-Based Question Answering

• 52.8%–55.6% accuracy on multi-period trend comparison tasks contrasts with 66%–87% on threshold comparison tasks across the four models.For extreme-value reasoning, accuracy declines by 31.5 and 10.0 percentage points for Qwen3-30B and GPT-oss-20B, and by 4.0 and 4.5 points for GPT-oss-120B and GEMMA 4-31B, respectively.Source: §A.5 · p. 13, Table 9 · p. 14
• Forecast-based reasoning tasks require increasingly complex operations, from direct numerical comparison to temporal alignment, value extraction, and cross-period comparison.The reported results associate greater reasoning complexity with significantly lower model performance.Source: §A.5 · p. 13, P R F1 Acc MSE MAE MRE P R F1 Acc · p. 14

B Selection of Time-Series Forecasting Models

• The study selects open, close, high, and low prices rather than trading volume as forecasting targets because short-term volume can be affected by random fluctuations and sudden information shocks.The comparisons use a fixed workflow setting and separate NF and FR ablations across six forecasting approaches.Source: B Selection of Time-Series Forecasting… · p. 14
• On the original dataset, Time-MoE has the lowest reported time-series-baseline MREs: 0.0686 for open, 0.0799 for close, 0.0710 for high, and 0.0717 for low.I-Transformer has the largest reported forecast errors for open (0.1154), close (0.0998), and low (0.1167); for high, its MRE is 0.0695.Source: B Selection of Time-Series Forecasting… · p. 14
• Time-MoE is reported as the best-performing time-series forecast model across the original and STQA datasets.Its performance consistency across datasets and backbone configurations supports the authors’ conclusion about its stability.Source: B Selection of Time-Series Forecasting… · p. 15

C Statistics

• STQA comprises 31,400 QA pairs from 4,417 tables, divided among 10,500 query-reasoning, 10,400 numeric-forecasting, and 10,500 forecast-based reasoning pairs.Its data cover 26 question types, with 10 intent categories and 5 slot categories.Source: C Statistics · p. 15

D Bottleneck Exploration

• GT SLU yields QR and FR average accuracy gains of +1.39% and +1.01%, while GT WF gains are +0.38% and +0.71%, respectively.Their NF MRE reductions are -0.00015 for SLU and -0.0002 for workflow; workflow accuracy averages 99.9%.Source: D Bottleneck Exploration · p. 15
• GT SQL improves FR accuracy by +0.257% and reduces NF MRE by -0.000962, exceeding the SLU and workflow gains on these measures.The passage reports SQL average accuracy of approximately 99.40% and identifies SQL generation as having meaningful room for optimization.Source: D Bottleneck Exploration · p. 15
• GT FD delivers the largest gains: FR accuracy improves by +18.74% and NF MRE decreases by -0.055228.Even with GT FD, FR accuracy reaches only 86.14%, leaving a 13.86% gap to the upper bound.Source: D Bottleneck Exploration · p. 15, QR NF FR · p. 18

QR NF FR

• Across the visualized GT settings, FR accuracy rises as supervision is added, while NF MRE falls; adding FD at the final stage brings the most pronounced FR improvement.The discussion identifies effective FD acquisition and use, and FR-oriented reasoning, as the primary bottlenecks.Source: Figure 3 · p. 18, QR NF FR · p. 18
• Even with all GT signals enabled, FR accuracy remains below 90% and NF MRE remains above zero.The findings indicate residual error and further room for improvement in both tasks.Source: QR NF FR · p. 18

E Rewrite Result

• Rewritten QA pairs contain a higher proportion of high-scoring samples than original versions under Qwen3-30B automatic evaluation.The evaluation compares 50 original instances with their 50 rewritten versions.Source: E Rewrite Result · p. 18
• Eight evaluators independently scored 100 shuffled instances comprising 50 original questions and their 50 rewritten versions.Scores were averaged across evaluators, and questions were pre-labeled before random assignment.Source: E Rewrite Result · p. 19

F Prompts

• Appendix F documents prompt templates for query and history SQL, dynamic routing, forecast selection, and forecast-based reasoning.The templates are presented as reproducibility and implementation references.Source: F Prompts · p. 19

G Case Study

• Faulty dynamic routing can select the wrong workflow, deviate after negative tool feedback, and end in a fabricated answer.The analysis associates this pattern with weaknesses in task allocation and error recovery.Source: G Case Study · p. 19
• History2SQL can drift toward prior knowledge or fixed SQL templates despite correct upstream outputs, yielding inaccurate queries and unreliable historical retrieval.The resulting SQL deviations propagate downstream through the workflow.Source: G Case Study · p. 19
• The forecasting module can produce large-MRE predictions even when the preceding SLU, History2SQL, and database-query outputs are correct.The case study identifies forecasting as a major limiting factor for tasks dependent on future estimates.Source: G Case Study · p. 19
• Forecast Selection can extract the wrong semantic value from correct TSF outputs, such as returning a date when asked for an opening price.The analysis points to incomplete understanding of answer types and semantic intent.Source: G Case Study · p. 19
• The system may return answers unrelated to ground truth even when the workflow modules produce correct outputs.The passage attributes such unsupported inferences to unstable reasoning chains, degraded decision-making, or misinterpreted task objectives.Source: G Case Study · p. 20